From 7d298cdc6009f9d31ddb5db1f1da0ef0f2b00cf0 Mon Sep 17 00:00:00 2001 From: "claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla)" Date: Wed, 12 Aug 2026 22:41:06 +0000 Subject: [PATCH 1/7] Serve the feed timeline from D1; Fly proxy becomes an ingest-push crawler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A refresh was `1 + ceil((N-8)/50)` sequential batch calls, each hopping Worker -> Fly and then paying ~18 chunked D1 queries to annotate read state. Now the proxy pushes new/edited items into a central D1 archive and a refresh is one `GET /api/v2/timeline` — a single query that joins subscriptions and read state, so `getReadKeys` leaves the feed path entirely and reads never touch Fly. Backend: migration 0061 drops the dormant pre-Fly trio and creates `feeds` + `feed_items` (monotonic seq, unique (feed_url, guid), NOT NULL content_hash) plus the items_generation token; `routes/ingest.ts` adds the secret-authenticated, fail-closed `POST /api/internal/ingest` and `GET /api/internal/crawl-set` with an idempotent upsert, an 8 KB stored content cap and the 5,000-item per-feed sanity cap (the only pruning — D1 is an archive); `routes/timeline.ts` serves incremental drains and per-feed cold starts; `/api/v2/feeds/fetch` is re-backed with D1 plus a pull-through for a feed nobody has crawled yet. Proxy: the durable log is the outbox. `push_state` marks what reached D1, dirty rows drain in seq order every 15 s with capped backoff, the crawl set is pulled back every 5 min to keep feeds warm now that reads no longer stamp them, and push_state cascades on every feed_items delete. Everything is gated on INGEST_URL, so an environment without it crawls exactly as before. Frontend: one drain loop against a single global cursor in Dexie metadata, with the legacy batch path kept as a fallback for an old backend, a rollback, or an environment whose crawler isn't pushing yet. Also: a staging Fly proxy (fly.staging.toml + CI job + staging FEED_PROXY_URL) so the two environments stop sharing one machine, and admin feed health re-pointed at the archive with storage/churn alerts. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) --- .github/workflows/feed-proxy-deploy.yml | 21 + CLAUDE.md | 9 +- README.md | 3 + admin/src/lib/metrics/feeds.ts | 91 +++- admin/src/lib/metrics/system.ts | 2 +- admin/src/lib/queries/feeds.ts | 38 +- admin/src/lib/queries/system.ts | 3 +- admin/src/lib/types.ts | 7 +- admin/src/routes/feeds/+page.server.ts | 2 +- admin/src/routes/feeds/+page.svelte | 36 +- backend/CLAUDE.md | 20 +- backend/migrations/0061_feed_timeline.sql | 55 +++ backend/src/index.ts | 20 +- backend/src/routes/feeds-v2.ts | 87 ++-- backend/src/routes/ingest.ts | 351 +++++++++++++++ backend/src/routes/timeline.ts | 292 +++++++++++++ backend/src/services/rate-limit.ts | 3 + backend/src/types.ts | 3 + backend/test/feed-timeline.spec.ts | 412 ++++++++++++++++++ backend/wrangler.toml | 5 +- docs/plans/D1_FEED_TIMELINE.md | 141 ++++++ docs/plans/RETENTION_SYNC_PLAN.md | 6 + e2e/seed.ts | 49 +++ e2e/timeline.spec.ts | 76 ++++ feed-proxy/README.md | 42 +- feed-proxy/fly.staging.toml | 55 +++ feed-proxy/fly.toml | 9 + feed-proxy/src/app.ts | 72 ++- feed-proxy/src/index.ts | 98 +++++ feed-proxy/src/ingest-push.test.ts | 335 ++++++++++++++ feed-proxy/src/ingest-push.ts | 242 ++++++++++ frontend/CLAUDE.md | 20 +- .../src/lib/components/ImportOPMLModal.svelte | 8 +- frontend/src/lib/services/api.ts | 67 ++- frontend/src/lib/services/feedFetcher.ts | 217 +++++++-- .../src/lib/services/timelineSync.test.ts | 154 +++++++ frontend/src/lib/services/timelineSync.ts | 147 +++++++ frontend/src/lib/types/index.ts | 3 + scripts/dev-local.sh | 17 +- 39 files changed, 3083 insertions(+), 135 deletions(-) create mode 100644 backend/migrations/0061_feed_timeline.sql create mode 100644 backend/src/routes/ingest.ts create mode 100644 backend/src/routes/timeline.ts create mode 100644 backend/test/feed-timeline.spec.ts create mode 100644 docs/plans/D1_FEED_TIMELINE.md create mode 100644 e2e/timeline.spec.ts create mode 100644 feed-proxy/fly.staging.toml create mode 100644 feed-proxy/src/ingest-push.test.ts create mode 100644 feed-proxy/src/ingest-push.ts create mode 100644 frontend/src/lib/services/timelineSync.test.ts create mode 100644 frontend/src/lib/services/timelineSync.ts diff --git a/.github/workflows/feed-proxy-deploy.yml b/.github/workflows/feed-proxy-deploy.yml index f0542f33..42df642f 100644 --- a/.github/workflows/feed-proxy-deploy.yml +++ b/.github/workflows/feed-proxy-deploy.yml @@ -54,6 +54,27 @@ jobs: - name: Run tests run: bun test + deploy-staging: + name: Deploy to Staging + needs: [typecheck, test] + # Same cadence as the staging Worker: every push to main. Staging has its own + # Fly app (fly.staging.toml) so it never shares the prod machine or its cache, + # and config drift between the two files surfaces here before a release. + if: github.event_name == 'push' && github.ref == 'refs/heads/main' + runs-on: ubuntu-latest + environment: staging + steps: + - name: Checkout + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + + - name: Setup flyctl + uses: superfly/flyctl-actions/setup-flyctl@ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1 # master + + - name: Deploy + run: flyctl deploy --remote-only --config fly.staging.toml + env: + FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }} + deploy: name: Deploy to Production needs: [typecheck, test] diff --git a/CLAUDE.md b/CLAUDE.md index e08e5adc..69061737 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ This is a monorepo with 6 packages: - `backend/` - Cloudflare Workers API - `frontend/` - SvelteKit PWA - `admin/` - SvelteKit admin dashboard (Cloudflare Pages) -- `feed-proxy/` - Feed caching proxy (Fly.io) +- `feed-proxy/` - Feed crawler + article extraction (Fly.io; one app per environment) - `linkblog-site/` - Standalone SvelteKit app rendering public linkblogs at `linkblogs.skyreader.app` (Cloudflare Pages) - `extension/` - Chrome extension for one-click saves with live-DOM article extraction (Manifest V3) @@ -85,6 +85,9 @@ This script runs D1 migrations, then starts the feed proxy (port 3000), backend ``` FRONTEND_URL=http://127.0.0.1:5173 FEED_PROXY_URL=http://127.0.0.1:3000 + # Shared with the proxy (dev-local.sh exports the same value as PROXY_SECRET). + # Feed ingest is fail-closed, so without this the local reader stays empty. + FEED_PROXY_SECRET=dev-proxy-secret ``` **Resetting the local database:** @@ -213,7 +216,9 @@ AT Protocol (Bluesky PDS) + Fly.io Feed Proxy + Jetstream Firehose 1. **Auth:** Handle → DID resolution → OAuth PKCE/DPoP → session token 2. **Subscriptions:** Stored in user's PDS, cached locally in IndexedDB -3. **Feed Updates:** Frontend requests feeds → backend proxies via Fly.io feed cache +3. **Feed Updates:** Fly.io proxy crawls feeds and pushes new items into D1 (`feed_items`); the + frontend refreshes with one `GET /api/v2/timeline` query that joins subscriptions and read + state. Reads never touch Fly — see `docs/plans/D1_FEED_TIMELINE.md` 4. **Social:** Jetstream firehose → D1 shares table → frontend polls for updates ## AT Protocol Integration diff --git a/README.md b/README.md index b5f910d5..cfebcddb 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,9 @@ cd skyreader ``` FRONTEND_URL=http://127.0.0.1:5173 FEED_PROXY_URL=http://127.0.0.1:3000 + # Shared with the proxy (dev-local.sh exports the same value as PROXY_SECRET). + # Feed ingest is fail-closed, so without this the local reader stays empty. + FEED_PROXY_SECRET=dev-proxy-secret ``` ### Local Development diff --git a/admin/src/lib/metrics/feeds.ts b/admin/src/lib/metrics/feeds.ts index f173d22f..92505291 100644 --- a/admin/src/lib/metrics/feeds.ts +++ b/admin/src/lib/metrics/feeds.ts @@ -1,26 +1,100 @@ import type { MetricDefinition } from '$lib/types'; +// A subscribed feed whose last ingest is older than this isn't being crawled any +// more — the headline failure mode of the ingest architecture (nothing stamps +// the crawl set, so feeds silently age out of the proxy's warm loop). The warm +// loop refreshes on the order of minutes, so an hour is a real alarm, not noise. +const STALE_INGEST_SECONDS = 60 * 60; + +// D1's hard ceiling is 10 GB. Alert well before it, so there's time to act +// (lower the ingest content cap → tier old bodies to R2 → revisit retention). +const ARCHIVE_ALERT_BYTES = 6 * 1024 * 1024 * 1024; + +// The per-feed sanity cap is 5,000 items (backend/src/routes/ingest.ts). A feed +// approaching it is a GUID-churn bug signal, not steady state. +const CHURN_WARN_ITEMS = 3000; + export const feedMetrics: MetricDefinition[] = [ { id: 'total_feeds', category: 'Feeds', query: async (db) => { + const r = await db.prepare('SELECT COUNT(*) as count FROM feeds').first<{ count: number }>(); + return { label: 'Crawled Feeds', value: r?.count ?? 0 }; + }, + }, + { + id: 'stale_ingest_feeds', + category: 'Feeds', + query: async (db) => { + const cutoff = Math.floor(Date.now() / 1000) - STALE_INGEST_SECONDS; const r = await db - .prepare('SELECT COUNT(*) as count FROM feed_metadata') + .prepare( + `SELECT COUNT(*) as count FROM feeds f + WHERE (f.last_ingest_at IS NULL OR f.last_ingest_at < ?) + AND EXISTS (SELECT 1 FROM subscriptions_cache sc + WHERE sc.feed_url = f.feed_url AND sc.active = 1)` + ) + .bind(cutoff) .first<{ count: number }>(); - return { label: 'Total Feeds', value: r?.count ?? 0 }; + const count = r?.count ?? 0; + return { + label: 'Subscribed Feeds Not Ingesting', + value: count, + status: count > 0 ? 'warning' : 'healthy', + }; + }, + }, + { + id: 'archive_items', + category: 'Feeds', + query: async (db) => { + const r = await db + .prepare('SELECT COUNT(*) as count FROM feed_items') + .first<{ count: number }>(); + return { label: 'Archived Items', value: r?.count ?? 0 }; + }, + }, + { + id: 'archive_size', + category: 'Feeds', + query: async (db) => { + // Estimated, not exact: summing LENGTH(item_json) across the whole archive + // is a full scan that only grows. Average the newest 1,000 rows (written + // under the current content cap) and multiply by the row count. + const [countRow, avgRow] = await Promise.all([ + db.prepare('SELECT COUNT(*) as count FROM feed_items').first<{ count: number }>(), + db + .prepare( + `SELECT AVG(LENGTH(item_json)) as avg + FROM (SELECT item_json FROM feed_items ORDER BY seq DESC LIMIT 1000)` + ) + .first<{ avg: number | null }>(), + ]); + const bytes = Math.round((countRow?.count ?? 0) * (avgRow?.avg ?? 0)); + return { + label: 'Archive Size (est.)', + value: (bytes / (1024 * 1024 * 1024)).toFixed(2), + unit: 'GB', + status: bytes > ARCHIVE_ALERT_BYTES ? 'warning' : 'healthy', + }; }, }, { - id: 'feeds_with_errors', + id: 'churn_feeds', category: 'Feeds', query: async (db) => { const r = await db - .prepare('SELECT COUNT(*) as count FROM feed_metadata WHERE error_count > 0') + .prepare( + `SELECT COUNT(*) as count FROM ( + SELECT feed_url FROM feed_items GROUP BY feed_url HAVING COUNT(*) > ? + )` + ) + .bind(CHURN_WARN_ITEMS) .first<{ count: number }>(); const count = r?.count ?? 0; return { - label: 'Feeds with Errors', + label: 'Feeds Near Sanity Cap', value: count, status: count > 0 ? 'warning' : 'healthy', }; @@ -31,7 +105,12 @@ export const feedMetrics: MetricDefinition[] = [ category: 'Feeds', query: async (db) => { const r = await db - .prepare('SELECT ROUND(AVG(subscriber_count), 1) as avg FROM feed_metadata') + .prepare( + `SELECT ROUND(AVG(subs), 1) as avg FROM ( + SELECT COUNT(*) as subs FROM subscriptions_cache + WHERE active = 1 GROUP BY feed_url + )` + ) .first<{ avg: number }>(); return { label: 'Avg Subscribers/Feed', value: r?.avg ?? 0 }; }, diff --git a/admin/src/lib/metrics/system.ts b/admin/src/lib/metrics/system.ts index 9a121e7e..baac25e1 100644 --- a/admin/src/lib/metrics/system.ts +++ b/admin/src/lib/metrics/system.ts @@ -2,7 +2,7 @@ import type { MetricDefinition } from '$lib/types'; const rowCountTables = [ { table: 'users', label: 'Users' }, - { table: 'feed_metadata', label: 'Feeds' }, + { table: 'feeds', label: 'Feeds' }, { table: 'feed_items', label: 'Feed Items' }, { table: 'subscriptions_cache', label: 'Subscriptions' }, ]; diff --git a/admin/src/lib/queries/feeds.ts b/admin/src/lib/queries/feeds.ts index 80c6087e..21c30d4f 100644 --- a/admin/src/lib/queries/feeds.ts +++ b/admin/src/lib/queries/feeds.ts @@ -1,9 +1,18 @@ import type { FeedRow, PaginatedResult } from '$lib/types'; +// Matches the dashboard's stale-ingest metric: a subscribed feed that hasn't +// been ingested within an hour isn't being crawled. +const STALE_INGEST_SECONDS = 60 * 60; + +/** + * Feed health from the D1 archive. Fetch errors and backoff now live with the + * crawler (the Fly proxy), so "healthy" here means "still ingesting": the last + * push we received for the feed is recent. + */ export async function listFeeds( db: D1Database, opts: { - filter?: 'all' | 'healthy' | 'erroring'; + filter?: 'all' | 'healthy' | 'stale'; sort?: string; order?: 'asc' | 'desc'; page?: number; @@ -19,27 +28,34 @@ export async function listFeeds( } = opts; const offset = (page - 1) * perPage; - const allowedSorts = ['feed_url', 'title', 'subscriber_count', 'error_count', 'last_fetched_at']; + const allowedSorts = ['feed_url', 'title', 'subscriber_count', 'item_count', 'last_ingest_at']; const sortCol = allowedSorts.includes(sort) ? sort : 'subscriber_count'; const sortDir = order === 'asc' ? 'ASC' : 'DESC'; + const cutoff = Math.floor(Date.now() / 1000) - STALE_INGEST_SECONDS; let where = ''; - if (filter === 'healthy') where = 'WHERE error_count = 0'; - else if (filter === 'erroring') where = 'WHERE error_count > 0'; + if (filter === 'healthy') where = 'WHERE f.last_ingest_at IS NOT NULL AND f.last_ingest_at >= ?'; + else if (filter === 'stale') where = 'WHERE f.last_ingest_at IS NULL OR f.last_ingest_at < ?'; + const filterBindings = filter === 'all' ? [] : [cutoff]; const countResult = await db - .prepare(`SELECT COUNT(*) as count FROM feed_metadata ${where}`) + .prepare(`SELECT COUNT(*) as count FROM feeds f ${where}`) + .bind(...filterBindings) .first<{ count: number }>(); const rows = await db .prepare( - `SELECT feed_url, title, site_url, subscriber_count, error_count, fetch_error, last_fetched_at - FROM feed_metadata - ${where} - ORDER BY ${sortCol} ${sortDir} - LIMIT ? OFFSET ?` + `SELECT f.feed_url, f.title, f.site_url, f.last_ingest_at, + (SELECT COUNT(*) FROM subscriptions_cache sc + WHERE sc.feed_url = f.feed_url AND sc.active = 1) AS subscriber_count, + (SELECT COUNT(*) FROM feed_items fi + WHERE fi.feed_url = f.feed_url) AS item_count + FROM feeds f + ${where} + ORDER BY ${sortCol} ${sortDir} + LIMIT ? OFFSET ?` ) - .bind(perPage, offset) + .bind(...filterBindings, perPage, offset) .all(); return { diff --git a/admin/src/lib/queries/system.ts b/admin/src/lib/queries/system.ts index 48febdd7..20971b7e 100644 --- a/admin/src/lib/queries/system.ts +++ b/admin/src/lib/queries/system.ts @@ -10,9 +10,8 @@ export async function getTableRowCounts(db: D1Database): Promise { 'subscriptions_cache', 'item_labels_cache', 'documents', - 'feed_metadata', + 'feeds', 'feed_items', - 'feed_cache', 'follows_cache', 'inapp_follows', 'sync_state', diff --git a/admin/src/lib/types.ts b/admin/src/lib/types.ts index 21d1d8ca..7682c1ee 100644 --- a/admin/src/lib/types.ts +++ b/admin/src/lib/types.ts @@ -29,9 +29,10 @@ export interface FeedRow { title: string | null; site_url: string | null; subscriber_count: number; - error_count: number; - fetch_error: string | null; - last_fetched_at: number | null; + // Rows this feed holds in the D1 archive. + item_count: number; + // Unix seconds of the last push we received from the crawler for this feed. + last_ingest_at: number | null; } export interface SubscriptionRow { diff --git a/admin/src/routes/feeds/+page.server.ts b/admin/src/routes/feeds/+page.server.ts index 025e1aa1..1e7e5c7d 100644 --- a/admin/src/routes/feeds/+page.server.ts +++ b/admin/src/routes/feeds/+page.server.ts @@ -3,7 +3,7 @@ import type { PageServerLoad } from './$types'; export const load: PageServerLoad = async ({ platform, url }) => { const db = platform!.env.DB; - const filter = (url.searchParams.get('filter') as 'all' | 'healthy' | 'erroring') ?? 'all'; + const filter = (url.searchParams.get('filter') as 'all' | 'healthy' | 'stale') ?? 'all'; const sort = url.searchParams.get('sort') ?? undefined; const order = (url.searchParams.get('order') as 'asc' | 'desc') ?? undefined; const page = parseInt(url.searchParams.get('page') ?? '1', 10); diff --git a/admin/src/routes/feeds/+page.svelte b/admin/src/routes/feeds/+page.svelte index 10441dfb..ad235f13 100644 --- a/admin/src/routes/feeds/+page.svelte +++ b/admin/src/routes/feeds/+page.svelte @@ -8,10 +8,19 @@ const filters = [ { value: 'all', label: 'All' }, - { value: 'healthy', label: 'Healthy' }, - { value: 'erroring', label: 'Erroring' }, + { value: 'healthy', label: 'Ingesting' }, + { value: 'stale', label: 'Stale' }, ] as const; + // Mirrors the backend/admin stale-ingest threshold: an hour without a push + // from the crawler means this feed isn't being crawled any more. + const STALE_INGEST_SECONDS = 60 * 60; + + function isStale(lastIngestAt: number | null): boolean { + if (!lastIngestAt) return true; + return lastIngestAt < Math.floor(Date.now() / 1000) - STALE_INGEST_SECONDS; + } + function filterUrl(filter: string): string { const url = new URL(pageState.url); url.searchParams.set('filter', filter); @@ -73,9 +82,8 @@ Title{sortIndicator('title')} URL Subs{sortIndicator('subscriber_count')} - Errors{sortIndicator('error_count')} - Last Error - Last Fetched{sortIndicator('last_fetched_at')} + Archived{sortIndicator('item_count')} + Last Ingest{sortIndicator('last_ingest_at')} Status {/snippet} @@ -84,17 +92,16 @@ {feed.title ?? '—'} {feed.feed_url} {feed.subscriber_count} - {feed.error_count} - {feed.fetch_error ?? '—'} - {formatDate(feed.last_fetched_at)} + {feed.item_count} + {formatDate(feed.last_ingest_at)} - 0 ? 'error' : 'healthy'} /> + {:else} No feeds found @@ -142,13 +149,4 @@ white-space: nowrap; font-size: 0.85rem; } - - .error-cell { - max-width: 200px; - overflow: hidden; - text-overflow: ellipsis; - white-space: nowrap; - font-size: 0.8rem; - color: var(--color-error); - } diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 957f245c..424a7bef 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -4,7 +4,14 @@ ## Project Overview -Skyreader backend is a Cloudflare Workers API that serves as a gateway between the frontend and the AT Protocol ecosystem. It handles authentication, RSS feed fetching via a Fly.io proxy, social features, saved articles, labels, and background Jetstream polling. +Skyreader backend is a Cloudflare Workers API that serves as a gateway between the frontend and the AT Protocol ecosystem. It handles authentication, the feed timeline, social features, saved articles, labels, and background Jetstream polling. + +**Feed reads are served from D1, not the proxy.** The Fly.io proxy is the crawler: it pushes new +and edited items into `feed_items` (`POST /api/internal/ingest`) and pulls the set of feeds to +crawl (`GET /api/internal/crawl-set`), both authenticated with the shared `FEED_PROXY_SECRET` +(fail-closed when unset). A client refresh is one `GET /api/v2/timeline` — a single query joining +subscriptions and read state. See `docs/plans/D1_FEED_TIMELINE.md`. The proxy is still on the path +for `/api/extract`, feed discovery, standard.site documents, and social context. ## Key Concepts @@ -46,7 +53,9 @@ See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed documentation. | File | Purpose | | ----------------------------- | ----------------------------------------------------- | | `src/routes/auth.ts` | OAuth flow (login, callback, logout, client metadata) | -| `src/routes/feeds-v2.ts` | RSS fetching via Fly.io proxy | +| `src/routes/timeline.ts` | `GET /api/v2/timeline` — the whole refresh, one query | +| `src/routes/ingest.ts` | Crawler endpoints: item ingest + crawl set | +| `src/routes/feeds-v2.ts` | Single-feed read (D1 + pull-through), discover, docs | | `src/routes/social.ts` | Social feed, popular, grouped, detect-content | | `src/routes/shares.ts` | User shares CRUD (with PDS sync) | | `src/routes/subscriptions.ts` | Subscription CRUD (with PDS sync) | @@ -88,9 +97,10 @@ Key tables: - `sessions` - Server-side sessions (tokens, DPoP key, expiry) - `subscriptions_cache` - Cached feed subscriptions from PDS - `shares` - Aggregated share data from Jetstream -- `feed_metadata` - Feed caching metadata (ETags, errors, shard_id) -- `feed_cache` - Parsed feed cache -- `feed_items` - Individual feed items +- `feeds` - One row per crawled feed (title/site/image + `last_ingest_at`) +- `feed_items` - The feed archive the timeline serves: every item the crawler has ever pushed, + keyed `(feed_url, guid)` with a monotonic `seq`. Never pruned in ordinary operation — see + `docs/plans/D1_FEED_TIMELINE.md` - `documents` - `site.standard.document` records from follows - `item_labels_cache` - Unified labels (read/starred/archived/tags) - `saved_articles` - Saved/bookmarked articles diff --git a/backend/migrations/0061_feed_timeline.sql b/backend/migrations/0061_feed_timeline.sql new file mode 100644 index 00000000..7baf881a --- /dev/null +++ b/backend/migrations/0061_feed_timeline.sql @@ -0,0 +1,55 @@ +-- D1-served feed timeline (ingest-push architecture). +-- +-- The Fly proxy stays the crawler but leaves the read path entirely: it pushes +-- new/edited items into the tables below, and a client refresh becomes ONE query +-- (feed_items JOIN subscriptions_cache LEFT JOIN item_labels_cache) instead of +-- N batched Worker->Fly hops plus chunked read-key lookups. +-- +-- Retention: D1 is the ARCHIVE. Ordinary ingest never deletes an item; the proxy's +-- own K=200 log is only the outbox (a delivery window). The single trim that +-- exists is a per-feed sanity cap (5,000) that healthy feeds never reach — see +-- routes/ingest.ts. + +-- The dormant pre-Fly trio. Zero references in backend/src; feed_cache was +-- already wiped to '{}' by 0018. Dropped so the names are free for the new shapes. +DROP TABLE IF EXISTS feed_items; +DROP TABLE IF EXISTS feed_cache; +DROP TABLE IF EXISTS feed_metadata; + +-- One row per crawled feed: metadata for the reader plus ingest observability. +CREATE TABLE feeds ( + feed_url TEXT PRIMARY KEY, + title TEXT, + site_url TEXT, + description TEXT, + image_url TEXT, + last_ingest_at INTEGER, + created_at INTEGER NOT NULL DEFAULT (unixepoch()) +); + +-- The durable item archive. Same invariants as the proxy's proven log: +-- AUTOINCREMENT seq is a never-reused monotonic cursor, an edit updates in place +-- (seq unchanged, so it is not re-delivered), content_hash is NOT NULL because a +-- NULL would make the edit predicate (`<>`) silently never match. +CREATE TABLE feed_items ( + seq INTEGER PRIMARY KEY AUTOINCREMENT, + feed_url TEXT NOT NULL, + guid TEXT NOT NULL, + item_json TEXT NOT NULL, + published_at INTEGER, + first_seen_at INTEGER NOT NULL, + content_hash TEXT NOT NULL, + UNIQUE(feed_url, guid) +); + +CREATE INDEX idx_feed_items_feed_seq ON feed_items(feed_url, seq); + +-- The timeline join probes subscriptions by (user_did, feed_url). +CREATE INDEX IF NOT EXISTS idx_subscriptions_cache_user_feed + ON subscriptions_cache(user_did, feed_url); + +-- Generation token for the archive (D1 recreation / Time Travel restore guard). +-- Clients store it beside their cursor and cold-start on mismatch. Any D1 restore +-- must bump this (seqs rewind while the token would otherwise stay the same). +INSERT OR IGNORE INTO sync_state (key, value) +VALUES ('items_generation', lower(hex(randomblob(16)))); diff --git a/backend/src/index.ts b/backend/src/index.ts index 78bcc127..4b6eda95 100644 --- a/backend/src/index.ts +++ b/backend/src/index.ts @@ -16,6 +16,8 @@ import { handleV2Mentions, handleV2MentionLane, } from './routes/feeds-v2'; +import { handleIngest, handleCrawlSet } from './routes/ingest'; +import { handleTimeline } from './routes/timeline'; import { handleDetectContent } from './routes/social'; import { handleCreateLinkblogShare, @@ -220,10 +222,26 @@ export default { response = await handleAuthMe(request, env); break; + // Internal crawler endpoints (Fly proxy → Worker). No user session: they + // authenticate with the shared FEED_PROXY_SECRET, fail-closed when it's + // unset. Matched before the session-guarded routes below. + case url.pathname === '/api/internal/ingest': + response = await handleIngest(request, env); + break; + case url.pathname === '/api/internal/crawl-set': + response = await handleCrawlSet(request, env); + break; + + // The whole feed refresh in one request, served from D1. + case url.pathname === '/api/v2/timeline': + if (!session) return unauthorizedResponse(headers); + response = await handleTimeline(request, env, session); + break; + // Feed routes (v2 via Fly.io proxy) case url.pathname === '/api/v2/feeds/fetch': if (!session) return unauthorizedResponse(headers); - response = await handleV2FeedFetch(request, env); + response = await handleV2FeedFetch(request, env, session); break; case url.pathname === '/api/v2/feeds/batch': if (!session) return unauthorizedResponse(headers); diff --git a/backend/src/routes/feeds-v2.ts b/backend/src/routes/feeds-v2.ts index 713b69a7..2b327064 100644 --- a/backend/src/routes/feeds-v2.ts +++ b/backend/src/routes/feeds-v2.ts @@ -8,6 +8,8 @@ import type { } from '../services/feed-proxy-client'; import { resolveStandardSite } from '../utils/canonical-url'; import { getReadKeys } from './reading'; +import { ingestProxyFeed } from './ingest'; +import { readFeedMetadata, readFeedSlice } from './timeline'; interface V2FeedResponse { title: string; @@ -15,10 +17,9 @@ interface V2FeedResponse { siteUrl?: string; imageUrl?: string; items: FeedItem[]; + // Unix ms of the last ingest for this feed (freshness, for the client's UI) — + // no longer a live upstream fetch time, since reads never touch the proxy. fetchedAt: number; - cursor?: number; - generation?: string; - hasMore?: boolean; } interface V2BatchFeedResult { @@ -48,22 +49,37 @@ interface V2BatchResponse { readCursor?: number; } +// Newest-N a single-feed fetch delivers from the D1 archive. +const SINGLE_FEED_LIMIT = 30; +const SINGLE_FEED_MAX_LIMIT = 200; + /** * GET /api/v2/feeds/fetch * - * Fetch a single feed via Fly.io proxy with GUID-based incremental sync. + * One feed's newest slice, served from the D1 archive with read state joined in + * (the timeline's per-feed sibling). This is the "new subscription gap" path: + * a feed the user just subscribed to contributes nothing to the global timeline + * cursor (its items sit below it), so the client fetches it directly here. + * + * If D1 has nothing for the feed (nobody was subscribed, so the crawler never + * pushed it), we PULL THROUGH: fetch it from the proxy once, ingest the result, + * then serve from D1. Steady state never touches Fly. * * Query params: * - url: Feed URL (required) - * - since_guids: Comma-separated GUIDs the client already has (optional) - * - limit: Max items to return (optional, default 100) + * - limit: Max items to return (optional, default 30) + * - refresh: `1` to force the pull-through even when the archive already has the + * feed — the "retry this feed" action, the one path that still asks the + * crawler for a fresh fetch on demand. + * - since_guids: accepted and ignored (legacy); the client dedupes by GUID. */ -export async function handleV2FeedFetch(request: Request, env: Env): Promise { +export async function handleV2FeedFetch( + request: Request, + env: Env, + session: Session +): Promise { const url = new URL(request.url); const feedUrl = url.searchParams.get('url'); - const sinceGuidsParam = url.searchParams.get('since_guids'); - const sinceSeqParam = url.searchParams.get('since_seq'); - const generationParam = url.searchParams.get('generation'); const limitParam = url.searchParams.get('limit'); if (!feedUrl) { @@ -83,28 +99,43 @@ export async function handleV2FeedFetch(request: Request, env: Env): Promise { + const payload = `${item.title}|${item.url}|${item.content ?? ''}|${item.summary ?? ''}`; + const digest = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(payload)); + return [...new Uint8Array(digest)] + .map((b) => b.toString(16).padStart(2, '0')) + .join('') + .slice(0, 16); +} + +/** + * The write itself, shared by the pushed-batch endpoint and the subscribe-time + * pull-through in feeds-v2.ts. Idempotent: a re-pushed item with the same content + * hash is a no-op, a changed one updates in place (seq unchanged → not + * re-delivered to clients that already saw it). That makes the proxy's + * at-least-once delivery safe. + * + * Throws on a D1 write failure so callers can surface a 5xx (which leaves the + * pusher's outbox state untouched, so it retries). + */ +export async function ingestBatch( + env: Env, + feeds: IngestFeed[], + items: IngestItem[] +): Promise<{ inserted: number; updated: number }> { + const now = Math.floor(Date.now() / 1000); + const statements: D1PreparedStatement[] = []; + + for (const feed of feeds) { + if (!feed?.feedUrl) continue; + statements.push( + env.DB.prepare( + `INSERT INTO feeds (feed_url, title, site_url, description, image_url, last_ingest_at) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(feed_url) DO UPDATE SET + title = COALESCE(excluded.title, feeds.title), + site_url = COALESCE(excluded.site_url, feeds.site_url), + description = COALESCE(excluded.description, feeds.description), + image_url = COALESCE(excluded.image_url, feeds.image_url), + last_ingest_at = excluded.last_ingest_at` + ).bind( + feed.feedUrl, + feed.title ?? null, + feed.siteUrl ?? null, + feed.description ?? null, + feed.imageUrl ?? null, + now + ) + ); + } + + // Max seq before this batch: everything above it in the RETURNING rows below is + // a fresh insert, everything at or below is an edit-in-place. Purely for the + // response's counters — no correctness depends on it. + const before = await env.DB.prepare('SELECT MAX(seq) AS max_seq FROM feed_items').first<{ + max_seq: number | null; + }>(); + const maxSeqBefore = before?.max_seq ?? 0; + + const touchedFeeds = new Set(); + for (const entry of items) { + if (!entry?.feedUrl || !entry.guid || !entry.item || !entry.contentHash) continue; + touchedFeeds.add(entry.feedUrl); + statements.push( + env.DB.prepare( + `INSERT INTO feed_items (feed_url, guid, item_json, published_at, first_seen_at, content_hash) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(feed_url, guid) DO UPDATE SET + item_json = excluded.item_json, + content_hash = excluded.content_hash + WHERE feed_items.content_hash <> excluded.content_hash + RETURNING seq` + ).bind( + entry.feedUrl, + entry.guid, + JSON.stringify(capItemContent(entry.item)), + entry.publishedAt ?? null, + entry.firstSeenAt || Date.now(), + entry.contentHash + ) + ); + } + + let inserted = 0; + let updated = 0; + for (let i = 0; i < statements.length; i += INGEST_BATCH_SIZE) { + const results = await env.DB.batch<{ seq: number }>(statements.slice(i, i + INGEST_BATCH_SIZE)); + for (const result of results) { + for (const row of result.results ?? []) { + if (row.seq > maxSeqBefore) inserted++; + else updated++; + } + } + } + + await trimFeedsToSanityCap(env, [...touchedFeeds]); + + return { inserted, updated }; +} + +/** + * The ONLY pruning that happens at ingest. Under the cap the OFFSET subquery + * yields no row, so `seq <= NULL` matches nothing and the DELETE is a no-op + * costing a bounded index scan — which is the expected case for every healthy + * feed. `cap` is a parameter so tests can exercise the trim without writing + * 5,000 rows. + */ +export async function trimFeedsToSanityCap( + env: Env, + feedUrls: string[], + cap = SANITY_CAP +): Promise { + if (feedUrls.length === 0) return; + const trims = feedUrls.map((feedUrl) => + env.DB.prepare( + `DELETE FROM feed_items + WHERE feed_url = ?1 + AND seq <= (SELECT seq FROM feed_items WHERE feed_url = ?1 + ORDER BY seq DESC LIMIT 1 OFFSET ?2)` + ).bind(feedUrl, cap) + ); + for (let i = 0; i < trims.length; i += INGEST_BATCH_SIZE) { + await env.DB.batch(trims.slice(i, i + INGEST_BATCH_SIZE)); + } +} + +/** + * Ingest a feed fetched straight from the proxy (the subscribe-time pull-through + * in feeds-v2.ts). Hashes match what the crawler will push later, so the first + * real push over these rows is a no-op rather than a phantom edit. + */ +export async function ingestProxyFeed( + env: Env, + feedUrl: string, + feed: { + title?: string; + description?: string; + siteUrl?: string; + imageUrl?: string; + items: FeedItem[]; + } +): Promise { + const nowMs = Date.now(); + const items: IngestItem[] = await Promise.all( + feed.items.map(async (item) => { + const publishedMs = new Date(item.publishedAt).getTime(); + return { + feedUrl, + guid: item.guid, + item, + publishedAt: Number.isNaN(publishedMs) ? null : publishedMs, + firstSeenAt: nowMs, + contentHash: await computeContentHash(item), + }; + }) + ); + + await ingestBatch( + env, + [ + { + feedUrl, + title: feed.title ?? null, + siteUrl: feed.siteUrl ?? null, + description: feed.description ?? null, + imageUrl: feed.imageUrl ?? null, + }, + ], + items + ); +} + +/** + * POST /api/internal/ingest + * + * Upsert a batch of feed metadata + items pushed by the crawler. Any 5xx leaves + * the proxy's outbox state untouched — the idempotent upsert above makes + * at-least-once delivery safe. + */ +export async function handleIngest(request: Request, env: Env): Promise { + if (request.method !== 'POST') return badRequest('Method not allowed', 405); + if (!isAuthorizedProxyRequest(request, env)) return unauthorized(); + + const declaredLength = Number(request.headers.get('Content-Length') ?? '0'); + if (declaredLength > MAX_INGEST_BODY_BYTES) return badRequest('Payload too large', 413); + + let body: { feeds?: IngestFeed[]; items?: IngestItem[] }; + try { + body = await request.json(); + } catch { + return badRequest('Invalid JSON body'); + } + + const feeds = Array.isArray(body.feeds) ? body.feeds : []; + const items = Array.isArray(body.items) ? body.items : []; + if (items.length > MAX_INGEST_ITEMS) return badRequest('Too many items'); + + try { + const { inserted, updated } = await ingestBatch(env, feeds, items); + return new Response(JSON.stringify({ ok: true, inserted, updated }), { + headers: { 'Content-Type': 'application/json' }, + }); + } catch (error) { + console.error('[ingest] D1 WRITE ERROR:', error); + return new Response(JSON.stringify({ error: 'Ingest failed' }), { + status: 500, + headers: { 'Content-Type': 'application/json' }, + }); + } +} + +/** + * GET /api/internal/crawl-set + * + * The feeds this environment wants crawled. Replaces the proxy's request-driven + * warmth: once clients stop reading through Fly, nothing stamps + * `last_requested_at`, so every feed would silently age out of the warm loop. + * The proxy polls this and stamps the rows itself. + */ +export async function handleCrawlSet(request: Request, env: Env): Promise { + if (request.method !== 'GET') return badRequest('Method not allowed', 405); + if (!isAuthorizedProxyRequest(request, env)) return unauthorized(); + + const rows = await env.DB.prepare( + `SELECT feed_url, COUNT(*) AS subscribers + FROM subscriptions_cache + WHERE active = 1 + AND feed_url IS NOT NULL AND feed_url <> '' + AND ${rssSubscriptionPredicate()} + GROUP BY feed_url` + ).all<{ feed_url: string; subscribers: number }>(); + + const feeds = rows.results.map((row) => ({ + feedUrl: row.feed_url, + subscribers: row.subscribers, + })); + + return new Response(JSON.stringify({ feeds, count: feeds.length }), { + headers: { 'Content-Type': 'application/json' }, + }); +} diff --git a/backend/src/routes/timeline.ts b/backend/src/routes/timeline.ts new file mode 100644 index 00000000..ecd5d40a --- /dev/null +++ b/backend/src/routes/timeline.ts @@ -0,0 +1,292 @@ +import type { Env, FeedItem, Session } from '../types'; +import { rssSubscriptionPredicate } from './ingest'; + +/** + * GET /api/v2/timeline — the whole feed refresh, in one request. + * + * Replaces the `1 + ceil((N-8)/50)` batched `POST /api/v2/feeds/batch` calls + * (each a Worker → Fly hop plus `ceil(GUIDs/88)` chunked read-key queries). The + * crawler pushes items into D1; this serves them with subscriptions AND read + * state resolved in the same query, so `getReadKeys` never runs on the feed path. + * + * Named `/timeline` rather than `/sync` to avoid colliding with the existing + * PDS-subscription `/api/sync/*` routes. + */ + +// Max items in one page. Also the cap on how much a single response can weigh — +// items carry (capped) content bodies. +const MAX_LIMIT = 200; +const DEFAULT_LIMIT = 200; + +// Cold start delivers a per-feed newest slice, not a global one: a global +// `ORDER BY seq DESC LIMIT n` would let one chatty feed starve every other. +const COLD_START_PER_FEED = 30; +// Statements per D1 batch on the cold-start path. +const COLD_START_CHUNK = 25; +// A cold start touches every subscribed feed; bound the work (and log if hit — +// no silent truncation). +const COLD_START_MAX_FEEDS = 500; + +export interface TimelineItem extends FeedItem { + seq: number; + feedUrl: string; + read: boolean; +} + +interface ItemRow { + seq: number; + feed_url: string; + item_json: string; + read: number; +} + +// The per-user read probe. An EXISTS (rather than a LEFT JOIN) keeps the result +// one row per item even if a user somehow holds duplicate label rows, while +// probing the same (user_did, item_key) index a join would. +const READ_FLAG_SQL = `EXISTS ( + SELECT 1 FROM item_labels_cache il + WHERE il.user_did = ?1 AND il.item_key = fi.guid + AND il.item_type = 'article' AND il.label = 'read' AND il.deleted_at IS NULL + ) AS read`; + +function toTimelineItems(rows: ItemRow[]): TimelineItem[] { + const items: TimelineItem[] = []; + for (const row of rows) { + try { + const item = JSON.parse(row.item_json) as FeedItem; + items.push({ ...item, seq: row.seq, feedUrl: row.feed_url, read: row.read === 1 }); + } catch { + // A corrupt row must not poison the whole page; skip it. The cursor still + // advances past it, so it can't wedge the client's drain loop. + console.error(`[timeline] Unparseable item_json at seq ${row.seq}`); + } + } + return items; +} + +/** + * Newest slice of one feed, read-annotated — the single-feed read path + * (`GET /api/v2/feeds/fetch`), which now serves D1 like everything else. + */ +export async function readFeedSlice( + env: Env, + userDid: string, + feedUrl: string, + limit: number +): Promise { + const rows = await env.DB.prepare( + `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} + FROM feed_items fi + WHERE fi.feed_url = ?2 + ORDER BY fi.seq DESC + LIMIT ?3` + ) + .bind(userDid, feedUrl, limit) + .all(); + return toTimelineItems(rows.results); +} + +export interface FeedMetadataRow { + title: string | null; + site_url: string | null; + description: string | null; + image_url: string | null; + last_ingest_at: number | null; +} + +export async function readFeedMetadata(env: Env, feedUrl: string): Promise { + return env.DB.prepare( + `SELECT title, site_url, description, image_url, last_ingest_at FROM feeds WHERE feed_url = ?` + ) + .bind(feedUrl) + .first(); +} + +export async function getItemsGeneration(env: Env): Promise { + const row = await env.DB.prepare( + `SELECT value FROM sync_state WHERE key = 'items_generation'` + ).first<{ value: string }>(); + return row?.value ?? ''; +} + +async function subscribedFeedUrls(env: Env, userDid: string): Promise { + const rows = await env.DB.prepare( + `SELECT DISTINCT feed_url FROM subscriptions_cache + WHERE user_did = ? AND active = 1 + AND feed_url IS NOT NULL AND feed_url <> '' + AND ${rssSubscriptionPredicate()}` + ) + .bind(userDid) + .all<{ feed_url: string }>(); + return rows.results.map((r) => r.feed_url); +} + +/** + * Feed-level metadata for the feeds the caller subscribes to. Small (tens of + * rows) and only sent alongside a non-empty page, so a steady-state poll that + * returns nothing costs exactly one query. + */ +async function subscribedFeedMetadata( + env: Env, + userDid: string +): Promise> { + const rows = await env.DB.prepare( + `SELECT f.feed_url, f.title, f.site_url, f.image_url + FROM feeds f + JOIN (SELECT DISTINCT feed_url FROM subscriptions_cache + WHERE user_did = ? AND active = 1 + AND ${rssSubscriptionPredicate()}) sc + ON sc.feed_url = f.feed_url` + ) + .bind(userDid) + .all<{ + feed_url: string; + title: string | null; + site_url: string | null; + image_url: string | null; + }>(); + + const feeds: Record = {}; + for (const row of rows.results) { + feeds[row.feed_url] = { + title: row.title ?? undefined, + siteUrl: row.site_url ?? undefined, + imageUrl: row.image_url ?? undefined, + }; + } + return feeds; +} + +export async function handleTimeline( + request: Request, + env: Env, + session: Session +): Promise { + if (request.method !== 'GET') { + return new Response(JSON.stringify({ error: 'Method not allowed' }), { + status: 405, + headers: { 'Content-Type': 'application/json' }, + }); + } + + const url = new URL(request.url); + const sinceSeqParam = url.searchParams.get('since_seq'); + const generationParam = url.searchParams.get('generation'); + const limitParam = url.searchParams.get('limit'); + + const parsedLimit = limitParam ? parseInt(limitParam, 10) : DEFAULT_LIMIT; + const limit = Number.isInteger(parsedLimit) + ? Math.min(Math.max(parsedLimit, 1), MAX_LIMIT) + : DEFAULT_LIMIT; + + const parsedSince = sinceSeqParam !== null ? parseInt(sinceSeqParam, 10) : NaN; + const sinceSeq = Number.isInteger(parsedSince) && parsedSince >= 0 ? parsedSince : undefined; + + const generation = await getItemsGeneration(env); + // Server time (unix seconds) at annotation. The client seeds its forward + // read-delta cursor from this, exactly as /batch does today, so the delta + // starts from bootstrap with no client/server clock skew. + const readCursor = Math.floor(Date.now() / 1000); + + const incremental = sinceSeq !== undefined && generationParam === generation && generation !== ''; + + try { + if (incremental) { + // Drain oldest-unseen first so a backlog larger than one page is paged + // across polls, never skipped. limit+1 probes hasMore without a second query. + const rows = await env.DB.prepare( + `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} + FROM feed_items fi + WHERE fi.seq > ?2 + AND EXISTS ( + SELECT 1 FROM subscriptions_cache sc + WHERE sc.user_did = ?1 AND sc.feed_url = fi.feed_url AND sc.active = 1 + AND ${rssSubscriptionPredicate('sc')} + ) + ORDER BY fi.seq ASC + LIMIT ?3` + ) + .bind(session.did, sinceSeq, limit + 1) + .all(); + + const hasMore = rows.results.length > limit; + const page = hasMore ? rows.results.slice(0, limit) : rows.results; + // Cursor comes from the returned rows, never a separate MAX(seq): the + // latter races ingest and would skip everything written in between. + const cursor = page.length > 0 ? page[page.length - 1].seq : sinceSeq; + const items = toTimelineItems(page); + + return json({ + items, + cursor, + generation, + hasMore, + readCursor, + coldStart: false, + feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined, + }); + } + + // Cold start: no cursor, or a generation mismatch (D1 recreated / restored). + const allFeedUrls = await subscribedFeedUrls(env, session.did); + const feedUrls = allFeedUrls.slice(0, COLD_START_MAX_FEEDS); + if (allFeedUrls.length > feedUrls.length) { + console.warn( + `[timeline] Cold start covering ${feedUrls.length} of ${allFeedUrls.length} feeds for ${session.did}; the rest fill in as new items arrive.` + ); + } + + const rows: ItemRow[] = []; + for (let i = 0; i < feedUrls.length; i += COLD_START_CHUNK) { + const statements = feedUrls.slice(i, i + COLD_START_CHUNK).map((feedUrl) => + env.DB.prepare( + `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} + FROM feed_items fi + WHERE fi.feed_url = ?2 + ORDER BY fi.seq DESC + LIMIT ?3` + ).bind(session.did, feedUrl, COLD_START_PER_FEED) + ); + if (statements.length === 0) continue; + const results = await env.DB.batch(statements); + for (const result of results) rows.push(...(result.results ?? [])); + } + + let cursor = 0; + for (const row of rows) if (row.seq > cursor) cursor = row.seq; + if (cursor === 0) { + // Nothing to deliver (fresh account, or the feeds aren't ingested yet). + // Start the client at the current head rather than 0, so its first + // incremental poll doesn't scan the archive from the beginning. Anything + // ingested from here on is above this cursor; history for a brand-new + // subscription arrives via the single-feed endpoint's pull-through. + const head = await env.DB.prepare('SELECT MAX(seq) AS max_seq FROM feed_items').first<{ + max_seq: number | null; + }>(); + cursor = head?.max_seq ?? 0; + } + + const items = toTimelineItems(rows); + return json({ + items, + cursor, + generation, + hasMore: false, + readCursor, + coldStart: true, + feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined, + }); + } catch (error) { + console.error('[timeline] Query error:', error); + return new Response(JSON.stringify({ error: 'Failed to load timeline' }), { + status: 500, + headers: { 'Content-Type': 'application/json' }, + }); + } +} + +function json(body: unknown): Response { + return new Response(JSON.stringify(body), { + headers: { 'Content-Type': 'application/json' }, + }); +} diff --git a/backend/src/services/rate-limit.ts b/backend/src/services/rate-limit.ts index e4f75950..ed711ef5 100644 --- a/backend/src/services/rate-limit.ts +++ b/backend/src/services/rate-limit.ts @@ -54,6 +54,9 @@ const RATE_LIMITS: Record = { '/api/leaflet/subscriptions': STANDARD_LIMIT, // Light operations (cached data, simple queries) + // The whole feed refresh is one D1 query per page now, and a returning reader + // drains several pages in a burst — a light limit, not a standard one. + '/api/v2/timeline': LIGHT_LIMIT, '/api/feeds/cached': LIGHT_LIMIT, '/api/feeds/batch': LIGHT_LIMIT, '/api/feeds/status': LIGHT_LIMIT, diff --git a/backend/src/types.ts b/backend/src/types.ts index 5dbdca6d..f7959165 100644 --- a/backend/src/types.ts +++ b/backend/src/types.ts @@ -56,6 +56,9 @@ export interface FeedItem { // Stamped by the authed batch fetch handler (feeds-v2.ts) from a per-user read // join. Not a stored feed field — only present on annotated responses. read?: boolean; + // Set at ingest when `content` exceeded the stored-content cap and was dropped + // (routes/ingest.ts). The reader falls back to /extract for the full text. + contentTruncated?: boolean; } export interface ParsedFeed { diff --git a/backend/test/feed-timeline.spec.ts b/backend/test/feed-timeline.spec.ts new file mode 100644 index 00000000..06141867 --- /dev/null +++ b/backend/test/feed-timeline.spec.ts @@ -0,0 +1,412 @@ +import { env } from 'cloudflare:test'; +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { handleIngest, handleCrawlSet, trimFeedsToSanityCap } from '../src/routes/ingest'; +import { handleTimeline } from '../src/routes/timeline'; +import type { Env, FeedItem, Session } from '../src/types'; + +const TEST_DID = 'did:plc:timeline123'; +const OTHER_DID = 'did:plc:timelineother'; +const SECRET = 'test-proxy-secret'; + +const FEED_A = 'https://example.com/a.xml'; +const FEED_B = 'https://example.com/b.xml'; + +const SESSION: Session = { + did: TEST_DID, + handle: 'timeline.bsky.social', + pdsUrl: 'https://test.pds.example', + accessToken: 'token', + refreshToken: 'refresh', + dpopPrivateKey: '{}', + expiresAt: Date.now() + 3600000, +}; + +function item(guid: string, overrides: Partial = {}): FeedItem { + return { + guid, + url: `https://example.com/${guid}`, + title: `Title ${guid}`, + publishedAt: '2026-01-01T00:00:00.000Z', + ...overrides, + }; +} + +function ingestRequest( + body: unknown, + secret: string | null = SECRET, + extraHeaders: Record = {} +): Request { + const headers: Record = { + 'Content-Type': 'application/json', + ...extraHeaders, + }; + if (secret !== null) headers['X-Proxy-Secret'] = secret; + return new Request('https://api.example/api/internal/ingest', { + method: 'POST', + headers, + body: JSON.stringify(body), + }); +} + +async function ingest( + feedUrl: string, + items: Array<{ item: FeedItem; contentHash: string }> +): Promise<{ inserted: number; updated: number }> { + const res = await handleIngest( + ingestRequest({ + feeds: [{ feedUrl, title: 'Feed A' }], + items: items.map((entry) => ({ + feedUrl, + guid: entry.item.guid, + item: entry.item, + publishedAt: Date.parse(entry.item.publishedAt), + firstSeenAt: Date.now(), + contentHash: entry.contentHash, + })), + }), + env + ); + expect(res.status).toBe(200); + return (await res.json()) as { inserted: number; updated: number }; +} + +async function addSubscription( + did: string, + feedUrl: string, + opts: { active?: number; sourceType?: string | null } = {} +) { + await env.DB.prepare( + `INSERT INTO subscriptions_cache (user_did, record_uri, feed_url, title, source_type, active) + VALUES (?, ?, ?, ?, ?, ?)` + ) + .bind( + did, + `at://${did}/app.skyreader.feed.subscription/${Math.random().toString(36).slice(2)}`, + feedUrl, + 'Feed', + opts.sourceType ?? null, + opts.active ?? 1 + ) + .run(); +} + +async function timeline(params: Record = {}) { + const search = new URLSearchParams(params).toString(); + const res = await handleTimeline( + new Request(`https://api.example/api/v2/timeline${search ? `?${search}` : ''}`), + env, + SESSION + ); + expect(res.status).toBe(200); + return (await res.json()) as { + items: Array<{ seq: number; feedUrl: string; guid: string; read: boolean; content?: string }>; + cursor: number; + generation: string; + hasMore: boolean; + readCursor: number; + coldStart: boolean; + feeds?: Record; + }; +} + +describe('feed timeline (D1 ingest + serve)', () => { + let savedSecret: string | undefined; + + beforeEach(async () => { + savedSecret = (env as Env).FEED_PROXY_SECRET; + (env as Env).FEED_PROXY_SECRET = SECRET; + for (const did of [TEST_DID, OTHER_DID]) { + await env.DB.prepare( + `INSERT OR IGNORE INTO users (did, handle, pds_url, tier, created_at) + VALUES (?, ?, 'https://test.pds.example', 'free', unixepoch())` + ) + .bind(did, `${did}.test`) + .run(); + } + }); + + afterEach(async () => { + (env as Env).FEED_PROXY_SECRET = savedSecret as string; + await env.DB.prepare('DELETE FROM feed_items').run(); + await env.DB.prepare('DELETE FROM feeds').run(); + await env.DB.prepare('DELETE FROM subscriptions_cache').run(); + await env.DB.prepare('DELETE FROM item_labels_cache').run(); + }); + + describe('ingest auth', () => { + it('rejects a request with no secret', async () => { + const res = await handleIngest(ingestRequest({ feeds: [], items: [] }, null), env); + expect(res.status).toBe(401); + }); + + it('rejects a mismatched secret', async () => { + const res = await handleIngest(ingestRequest({ feeds: [], items: [] }, 'nope'), env); + expect(res.status).toBe(401); + }); + + it('fails closed when the server secret is unset', async () => { + (env as Env).FEED_PROXY_SECRET = '' as string; + const res = await handleIngest(ingestRequest({ feeds: [], items: [] }, SECRET), env); + expect(res.status).toBe(401); + }); + + it('rejects an oversized body by Content-Length', async () => { + const res = await handleIngest( + ingestRequest({ feeds: [], items: [] }, SECRET, { + 'Content-Length': String(64 * 1024 * 1024), + }), + env + ); + expect(res.status).toBe(413); + }); + }); + + describe('ingest writes', () => { + it('inserts items and upserts feed metadata', async () => { + const result = await ingest(FEED_A, [ + { item: item('g1'), contentHash: 'h1' }, + { item: item('g2'), contentHash: 'h2' }, + ]); + expect(result.inserted).toBe(2); + expect(result.updated).toBe(0); + + const feed = await env.DB.prepare('SELECT * FROM feeds WHERE feed_url = ?') + .bind(FEED_A) + .first<{ title: string; last_ingest_at: number }>(); + expect(feed?.title).toBe('Feed A'); + expect(feed?.last_ingest_at).toBeGreaterThan(0); + }); + + it('is idempotent: re-pushing the same batch creates no duplicates', async () => { + await ingest(FEED_A, [{ item: item('g1'), contentHash: 'h1' }]); + const before = await env.DB.prepare('SELECT seq FROM feed_items WHERE guid = ?') + .bind('g1') + .first<{ seq: number }>(); + + const second = await ingest(FEED_A, [{ item: item('g1'), contentHash: 'h1' }]); + expect(second.inserted).toBe(0); + expect(second.updated).toBe(0); + + const rows = await env.DB.prepare('SELECT seq FROM feed_items WHERE guid = ?') + .bind('g1') + .all<{ seq: number }>(); + expect(rows.results.length).toBe(1); + expect(rows.results[0].seq).toBe(before?.seq); + }); + + it('edits in place: a changed hash rewrites the row without a new seq', async () => { + await ingest(FEED_A, [{ item: item('g1', { title: 'Old' }), contentHash: 'h1' }]); + const before = await env.DB.prepare('SELECT seq FROM feed_items WHERE guid = ?') + .bind('g1') + .first<{ seq: number }>(); + + const result = await ingest(FEED_A, [ + { item: item('g1', { title: 'New' }), contentHash: 'h2' }, + ]); + expect(result.updated).toBe(1); + expect(result.inserted).toBe(0); + + const after = await env.DB.prepare('SELECT seq, item_json FROM feed_items WHERE guid = ?') + .bind('g1') + .first<{ seq: number; item_json: string }>(); + expect(after?.seq).toBe(before?.seq); + expect(JSON.parse(after!.item_json).title).toBe('New'); + }); + + it('never prunes: a feed accumulates past the proxy’s 200-item window', async () => { + // Five pushes of 50 distinct items each — well past the proxy's K = 200 + // outbox window. D1 is the archive; ordinary ingest deletes nothing. + for (let batch = 0; batch < 5; batch++) { + await ingest( + FEED_A, + Array.from({ length: 50 }, (_, i) => ({ + item: item(`b${batch}-i${i}`), + contentHash: `h${batch}-${i}`, + })) + ); + } + const count = await env.DB.prepare('SELECT COUNT(*) AS c FROM feed_items WHERE feed_url = ?') + .bind(FEED_A) + .first<{ c: number }>(); + expect(count?.c).toBe(250); + }); + + it('sanity cap trims oldest-first, and only above the cap', async () => { + await ingest( + FEED_A, + Array.from({ length: 5 }, (_, i) => ({ item: item(`s${i}`), contentHash: `h${i}` })) + ); + + // Under the cap: a no-op. + await trimFeedsToSanityCap(env, [FEED_A], 10); + let rows = await env.DB.prepare('SELECT guid FROM feed_items WHERE feed_url = ? ORDER BY seq') + .bind(FEED_A) + .all<{ guid: string }>(); + expect(rows.results.length).toBe(5); + + // Above the cap: the oldest go, newest survive. + await trimFeedsToSanityCap(env, [FEED_A], 3); + rows = await env.DB.prepare('SELECT guid FROM feed_items WHERE feed_url = ? ORDER BY seq') + .bind(FEED_A) + .all<{ guid: string }>(); + expect(rows.results.map((r) => r.guid)).toEqual(['s2', 's3', 's4']); + }); + + it('caps stored content and marks it truncated, leaving small items alone', async () => { + const big = 'x'.repeat(9000); + await ingest(FEED_A, [ + { item: item('big', { content: big, summary: 'kept' }), contentHash: 'hbig' }, + { item: item('small', { content: 'tiny' }), contentHash: 'hsmall' }, + ]); + + const bigRow = await env.DB.prepare('SELECT item_json FROM feed_items WHERE guid = ?') + .bind('big') + .first<{ item_json: string }>(); + const parsedBig = JSON.parse(bigRow!.item_json); + expect(parsedBig.content).toBeUndefined(); + expect(parsedBig.contentTruncated).toBe(true); + expect(parsedBig.summary).toBe('kept'); + + const smallRow = await env.DB.prepare('SELECT item_json FROM feed_items WHERE guid = ?') + .bind('small') + .first<{ item_json: string }>(); + const parsedSmall = JSON.parse(smallRow!.item_json); + expect(parsedSmall.content).toBe('tiny'); + expect(parsedSmall.contentTruncated).toBeUndefined(); + }); + }); + + describe('crawl set', () => { + it('requires the shared secret', async () => { + const res = await handleCrawlSet( + new Request('https://api.example/api/internal/crawl-set'), + env + ); + expect(res.status).toBe(401); + }); + + it('returns active RSS feeds with subscriber counts', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(OTHER_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B, { active: 0 }); + await addSubscription(TEST_DID, 'at://did:plc:x/pub', { sourceType: 'atproto.documents' }); + + const res = await handleCrawlSet( + new Request('https://api.example/api/internal/crawl-set', { + headers: { 'X-Proxy-Secret': SECRET }, + }), + env + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { + feeds: Array<{ feedUrl: string; subscribers: number }>; + }; + expect(body.feeds).toEqual([{ feedUrl: FEED_A, subscribers: 2 }]); + }); + }); + + describe('timeline serving', () => { + it('cold-starts with a per-feed newest slice and inline read flags', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B); + await ingest(FEED_A, [ + { item: item('a1'), contentHash: 'h1' }, + { item: item('a2'), contentHash: 'h2' }, + ]); + await ingest(FEED_B, [{ item: item('b1'), contentHash: 'h3' }]); + + await env.DB.prepare( + `INSERT INTO item_labels_cache (user_did, item_key, item_type, label, created_at, updated_at) + VALUES (?, 'a1', 'article', 'read', unixepoch(), unixepoch())` + ) + .bind(TEST_DID) + .run(); + + const body = await timeline(); + expect(body.coldStart).toBe(true); + expect(body.items.map((i) => i.guid).sort()).toEqual(['a1', 'a2', 'b1']); + const byGuid = Object.fromEntries(body.items.map((i) => [i.guid, i.read])); + expect(byGuid['a1']).toBe(true); + expect(byGuid['a2']).toBe(false); + expect(body.cursor).toBeGreaterThan(0); + expect(body.hasMore).toBe(false); + expect(typeof body.readCursor).toBe('number'); + expect(body.feeds?.[FEED_A]?.title).toBe('Feed A'); + }); + + it('excludes parked, unsubscribed and atproto sources', async () => { + await addSubscription(TEST_DID, FEED_A, { active: 0 }); + await addSubscription(OTHER_DID, FEED_B); + await ingest(FEED_A, [{ item: item('a1'), contentHash: 'h1' }]); + await ingest(FEED_B, [{ item: item('b1'), contentHash: 'h2' }]); + + const cold = await timeline(); + expect(cold.items).toEqual([]); + + // ...and the incremental path applies the same filter. + const incremental = await timeline({ since_seq: '0', generation: cold.generation }); + expect(incremental.items).toEqual([]); + }); + + it('drains incrementally from the cursor, paging with hasMore', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [ + { item: item('i1'), contentHash: 'h1' }, + { item: item('i2'), contentHash: 'h2' }, + { item: item('i3'), contentHash: 'h3' }, + ]); + + const generation = (await timeline()).generation; + const first = await timeline({ since_seq: '0', generation, limit: '2' }); + expect(first.coldStart).toBe(false); + expect(first.items.map((i) => i.guid)).toEqual(['i1', 'i2']); + expect(first.hasMore).toBe(true); + + const second = await timeline({ + since_seq: String(first.cursor), + generation, + limit: '2', + }); + expect(second.items.map((i) => i.guid)).toEqual(['i3']); + expect(second.hasMore).toBe(false); + + // Steady state: nothing new, cursor held, one empty page. + const third = await timeline({ since_seq: String(second.cursor), generation, limit: '2' }); + expect(third.items).toEqual([]); + expect(third.cursor).toBe(second.cursor); + expect(third.hasMore).toBe(false); + expect(third.feeds).toBeUndefined(); + }); + + it('does not re-deliver an edited item (seq unchanged)', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('e1', { title: 'Old' }), contentHash: 'h1' }]); + const cold = await timeline(); + + await ingest(FEED_A, [{ item: item('e1', { title: 'New' }), contentHash: 'h2' }]); + const after = await timeline({ + since_seq: String(cold.cursor), + generation: cold.generation, + }); + expect(after.items).toEqual([]); + }); + + it('cold-starts again on a generation mismatch', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('g1'), contentHash: 'h1' }]); + + const body = await timeline({ since_seq: '999999', generation: 'stale-generation' }); + expect(body.coldStart).toBe(true); + expect(body.items.map((i) => i.guid)).toEqual(['g1']); + }); + + it('starts an empty account at the archive head, not at zero', async () => { + await addSubscription(TEST_DID, FEED_B); + await ingest(FEED_A, [{ item: item('other'), contentHash: 'h1' }]); + + const body = await timeline(); + expect(body.items).toEqual([]); + expect(body.cursor).toBeGreaterThan(0); + }); + }); +}); diff --git a/backend/wrangler.toml b/backend/wrangler.toml index bbf275f4..3243036f 100644 --- a/backend/wrangler.toml +++ b/backend/wrangler.toml @@ -69,7 +69,10 @@ vars = { FRONTEND_URL = "http://127.0.0.1:5173" } # Staging environment [env.staging] name = "skyreader-api-staging" -vars = { FRONTEND_URL = "https://staging.skyreader.app", ALLOWED_ORIGINS = "https://staging.skyreader.app,https://staging-linkblogs.skyreader.app", FEED_PROXY_URL = "https://skyreader-feed-proxy.fly.dev", LINKBLOG_PUBLIC_URL = "https://staging-linkblogs.skyreader.app" } +# Staging talks to its OWN Fly proxy (skyreader-feed-proxy-staging), which pushes +# only into the staging D1. Prod and staging are two fully isolated pairs — no +# cross-links, distinct PROXY_SECRET/FEED_PROXY_SECRET values. +vars = { FRONTEND_URL = "https://staging.skyreader.app", ALLOWED_ORIGINS = "https://staging.skyreader.app,https://staging-linkblogs.skyreader.app", FEED_PROXY_URL = "https://skyreader-feed-proxy-staging.fly.dev", LINKBLOG_PUBLIC_URL = "https://staging-linkblogs.skyreader.app" } routes = [{ pattern = "api-staging.skyreader.app", custom_domain = true }] [env.staging.durable_objects] diff --git a/docs/plans/D1_FEED_TIMELINE.md b/docs/plans/D1_FEED_TIMELINE.md new file mode 100644 index 00000000..e1dcd7e2 --- /dev/null +++ b/docs/plans/D1_FEED_TIMELINE.md @@ -0,0 +1,141 @@ +# D1-Served Feed Timeline (ingest-push architecture) + +> Supersedes, in part, `RETENTION_SYNC_PLAN.md`. That plan made the Fly proxy a durable item +> log with a monotonic cursor and it stays exactly as built — but its **serving** role moves +> to D1. The proxy is now the **crawler and outbox**; D1 is the **archive and read path**. + +## What changed + +A refresh used to be `1 + ceil((N−8)/50)` sequential `POST /api/v2/feeds/batch` calls, each +hopping Worker → Fly proxy and then paying `ceil(GUIDs/88)` sequential D1 queries to annotate +read state (`getReadKeys`). Now: + +``` + ┌────────────── crawl set (pull, every ~5 min) ──────────────┐ + ▼ │ + Fly proxy (crawler only) Cloudflare Worker + D1 (serving store) + ┌──────────────────────────┐ push deltas ┌──────────────────────────────────────┐ + │ warm loop / etag / │ ──────────────▶ │ POST /api/internal/ingest │ + │ backoff / UA fallback │ (at-least-once,│ upsert feeds + feed_items │ + │ feed_items = outbox │ seq order, │ (archive: no pruning; sanity cap) │ + │ (bounded, K = 200) │ ONE target) │ │ + └──────────────────────────┘ │ GET /api/v2/timeline?since_seq=&… │ + │ ONE query: feed_items ⋈ subs ⋈ read│ + Client ◀───────────────── one paged request ─┴──────────────────────────────────────┘ + + (× 2: one such pair for prod, one for staging — no cross-links) +``` + +- **Reads never touch Fly.** If the Fly box is down, refresh still works from D1 (freshness + stalls, serving doesn't). +- **D1 is the archive; the proxy log is a window.** The proxy keeps its K = 200 recent items per + feed as a delivery buffer; D1 keeps everything it has ever ingested. Prod D1 is therefore the + system of record for feed history — its Time Travel / export story matters. +- The proxy keeps `/extract`, `/discover`, `/documents`, `/social-context`, `/mentions`, and the + single-feed `/feed` (now only for subscribe-time pull-through). + +## Shipped in this change + +**Backend** (`backend/`) + +- Migration `0061_feed_timeline.sql`: drops the dormant pre-Fly trio (`feed_items`, `feed_cache`, + `feed_metadata` — zero references in `src/`), creates `feeds` + `feed_items` (AUTOINCREMENT + `seq`, `UNIQUE(feed_url, guid)`, `content_hash NOT NULL`), the `(user_did, feed_url)` + subscription index, and mints `sync_state.items_generation`. +- `routes/ingest.ts`: `POST /api/internal/ingest` and `GET /api/internal/crawl-set`, both + authenticated by a constant-time compare against `FEED_PROXY_SECRET` and **fail-closed** when + it is unset. Idempotent upsert (edit-in-place keeps the seq), per-item content cap + (`MAX_ITEM_CONTENT_BYTES = 8 KB`, drops `content` and sets `contentTruncated`), and the per-feed + `SANITY_CAP = 5000` trim — the only pruning that ever runs. +- `routes/timeline.ts`: `GET /api/v2/timeline?since_seq=&generation=&limit=` — incremental drain + (cursor derived from returned rows, `hasMore` via `limit+1`) and a per-feed newest-30 cold start. + Read state is an `EXISTS` probe in the same query; `getReadKeys` is never called on the feed path. +- `GET /api/v2/feeds/fetch` re-backed with D1 + **pull-through**: if a feed isn't in the archive + yet (first subscriber), fetch it from the proxy once, ingest it, then serve. + +**Proxy** (`feed-proxy/`) + +- `ingest-push.ts`: the durable log *is* the outbox. `push_state(seq, pushed_hash)` marks what has + been delivered; a row is dirty when it's missing there or the hashes differ (race-free against a + mid-flight edit). Seq-ordered batches of 100 every 15 s, capped exponential backoff on failure. +- Crawl-set pull every 5 min: registers each feed's `cache` row and stamps `last_requested_at`, so + the existing warm loop / active window / eviction machinery keeps working now that read traffic + no longer stamps anything. +- `INGEST_URL` unset ⇒ both loops disabled. `push_state` cascades on the K = 200 cap trim, the + redelivery delete, and `cleanupCache` eviction. `/stats` reports `ingest.{items,pushed,pending}`. + +**Frontend** (`frontend/`) + +- `feedFetcher.fetchAllFeeds` prefers one `GET /api/v2/timeline` (plus drain pages) with a single + global cursor in Dexie `metadata` (`timelineCursor`). Pure helpers live in `timelineSync.ts`. +- Falls back to the legacy `/api/v2/feeds/batch` path when the timeline 404s (old or rolled-back + backend) **or** when a cold start returns nothing for a subscribed user (the environment's + crawler isn't pushing yet). The cursor is never committed in that case. +- OPML import now backfills via the per-feed endpoint: a freshly imported feed's items sit below + the global cursor, so only the single-feed path (with its pull-through) can deliver them. + +**Admin** (`admin/`) — feed health is re-pointed at `feeds`/`feed_items`: crawled feeds, subscribed +feeds not ingesting (the R1 alarm), archived item count, estimated archive size with a 6 GB alert, +and a churn detector for feeds nearing the sanity cap. + +## Operator steps (not code) + +### Phase 0 — measure (before enabling prod ingest) + +Snapshot prod so the growth math becomes a projection: proxy `feed_items` count and `item_json` +size distribution (drives backfill time and confirms the 8 KB content cap), per-feed new-item +velocity over a sample window (items/day, and it flags existing GUID-churn feeds), and current +refresh request counts from logs (the baseline the win is measured against). + +### Phase 3 — provision the staging Fly proxy, then soak + +```bash +# 1. Create the app (same org as skyreader-feed-proxy) +fly apps create skyreader-feed-proxy-staging + +# 2. Create its volume (same region as primary_region; 1 GB is ample for staging) +fly volumes create proxy_data -a skyreader-feed-proxy-staging -r sjc -s 1 + +# 3. Mint a DISTINCT staging secret (don't reuse prod's — smaller blast radius) +openssl rand -hex 32 # → +fly secrets set PROXY_SECRET= -a skyreader-feed-proxy-staging + +# 4. Set the same value on the staging Worker (it is both the outbound proxy auth +# and the inbound ingest auth) +cd backend && npx wrangler secret put FEED_PROXY_SECRET --env staging + +# 5. First deploy (CI takes over afterwards, on every push to main) +cd feed-proxy && fly deploy --remote-only --config fly.staging.toml +``` + +Sequencing matters: today's staging Worker authenticates to the **prod** proxy with prod's secret, +so provision + deploy the staging proxy first, then flip `FEED_PROXY_URL` (already committed) and +rotate `FEED_PROXY_SECRET` together in one staging Worker deploy. + +Verify: `fly status -a skyreader-feed-proxy-staging` shows exactly **one** machine (the singleton +invariant applies to this app too — never `fly scale count`); `/stats` answers and its +`ingest.pending` trends to ~0; staging D1 `feed_items` counts rise monotonically; the prod proxy's +logs show no staging-origin traffic. + +**Then enable prod:** uncomment `INGEST_URL` in `feed-proxy/fly.toml` and cut a release. Prod's +backfill drains through the normal pusher loop (≤ 200 × active feeds at 100 items / 15 s). Until +that happens, prod clients keep using the legacy batch path automatically — the frontend's +empty-archive fallback covers exactly this window. + +### Phase 5 — cleanup (a later release, once no legacy traffic remains) + +Remove `/api/v2/feeds/batch`'s proxy passthrough + its `getReadKeys` call (keep it for documents), +the proxy's `POST /feeds` batch read endpoint, the `since_guids`/`since_seq` client plumbing, Dexie +`feedCursors`, `liveDb.getRecentGuids`, and `fetchAllFeedsViaBatch`. Optionally add an hourly cron +step deleting `feeds`/`feed_items` rows whose feed has had **zero active subscribers for > 90 days** +— the one deliberate deletion path, and skippable if even orphans should stay. + +## Invariants worth keeping + +- **Cursor from returned rows, never `MAX(seq)`** — the latter races ingest and silently skips rows. +- **Any D1 restore bumps `items_generation`** (one `UPDATE sync_state`): Time Travel rewinds seqs + while the token would otherwise stay the same, so clients would sit above the head forever. +- **The pusher sends `cache.url`**, the registered URL, never a post-redirect one: the timeline + joins on that exact string. +- **Ordinary ingest deletes nothing.** A feed at the sanity cap is a bug signal (GUID churn), not + steady state — investigate the feed rather than letting it rotate. diff --git a/docs/plans/RETENTION_SYNC_PLAN.md b/docs/plans/RETENTION_SYNC_PLAN.md index a867484b..d961f86c 100644 --- a/docs/plans/RETENTION_SYNC_PLAN.md +++ b/docs/plans/RETENTION_SYNC_PLAN.md @@ -1,5 +1,11 @@ # Durable Item Retention: Catch Everything Since Last Visit +> **Superseded in part by `D1_FEED_TIMELINE.md`.** Everything below shipped and still runs — but +> the durable log's *serving* role has moved to D1. The proxy log is now the crawler's **outbox** +> (a bounded K = 200 delivery window pushed into D1); the **archive** clients read from is +> `feed_items` in D1, which never prunes, and the cursor clients hold is a single global one +> against that table rather than one per feed. + > Supersedes `BATCH_CURSOR_PLAN.md`. That plan optimized the *request payload* (500 GUIDs → > a cursor) but kept the proxy's replace-the-blob storage model, which structurally cannot > retain items beyond the source feed's live window. This plan changes the storage model so diff --git a/e2e/seed.ts b/e2e/seed.ts index 8f2f77cc..dac8afbd 100644 --- a/e2e/seed.ts +++ b/e2e/seed.ts @@ -136,6 +136,55 @@ export async function seedSavedArticle( return rkey; } +export interface SeedFeedItemOpts { + guid: string; + title: string; + url?: string; + publishedAt?: string; + summary?: string; +} + +/** + * Seed the server-side archive the timeline serves from: a `feeds` row plus its + * `feed_items`. Feed-scoped rather than user-scoped (the archive is shared by + * every subscriber), so it's cleaned up with `cleanupFeedItems`. + */ +export async function seedFeedItems( + feedUrl: string, + items: SeedFeedItemOpts[], + opts: { title?: string; siteUrl?: string } = {} +): Promise { + const nowMs = Date.now(); + const nowSeconds = Math.floor(nowMs / 1000); + + const statements = [ + `INSERT OR REPLACE INTO feeds (feed_url, title, site_url, last_ingest_at, created_at) VALUES (${sqlString(feedUrl)}, ${sqlNullableString(opts.title ?? null)}, ${sqlNullableString(opts.siteUrl ?? null)}, ${nowSeconds}, ${nowSeconds})`, + ]; + + items.forEach((item, index) => { + const publishedAt = item.publishedAt ?? new Date(nowMs - index * 60_000).toISOString(); + const itemJson = JSON.stringify({ + guid: item.guid, + url: item.url ?? `https://example.com/${item.guid}`, + title: item.title, + summary: item.summary ?? `Summary for ${item.title}`, + publishedAt, + }); + statements.push( + `INSERT OR REPLACE INTO feed_items (feed_url, guid, item_json, published_at, first_seen_at, content_hash) VALUES (${sqlString(feedUrl)}, ${sqlString(item.guid)}, ${sqlString(itemJson)}, ${Date.parse(publishedAt)}, ${nowMs}, ${sqlString(`hash-${item.guid}`)})` + ); + }); + + await execD1(statements); +} + +export async function cleanupFeedItems(feedUrl: string): Promise { + await execD1([ + `DELETE FROM feed_items WHERE feed_url = ${sqlString(feedUrl)}`, + `DELETE FROM feeds WHERE feed_url = ${sqlString(feedUrl)}`, + ]); +} + export interface SeedItemLabelOpts { itemKey: string; itemType: string; diff --git a/e2e/timeline.spec.ts b/e2e/timeline.spec.ts new file mode 100644 index 00000000..22252b53 --- /dev/null +++ b/e2e/timeline.spec.ts @@ -0,0 +1,76 @@ +import { test, expect } from './fixtures'; +import { seedSubscription, seedFeedItems, seedItemLabel, cleanupFeedItems } from './seed'; + +/** + * The D1-served timeline: a refresh is ONE request that already carries read + * state, instead of a per-feed batch fan-out against the Fly proxy. + */ +test.describe('Timeline refresh', () => { + const FEED_URL = 'https://example.com/timeline-feed.xml'; + const FEED_TITLE = 'Timeline Feed'; + + test.afterEach(async () => { + await cleanupFeedItems(FEED_URL); + }); + + test('renders archived items from a single timeline request, with read state', async ({ + authedPage, + testUser, + }) => { + await seedSubscription(testUser, { feedUrl: FEED_URL, title: FEED_TITLE }); + await seedFeedItems( + FEED_URL, + [ + { guid: 'timeline-item-1', title: 'Archived Article One' }, + { guid: 'timeline-item-2', title: 'Archived Article Two' }, + ], + { title: FEED_TITLE, siteUrl: 'https://example.com' } + ); + // Read on another device: the timeline join must stamp it inline. + await seedItemLabel(testUser, { + itemKey: 'timeline-item-1', + itemType: 'article', + label: 'read', + }); + + const timelineRequests: string[] = []; + const batchRequests: string[] = []; + authedPage.on('request', (request) => { + const url = request.url(); + if (url.includes('/api/v2/timeline')) timelineRequests.push(url); + if (url.includes('/api/v2/feeds/batch')) batchRequests.push(url); + }); + + await authedPage.reload(); + + // Both articles land in the reader. + await expect(authedPage.getByText('Archived Article One')).toBeVisible({ timeout: 20_000 }); + await expect(authedPage.getByText('Archived Article Two')).toBeVisible({ timeout: 20_000 }); + + // The refresh is a single timeline call (at most one extra drain page), and + // the legacy per-feed fan-out never runs — the point of the architecture. + expect(timelineRequests.length).toBeGreaterThanOrEqual(1); + expect(timelineRequests.length).toBeLessThanOrEqual(2); + expect(batchRequests.length).toBe(0); + + // Read state arrived with the articles (no separate read fetch). + const readGuids = await authedPage.evaluate(async () => { + const request = indexedDB.open('skyreader'); + const database: IDBDatabase = await new Promise((resolve, reject) => { + request.onsuccess = () => resolve(request.result); + request.onerror = () => reject(request.error); + }); + const store = database.transaction('itemLabels', 'readonly').objectStore('itemLabels'); + const all: Array<{ itemKey: string; label: string }> = await new Promise( + (resolve, reject) => { + const req = store.getAll(); + req.onsuccess = () => resolve(req.result); + req.onerror = () => reject(req.error); + } + ); + return all.filter((row) => row.label === 'read').map((row) => row.itemKey); + }); + expect(readGuids).toContain('timeline-item-1'); + expect(readGuids).not.toContain('timeline-item-2'); + }); +}); diff --git a/feed-proxy/README.md b/feed-proxy/README.md index f3716ec6..bcdc00ab 100644 --- a/feed-proxy/README.md +++ b/feed-proxy/README.md @@ -288,13 +288,34 @@ curl "/feed?url=...&since_guids=old-guid&limit=50" ## Configuration -| Environment Variable | Default | Description | -| -------------------- | -------- | ----------------------------------------- | -| `PROXY_SECRET` | (none) | Shared secret for `X-Proxy-Secret` header | -| `DATA_DIR` | `./data` | SQLite database location | -| `CACHE_TTL_SECONDS` | `900` | Fresh cache duration (15 min) | -| `STALE_TTL_SECONDS` | `3600` | Stale cache max age (1 hour) | -| `PORT` | `3000` | HTTP server port | +| Environment Variable | Default | Description | +| ---------------------------- | -------- | ------------------------------------------------------- | +| `PROXY_SECRET` | (none) | Shared secret for `X-Proxy-Secret` (inbound + outbound) | +| `DATA_DIR` | `./data` | SQLite database location | +| `CACHE_TTL_SECONDS` | `900` | Fresh cache duration (15 min) | +| `STALE_TTL_SECONDS` | `3600` | Stale cache max age (1 hour) | +| `PORT` | `3000` | HTTP server port | +| `INGEST_URL` | (none) | Paired Worker base URL. **Unset ⇒ ingest disabled** | +| `INGEST_INTERVAL_SECONDS` | `15` | Push cycle | +| `INGEST_BATCH_SIZE` | `100` | Items per push request | +| `CRAWL_SET_INTERVAL_SECONDS` | `300` | How often to pull the crawl set | + +## Ingest push (crawler mode) + +With `INGEST_URL` set, this proxy stops being a read path for the reader and becomes the crawler +for exactly ONE Worker + D1 pair (prod proxy → prod Worker, staging proxy → staging Worker): + +- **Push:** the durable item log _is_ the outbox. `push_state(seq, pushed_hash)` records what has + reached D1; a row is dirty when it's missing there or its `content_hash` has changed since. Dirty + rows drain in seq order to `POST {INGEST_URL}/api/internal/ingest`; delivery is at-least-once and + the Worker's upsert is idempotent. Failures back off and retry the same rows. +- **Pull:** every `CRAWL_SET_INTERVAL_SECONDS` the proxy fetches + `GET {INGEST_URL}/api/internal/crawl-set` and stamps `last_requested_at` on each feed, which is + what keeps the warm loop working now that reads no longer touch this box. +- The per-feed cap (`FEED_ITEMS_CAP = 200`) bounds the **outbox**, not the archive: D1 retains + everything it has ingested. `push_state` cascades with every `feed_items` delete. + +See `docs/plans/D1_FEED_TIMELINE.md`. ## Cache Behavior @@ -388,6 +409,11 @@ The `/stats` endpoint includes error statistics: ### Fly.io +Two apps, one per environment — `skyreader-feed-proxy` (prod, `fly.toml`) and +`skyreader-feed-proxy-staging` (staging, `fly.staging.toml`). Each pushes into its own Worker's D1 +and holds its own `PROXY_SECRET`; keep the two config files in sync apart from `app` and +`INGEST_URL`. Each app runs exactly ONE machine (see the singleton invariant in `fly.toml`). + ```bash # Create app fly apps create skyreader-feed-proxy @@ -395,7 +421,7 @@ fly apps create skyreader-feed-proxy # Set secret fly secrets set PROXY_SECRET=your-secret-here -# Deploy +# Deploy (staging: add --config fly.staging.toml) fly deploy ``` diff --git a/feed-proxy/fly.staging.toml b/feed-proxy/fly.staging.toml new file mode 100644 index 00000000..c234a55d --- /dev/null +++ b/feed-proxy/fly.staging.toml @@ -0,0 +1,55 @@ +app = "skyreader-feed-proxy-staging" +primary_region = "sjc" + +# KEEP IN SYNC WITH fly.toml. These two files must differ only in `app` and the +# `INGEST_URL` in [env]; everything else (mounts, http_service, vm, tuning) is +# deliberately identical so staging is a faithful soak of what prod will run. +# Staging deploys on every push to main, prod on release — so drift shows up here +# first, before a release ships it. +# +# SINGLETON INVARIANT — run exactly ONE machine. This app is not horizontally +# scalable as written: the cache is a SQLite DB on the per-machine `proxy_data` +# volume (volumes can't be shared between machines), and the Jetstream firehose, +# the self-warming loop, the ingest pusher, and the in-memory request-coalescing +# maps are all in-process singletons. A second machine would get its own empty +# volume + its own firehose + its own warm loop, with no shared coalescing — i.e. +# a split-brain cache, duplicate upstream fetches, duplicate pushes, and responses +# that differ by which machine you hit. Scale VERTICALLY via the [vm] block below, +# never by `fly scale count`. Keep only one machine provisioned; auto_start/stop +# below toggle that one machine, they do not create new ones. + +[build] + +[mounts] + source = "proxy_data" + destination = "/data" + +[http_service] + internal_port = 3000 + force_https = true + auto_stop_machines = true + auto_start_machines = true + # Keep one machine alive: the self-warming loop and the ingest pusher run + # in-process on timers and can only work while a machine is up. + min_machines_running = 1 + +[vm] + cpu_kind = "shared" + cpus = 2 + memory_mb = 1024 + +[env] + DATA_DIR = "/data" + # This proxy pushes its item log into the STAGING Worker's D1 and pulls the + # staging crawl set from it. One proxy, one Worker — never both environments. + INGEST_URL = "https://api-staging.skyreader.app" + CACHE_TTL_SECONDS = "300" + STALE_TTL_SECONDS = "3600" + WARM_ENABLED = "true" + WARM_INTERVAL_SECONDS = "60" + WARM_ACTIVE_WINDOW_SECONDS = "1209600" + WARM_BATCH_CAP = "500" + WARM_CONCURRENCY = "16" + WARM_MENTIONS = "false" + EXTRACT_CONCURRENCY = "4" + EXTRACT_QUEUE_MAX = "20" diff --git a/feed-proxy/fly.toml b/feed-proxy/fly.toml index 8db17795..f3cc6663 100644 --- a/feed-proxy/fly.toml +++ b/feed-proxy/fly.toml @@ -1,6 +1,10 @@ app = "skyreader-feed-proxy" primary_region = "sjc" +# KEEP IN SYNC WITH fly.staging.toml. The two files must differ only in `app` and +# the `INGEST_URL` in [env]; everything else is deliberately identical so staging +# soaks what prod will run. +# # SINGLETON INVARIANT — run exactly ONE machine. This app is not horizontally # scalable as written: the cache is a SQLite DB on the per-machine `proxy_data` # volume (volumes can't be shared between machines), and the Jetstream firehose, @@ -35,6 +39,11 @@ primary_region = "sjc" [env] DATA_DIR = "/data" + # Ingest push into the prod Worker's D1. Commented out until the staging pair + # has soaked (see docs/plans/D1_FEED_TIMELINE.md, Phase 3): with it unset this + # machine crawls exactly as before and pushes nothing. Enabling it is this one + # line plus a release — prod's backfill then drains through the normal pusher. + # INGEST_URL = "https://api.skyreader.app" # Fresh window. The self-warming loop refreshes active feeds before they reach # this age, so user requests stay HITs even at a tight 5-minute freshness. CACHE_TTL_SECONDS = "300" diff --git a/feed-proxy/src/app.ts b/feed-proxy/src/app.ts index 01eaa468..bca4d4e2 100644 --- a/feed-proxy/src/app.ts +++ b/feed-proxy/src/app.ts @@ -57,6 +57,9 @@ export interface AppConfig { // Defaults to "unhealthy / nothing subscribed" so behavior is unchanged when // the firehose isn't wired in. getFirehoseStatus?: () => FirehoseStatus; + // Whether this proxy pushes its item log into a paired Worker's D1 (INGEST_URL + // is set). Reported by /stats; the loops themselves live in index.ts. + ingestEnabled?: boolean; } export interface CacheRow { @@ -591,12 +594,20 @@ export function writeFeedItems( // delete) instead of an implicit commit per statement — a single fsync rather // than ~100 every warm-refresh per feed. Single-writer SQLite makes it safe. const deleteForRedelivery = db.query('DELETE FROM feed_items WHERE url_hash = ? AND guid = ?'); + // Cascade for the two deletes below. push_state is keyed by seq, so a seq that + // disappears from feed_items would otherwise leave an orphan row forever. + const dropPushStateForItem = db.query( + 'DELETE FROM push_state WHERE seq IN (SELECT seq FROM feed_items WHERE url_hash = ? AND guid = ?)' + ); const writeBatch = db.transaction(() => { if (redeliver) { // Keep retained history that has aged out of the source feed, but remove // every item in this parse so it receives a fresh seq and reaches clients // that already consumed its pre-upgrade representation. - for (const item of items) deleteForRedelivery.run(urlHash, item.guid); + for (const item of items) { + dropPushStateForItem.run(urlHash, item.guid); + deleteForRedelivery.run(urlHash, item.guid); + } } for (let i = items.length - 1; i >= 0; i--) { @@ -615,6 +626,20 @@ export function writeFeedItems( // Cheaper-than-anti-join cap: compute the (K+1)-th-newest seq once, range-delete // below it. The OFFSET subquery yields nothing when the feed has <= K rows, so // `seq <= NULL` matches nothing — a no-op. + // + // NOTE: this cap bounds the *outbox*, not the archive. Items trimmed here that + // already reached D1 stay there permanently (D1 does not prune); items trimmed + // before ever being pushed simply drop out of the dirty set — the same K-window + // bound the proxy has always had. + db.run( + `DELETE FROM push_state + WHERE seq IN ( + SELECT seq FROM feed_items + WHERE url_hash = ? + AND seq <= (SELECT seq FROM feed_items WHERE url_hash = ? ORDER BY seq DESC LIMIT 1 OFFSET ?) + )`, + [urlHash, urlHash, FEED_ITEMS_CAP] + ); db.run( `DELETE FROM feed_items WHERE url_hash = ? @@ -819,6 +844,18 @@ export function initDatabase(db: Database): void { db.run(`CREATE INDEX IF NOT EXISTS idx_feed_items_feed_seq ON feed_items(url_hash, seq)`); db.run(`CREATE INDEX IF NOT EXISTS idx_feed_items_first_seen ON feed_items(first_seen_at)`); + // Ingest delivery state (ingest-push.ts): which seqs, at which content hash, + // have reached the paired Worker's D1. A row is dirty when it's missing here or + // the hashes differ — race-free against a concurrent edit. Created + // unconditionally, even with pushing disabled, so the cascade deletes that keep + // this table bounded (writeFeedItems, cleanupCache) always have a table to hit. + db.run(` + CREATE TABLE IF NOT EXISTS push_state ( + seq INTEGER PRIMARY KEY, + pushed_hash TEXT NOT NULL + ) + `); + // Extracted article content (Defuddle output), keyed by source URL. Separate // from the feed cache: article content is effectively immutable per URL, so it // has its own long TTL and is not touched by the feed self-warming loop. @@ -1799,11 +1836,37 @@ export function createApp(db: Database, config: AppConfig) { >('SELECT COUNT(*) as count FROM cache WHERE next_retry_at > ? AND error_count >= 5') .get(now + 6 * 24 * 60 * 60 * 1000); // More than 6 days means it's a permanent error + // Ingest push (see ingest-push.ts). `pending` is the outbox backlog — the + // number that should hover near zero once a push cycle keeps up. + const itemCount = db + .query<{ count: number }, []>('SELECT COUNT(*) as count FROM feed_items') + .get(); + const pushedCount = db + .query<{ count: number }, []>('SELECT COUNT(*) as count FROM push_state') + .get(); + // Inlined rather than imported from ingest-push.ts, which imports this + // module (hashUrl/itemContentHash) — keep the dependency one-directional. + const pendingCount = db + .query<{ count: number }, []>( + `SELECT COUNT(*) AS count + FROM feed_items fi + JOIN cache c ON c.url_hash = fi.url_hash + LEFT JOIN push_state ps ON ps.seq = fi.seq + WHERE ps.seq IS NULL OR ps.pushed_hash <> COALESCE(fi.content_hash, '')` + ) + .get(); + return c.json({ total: total?.count || 0, fresh: fresh?.count || 0, stale: stale?.count || 0, inFlight: inFlight.size, + ingest: { + enabled: config.ingestEnabled ?? false, + items: itemCount?.count || 0, + pushed: pushedCount?.count || 0, + pending: pendingCount?.count || 0, + }, extract: { inUse: extractSemaphore.inUse, queued: extractSemaphore.queued }, cacheTtlSeconds: cacheTtlMs / 1000, staleTtlSeconds: staleTtlMs / 1000, @@ -2680,6 +2743,13 @@ export function cleanupCache(db: Database): number { // cache row goes (the delete keys off cache.fetched_at). Once the feed is cold // the retained items are unreachable anyway, and this bounds feed_items growth // to the active working set. + db.run( + `DELETE FROM push_state WHERE seq IN ( + SELECT seq FROM feed_items + WHERE url_hash IN (SELECT url_hash FROM cache WHERE fetched_at < ?) + )`, + [threshold] + ); db.run( 'DELETE FROM feed_items WHERE url_hash IN (SELECT url_hash FROM cache WHERE fetched_at < ?)', [threshold] diff --git a/feed-proxy/src/index.ts b/feed-proxy/src/index.ts index a1874c99..a283fee5 100644 --- a/feed-proxy/src/index.ts +++ b/feed-proxy/src/index.ts @@ -5,6 +5,7 @@ import { Database } from 'bun:sqlite'; import { mkdirSync } from 'fs'; import { createApp, initDatabase, cleanupCache } from './app'; import { DocumentFirehose } from './jetstream'; +import { pushDirtyItems, pullCrawlSet, type IngestConfig } from './ingest-push'; // Config const PROXY_SECRET = process.env.PROXY_SECRET; @@ -53,6 +54,24 @@ const WARM_MENTIONS_ENABLED = (process.env.WARM_MENTIONS ?? 'true') !== 'false'; const EXTRACT_CONCURRENCY = parseInt(process.env.EXTRACT_CONCURRENCY || '4', 10); const EXTRACT_QUEUE_MAX = parseInt(process.env.EXTRACT_QUEUE_MAX || '20', 10); +// Ingest push (ingest-push.ts): this proxy is the crawler for exactly one +// Worker + D1 pair. INGEST_URL unset ⇒ both the pusher and the crawl-set pull are +// disabled — the safe default for local dev and the gate for a staged rollout. +const INGEST_URL = process.env.INGEST_URL?.replace(/\/$/, '') || ''; +const INGEST_ENABLED = INGEST_URL.length > 0; +const INGEST_INTERVAL_MS = parseInt(process.env.INGEST_INTERVAL_SECONDS || '15', 10) * 1000; +const INGEST_BATCH_SIZE = parseInt(process.env.INGEST_BATCH_SIZE || '100', 10); +const CRAWL_SET_INTERVAL_MS = parseInt(process.env.CRAWL_SET_INTERVAL_SECONDS || '300', 10) * 1000; +// Backoff for a failing push. Shaped for a 15 s loop (the feed fetcher's +// 5-minute base would stall ingest for ten minutes over one blip), capped so a +// long Worker outage still retries a few times an hour. +const PUSH_BACKOFF_BASE_MS = 30 * 1000; +const PUSH_BACKOFF_MAX_MS = 10 * 60 * 1000; + +function pushBackoff(failures: number): number { + return Math.min(PUSH_BACKOFF_BASE_MS * 2 ** (failures - 1), PUSH_BACKOFF_MAX_MS); +} + // Jetstream document firehose: keeps standard.site documents fresh via the AT // Proto firehose (push) instead of re-listing every active author (pull). The // pull path stays for cold-start backfill and as the firehose-down fallback. @@ -92,6 +111,7 @@ const { app, warmStaleFeeds, warmStaleDocuments } = createApp(db, { extractConcurrency: EXTRACT_CONCURRENCY, extractQueueMax: EXTRACT_QUEUE_MAX, getFirehoseStatus: () => firehose?.status() ?? { healthy: false, isSubscribed: () => false }, + ingestEnabled: INGEST_ENABLED, }); // Document firehose: push-based freshness for standard.site documents. @@ -150,6 +170,84 @@ if (WARM_ENABLED) { console.log('[Proxy] Warmer: disabled'); } +// Ingest loops: push the durable item log into the paired Worker's D1, and pull +// the crawl set back so registered feeds stay warm now that read traffic no +// longer stamps them. Backfill is not a special case — on first enable every row +// is dirty and drains through the same loop. +if (INGEST_ENABLED) { + const ingestConfig: IngestConfig = { + ingestUrl: INGEST_URL, + secret: PROXY_SECRET, + batchSize: INGEST_BATCH_SIZE, + }; + + console.log( + `[Proxy] Ingest push: → ${INGEST_URL}, every ${INGEST_INTERVAL_MS / 1000}s, ` + + `batch ${INGEST_BATCH_SIZE}; crawl set every ${CRAWL_SET_INTERVAL_MS / 1000}s` + ); + + let pushRunning = false; + let pushFailures = 0; + let pushBlockedUntil = 0; + setInterval(() => { + // Skip-if-running (same guard as the warm loop) + capped exponential backoff + // on failure, so a wedged Worker doesn't turn into a hot retry loop. + if (pushRunning || Date.now() < pushBlockedUntil) return; + pushRunning = true; + pushDirtyItems(db, ingestConfig) + .then((result) => { + if (result.error) { + pushFailures++; + const delay = pushBackoff(pushFailures); + pushBlockedUntil = Date.now() + delay; + console.error( + `[Proxy] Ingest push failed (${pushFailures}): ${result.error}; ` + + `retrying after ${Math.round(delay / 1000)}s` + ); + return; + } + pushFailures = 0; + if (result.pushed > 0) { + console.log(`[Proxy] Ingest pushed ${result.pushed} item(s)`); + } + }) + .catch((err) => { + console.error('[Proxy] Ingest push error:', err); + Sentry.captureException(err, { tags: { source: 'ingest-push' } }); + }) + .finally(() => { + pushRunning = false; + }); + }, INGEST_INTERVAL_MS); + + let crawlSetRunning = false; + const refreshCrawlSet = () => { + if (crawlSetRunning) return; + crawlSetRunning = true; + pullCrawlSet(db, ingestConfig) + .then((result) => { + if (result.error) { + console.error(`[Proxy] Crawl-set pull failed: ${result.error}`); + } else { + console.log(`[Proxy] Crawl set: ${result.registered} feed(s) registered`); + } + }) + .catch((err) => { + console.error('[Proxy] Crawl-set pull error:', err); + Sentry.captureException(err, { tags: { source: 'crawl-set' } }); + }) + .finally(() => { + crawlSetRunning = false; + }); + }; + // Pull once at boot so a restarted (or brand-new) machine has its crawl set + // before the first warm tick, then on the interval. + refreshCrawlSet(); + setInterval(refreshCrawlSet, CRAWL_SET_INTERVAL_MS); +} else { + console.log('[Proxy] Ingest push: disabled (INGEST_URL unset)'); +} + // Flush the firehose cursor + close its socket cleanly on shutdown so we resume // where we left off instead of replaying. for (const signal of ['SIGTERM', 'SIGINT'] as const) { diff --git a/feed-proxy/src/ingest-push.test.ts b/feed-proxy/src/ingest-push.test.ts new file mode 100644 index 00000000..a72a227b --- /dev/null +++ b/feed-proxy/src/ingest-push.test.ts @@ -0,0 +1,335 @@ +import { describe, expect, it, beforeEach, afterEach, spyOn } from 'bun:test'; +import { Database } from 'bun:sqlite'; +import { + initDatabase, + hashUrl, + writeFeedItems, + cleanupCache, + itemContentHash, + FEED_ITEMS_CAP, +} from './app'; +import { + pushDirtyItems, + pullCrawlSet, + registerCrawlFeeds, + selectDirtyRows, + countDirtyRows, + type IngestConfig, +} from './ingest-push'; +import type { FeedItem } from './types'; + +const FEED_URL = 'https://example.com/feed.xml'; +const URL_HASH = hashUrl(FEED_URL); + +const CONFIG: IngestConfig = { + ingestUrl: 'https://api.example', + secret: 'test-secret', + batchSize: 2, +}; + +function item(guid: string, overrides: Partial = {}): FeedItem { + return { + guid, + url: `https://example.com/${guid}`, + title: `Title ${guid}`, + publishedAt: '2026-01-01T00:00:00.000Z', + ...overrides, + }; +} + +function seedCache(db: Database, url = FEED_URL, feedTitle = 'Test Blog'): void { + db.run( + `INSERT INTO cache (url_hash, url, parsed_json, parser_version, cached_at, fetched_at, last_requested_at) + VALUES (?, ?, ?, 1, ?, ?, ?)`, + [ + hashUrl(url), + url, + JSON.stringify({ + title: feedTitle, + siteUrl: 'https://example.com', + description: 'A test blog', + items: [], + }), + Date.now(), + Date.now(), + Date.now(), + ] + ); +} + +interface CapturedRequest { + url: string; + headers: Record; + body: { + feeds: Array<{ feedUrl: string; title?: string | null; siteUrl?: string | null }>; + items: Array<{ feedUrl: string; guid: string; contentHash: string; item: FeedItem }>; + }; +} + +// Capture what the pusher sends and control the Worker's reply. +function mockIngestEndpoint(status = 200): { calls: CapturedRequest[]; restore: () => void } { + const calls: CapturedRequest[] = []; + const spy = spyOn(globalThis, 'fetch').mockImplementation((async ( + input: string, + init?: RequestInit + ) => { + calls.push({ + url: String(input), + headers: (init?.headers ?? {}) as Record, + body: init?.body ? JSON.parse(String(init.body)) : { feeds: [], items: [] }, + }); + return new Response(JSON.stringify({ ok: status === 200 }), { status }); + }) as unknown as typeof fetch); + return { calls, restore: () => spy.mockRestore() }; +} + +describe('ingest push', () => { + let db: Database; + + beforeEach(() => { + db = new Database(':memory:'); + initDatabase(db); + seedCache(db); + }); + + afterEach(() => { + db.close(); + }); + + it('pushes dirty rows in seq order and acks them', async () => { + writeFeedItems(db, URL_HASH, [item('g2'), item('g1')], Date.now()); + expect(countDirtyRows(db)).toBe(2); + + const endpoint = mockIngestEndpoint(); + const result = await pushDirtyItems(db, CONFIG); + endpoint.restore(); + + expect(result.pushed).toBe(2); + expect(endpoint.calls.length).toBe(1); + expect(endpoint.calls[0].url).toBe('https://api.example/api/internal/ingest'); + expect(endpoint.calls[0].headers['X-Proxy-Secret']).toBe('test-secret'); + // Items are newest-first in the feed, written oldest-first, so seq order is g1, g2. + expect(endpoint.calls[0].body.items.map((i) => i.guid)).toEqual(['g1', 'g2']); + // Always the registered URL, never a post-redirect one. + expect(endpoint.calls[0].body.items.every((i) => i.feedUrl === FEED_URL)).toBe(true); + expect(endpoint.calls[0].body.feeds[0]).toMatchObject({ + feedUrl: FEED_URL, + title: 'Test Blog', + siteUrl: 'https://example.com', + }); + + expect(countDirtyRows(db)).toBe(0); + }); + + it('pages a backlog larger than one batch', async () => { + writeFeedItems(db, URL_HASH, [item('g3'), item('g2'), item('g1')], Date.now()); + + const endpoint = mockIngestEndpoint(); + const first = await pushDirtyItems(db, CONFIG); + expect(first.pushed).toBe(2); + expect(first.hasMore).toBe(true); + + const second = await pushDirtyItems(db, CONFIG); + endpoint.restore(); + expect(second.pushed).toBe(1); + expect(second.hasMore).toBe(false); + expect(countDirtyRows(db)).toBe(0); + }); + + it('leaves state untouched on a 5xx so the same rows retry', async () => { + writeFeedItems(db, URL_HASH, [item('g1')], Date.now()); + + const failing = mockIngestEndpoint(500); + const result = await pushDirtyItems(db, CONFIG); + failing.restore(); + expect(result.pushed).toBe(0); + expect(result.error).toContain('500'); + expect(countDirtyRows(db)).toBe(1); + + const ok = mockIngestEndpoint(); + const retry = await pushDirtyItems(db, CONFIG); + ok.restore(); + expect(retry.pushed).toBe(1); + expect(countDirtyRows(db)).toBe(0); + }); + + it('re-qualifies an item edited after it was pushed', async () => { + writeFeedItems(db, URL_HASH, [item('g1', { title: 'Old' })], Date.now()); + const first = mockIngestEndpoint(); + await pushDirtyItems(db, CONFIG); + first.restore(); + expect(countDirtyRows(db)).toBe(0); + + // An edit rewrites content_hash in place (seq unchanged) → dirty again. + writeFeedItems(db, URL_HASH, [item('g1', { title: 'New' })], Date.now()); + expect(countDirtyRows(db)).toBe(1); + + const second = mockIngestEndpoint(); + const result = await pushDirtyItems(db, CONFIG); + second.restore(); + expect(result.pushed).toBe(1); + expect(second.calls[0].body.items[0].item.title).toBe('New'); + }); + + it('acks the hash it pushed, so an edit mid-flight is not lost', async () => { + writeFeedItems(db, URL_HASH, [item('g1', { title: 'Old' })], Date.now()); + const rows = selectDirtyRows(db, 10); + expect(rows.length).toBe(1); + + const endpoint = spyOn(globalThis, 'fetch').mockImplementation((async () => { + // The warm loop rewrites the item while the request is in flight. + writeFeedItems(db, URL_HASH, [item('g1', { title: 'Newer' })], Date.now()); + return new Response('{}', { status: 200 }); + }) as unknown as typeof fetch); + await pushDirtyItems(db, CONFIG); + endpoint.mockRestore(); + + // The acked hash is the one we sent, which no longer matches — still dirty. + expect(countDirtyRows(db)).toBe(1); + }); + + it('supplies a hash for legacy rows that have none', async () => { + const parsed = item('legacy'); + db.run( + `INSERT INTO feed_items (url_hash, guid, item_json, published_at, first_seen_at, content_hash) + VALUES (?, ?, ?, ?, ?, NULL)`, + [URL_HASH, 'legacy', JSON.stringify(parsed), Date.now(), Date.now()] + ); + + const endpoint = mockIngestEndpoint(); + await pushDirtyItems(db, CONFIG); + endpoint.restore(); + + expect(endpoint.calls[0].body.items[0].contentHash).toBe(itemContentHash(parsed)); + // Acked as '' (matching COALESCE), so it doesn't re-push every cycle. + expect(countDirtyRows(db)).toBe(0); + }); + + it('cascades push_state when the per-feed cap trims items', async () => { + // Fill past the cap so the oldest rows are trimmed on the next write. + const initial = Array.from({ length: FEED_ITEMS_CAP }, (_, i) => item(`old-${i}`)); + writeFeedItems(db, URL_HASH, initial, Date.now()); + const endpoint = mockIngestEndpoint(); + let guard = 0; + while (countDirtyRows(db) > 0 && guard++ < 200) { + await pushDirtyItems(db, { ...CONFIG, batchSize: 100 }); + } + endpoint.restore(); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c).toBe( + FEED_ITEMS_CAP + ); + + writeFeedItems(db, URL_HASH, [item('fresh-1'), item('fresh-2')], Date.now()); + + const itemCount = db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM feed_items').get()?.c; + const stateCount = db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c; + expect(itemCount).toBe(FEED_ITEMS_CAP); + // Two trimmed rows dropped their delivery state with them (2 fresh rows are + // not yet pushed, so push_state holds cap - 2). + expect(stateCount).toBe(FEED_ITEMS_CAP - 2); + }); + + it('cascades push_state when an idle feed is evicted', async () => { + writeFeedItems(db, URL_HASH, [item('g1')], Date.now()); + const endpoint = mockIngestEndpoint(); + await pushDirtyItems(db, CONFIG); + endpoint.restore(); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c).toBe(1); + + // Age the feed past the 7-day eviction window. + db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [ + Date.now() - 8 * 24 * 60 * 60 * 1000, + URL_HASH, + ]); + cleanupCache(db); + + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM feed_items').get()?.c).toBe(0); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c).toBe(0); + }); +}); + +describe('crawl-set registration', () => { + let db: Database; + + beforeEach(() => { + db = new Database(':memory:'); + initDatabase(db); + }); + + afterEach(() => { + db.close(); + }); + + it('creates a cache row for an unknown feed, due for the warm loop', () => { + const now = Date.now(); + expect(registerCrawlFeeds(db, [FEED_URL], now)).toBe(1); + + const row = db + .query< + { url: string; fetched_at: number; parser_version: number; last_requested_at: number }, + [string] + >('SELECT url, fetched_at, parser_version, last_requested_at FROM cache WHERE url_hash = ?') + .get(URL_HASH); + expect(row?.url).toBe(FEED_URL); + expect(row?.last_requested_at).toBe(now); + // fetched_at 0 + a stale parser version make it immediately warm-eligible. + expect(row?.fetched_at).toBe(0); + expect(row?.parser_version).toBe(0); + }); + + it('keeps a known feed warm without any read traffic', () => { + seedCache(db); + db.run('UPDATE cache SET last_requested_at = ? WHERE url_hash = ?', [1000, URL_HASH]); + + const now = Date.now(); + registerCrawlFeeds(db, [FEED_URL], now); + + const row = db + .query< + { last_requested_at: number; parsed_json: string; parser_version: number }, + [string] + >('SELECT last_requested_at, parsed_json, parser_version FROM cache WHERE url_hash = ?') + .get(URL_HASH); + expect(row?.last_requested_at).toBe(now); + // The existing cached parse is untouched. + expect(row?.parser_version).toBe(1); + expect(JSON.parse(row!.parsed_json).title).toBe('Test Blog'); + }); + + it('pulls the crawl set from the paired Worker', async () => { + const spy = spyOn(globalThis, 'fetch').mockImplementation((async ( + input: string, + init?: RequestInit + ) => { + expect(String(input)).toBe('https://api.example/api/internal/crawl-set'); + expect((init?.headers as Record)['X-Proxy-Secret']).toBe('test-secret'); + return new Response( + JSON.stringify({ + feeds: [ + { feedUrl: FEED_URL, subscribers: 3 }, + { feedUrl: 'https://other.example/f.xml' }, + ], + }) + ); + }) as unknown as typeof fetch); + + const result = await pullCrawlSet(db, CONFIG); + spy.mockRestore(); + + expect(result.registered).toBe(2); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM cache').get()?.c).toBe(2); + }); + + it('reports a failed pull without touching the cache', async () => { + const spy = spyOn(globalThis, 'fetch').mockImplementation((async () => { + return new Response('nope', { status: 503 }); + }) as unknown as typeof fetch); + + const result = await pullCrawlSet(db, CONFIG); + spy.mockRestore(); + + expect(result.registered).toBe(0); + expect(result.error).toContain('503'); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM cache').get()?.c).toBe(0); + }); +}); diff --git a/feed-proxy/src/ingest-push.ts b/feed-proxy/src/ingest-push.ts new file mode 100644 index 00000000..a0293d8c --- /dev/null +++ b/feed-proxy/src/ingest-push.ts @@ -0,0 +1,242 @@ +import type { Database } from 'bun:sqlite'; +import type { FeedItem } from './types'; +import { hashUrl, itemContentHash } from './app'; + +/** + * Ingest push: the proxy stops being a read path and becomes a crawler that + * pushes deltas into its paired Worker's D1. + * + * The durable item log IS the outbox — there is no separate queue. A row is + * *dirty* when `push_state` has no row for its seq, or the hash we last pushed + * differs from the row's current `content_hash`. That comparison is race-free + * against concurrent edits: if `writeFeedItems` rewrites an item between select + * and ack, the hashes no longer match and the row simply re-qualifies. + * + * One proxy pushes to exactly ONE Worker (prod proxy → prod Worker, staging + * proxy → staging Worker), so delivery state is keyed by seq alone. With + * `INGEST_URL` unset both loops are disabled — the safe default for local dev + * and the gate for a staged rollout. + */ + +export interface IngestConfig { + // Base URL of the paired Worker, e.g. https://api.skyreader.app + ingestUrl: string; + // Shared secret, sent as X-Proxy-Secret (the same PROXY_SECRET the Worker uses + // to authenticate to us — one secret per environment, no new names). + secret?: string; + // Items per push request. The Worker's own cap is well above this. + batchSize?: number; + timeoutMs?: number; +} + +const DEFAULT_BATCH_SIZE = 100; +const DEFAULT_TIMEOUT_MS = 30_000; + +interface DirtyRow { + seq: number; + url_hash: string; + guid: string; + item_json: string; + published_at: number | null; + first_seen_at: number; + content_hash: string; + feed_url: string; +} + +interface FeedMetaRow { + url_hash: string; + url: string; + title: string | null; + site_url: string | null; + description: string | null; + image_url: string | null; +} + +export interface PushResult { + pushed: number; + // True when the log still holds dirty rows beyond this batch. + hasMore: boolean; + error?: string; +} + +export function selectDirtyRows(db: Database, limit: number): DirtyRow[] { + return db + .query( + `SELECT fi.seq, fi.url_hash, fi.guid, fi.item_json, fi.published_at, fi.first_seen_at, + COALESCE(fi.content_hash, '') AS content_hash, c.url AS feed_url + FROM feed_items fi + JOIN cache c ON c.url_hash = fi.url_hash + LEFT JOIN push_state ps ON ps.seq = fi.seq + WHERE ps.seq IS NULL OR ps.pushed_hash <> COALESCE(fi.content_hash, '') + ORDER BY fi.seq ASC + LIMIT ?` + ) + .all(limit); +} + +export function countDirtyRows(db: Database): number { + return ( + db + .query<{ count: number }, []>( + `SELECT COUNT(*) AS count + FROM feed_items fi + JOIN cache c ON c.url_hash = fi.url_hash + LEFT JOIN push_state ps ON ps.seq = fi.seq + WHERE ps.seq IS NULL OR ps.pushed_hash <> COALESCE(fi.content_hash, '')` + ) + .get()?.count ?? 0 + ); +} + +function feedMetadata(db: Database, urlHashes: string[]): FeedMetaRow[] { + if (urlHashes.length === 0) return []; + const placeholders = urlHashes.map(() => '?').join(','); + return db + .query( + `SELECT url_hash, url, + json_extract(parsed_json, '$.title') AS title, + json_extract(parsed_json, '$.siteUrl') AS site_url, + json_extract(parsed_json, '$.description') AS description, + json_extract(parsed_json, '$.imageUrl') AS image_url + FROM cache + WHERE url_hash IN (${placeholders})` + ) + .all(...urlHashes); +} + +function ackPushed(db: Database, rows: DirtyRow[]): void { + const upsert = db.query( + `INSERT INTO push_state (seq, pushed_hash) VALUES (?, ?) + ON CONFLICT(seq) DO UPDATE SET pushed_hash = excluded.pushed_hash` + ); + // Ack the hash we actually PUSHED, not the row's hash right now: if the item + // was edited mid-flight the two differ and the row stays dirty for the next + // cycle, which is exactly what we want. + db.transaction(() => { + for (const row of rows) upsert.run(row.seq, row.content_hash); + })(); +} + +/** + * Drain one batch of dirty rows to the paired Worker. Rows go in seq order, so + * within-feed order in D1 matches proxy first-seen order — all the cursor + * semantics need. On any failure nothing is acked and the same rows retry. + */ +export async function pushDirtyItems(db: Database, config: IngestConfig): Promise { + const batchSize = config.batchSize ?? DEFAULT_BATCH_SIZE; + const rows = selectDirtyRows(db, batchSize); + if (rows.length === 0) return { pushed: 0, hasMore: false }; + + const urlHashes = [...new Set(rows.map((r) => r.url_hash))]; + const feeds = feedMetadata(db, urlHashes).map((meta) => ({ + feedUrl: meta.url, + title: meta.title, + siteUrl: meta.site_url, + description: meta.description, + imageUrl: meta.image_url, + })); + + const items = rows.map((row) => { + const item = JSON.parse(row.item_json) as FeedItem; + return { + // ALWAYS the registered/requested URL (cache.url), never a post-redirect + // one: D1 joins subscriptions on this exact string. + feedUrl: row.feed_url, + guid: row.guid, + item, + publishedAt: row.published_at, + firstSeenAt: row.first_seen_at, + // A pre-hash legacy row still needs a hash for D1's NOT NULL column. + contentHash: row.content_hash || itemContentHash(item), + }; + }); + + const headers: Record = { 'Content-Type': 'application/json' }; + if (config.secret) headers['X-Proxy-Secret'] = config.secret; + + try { + const response = await fetch(`${config.ingestUrl}/api/internal/ingest`, { + method: 'POST', + headers, + body: JSON.stringify({ feeds, items }), + signal: AbortSignal.timeout(config.timeoutMs ?? DEFAULT_TIMEOUT_MS), + }); + + if (!response.ok) { + const body = await response.text().catch(() => ''); + return { + pushed: 0, + hasMore: true, + error: `HTTP ${response.status}${body ? `: ${body.slice(0, 200)}` : ''}`, + }; + } + + ackPushed(db, rows); + return { pushed: rows.length, hasMore: rows.length >= batchSize }; + } catch (error) { + return { + pushed: 0, + hasMore: true, + error: error instanceof Error ? error.message : String(error), + }; + } +} + +/** + * Register a feed in the crawl set: create its cache row if missing and stamp + * `last_requested_at`. That single stamp is the whole trick — the existing warm + * loop, active window and eviction machinery then work unchanged, with the + * Worker's subscription table (rather than read traffic) driving warmth. + * + * A newly created row starts with parser_version 0 and fetched_at 0 so the warm + * loop treats it as due immediately. + */ +export function registerCrawlFeeds(db: Database, feedUrls: string[], now: number): number { + if (feedUrls.length === 0) return 0; + const upsert = db.query( + `INSERT INTO cache (url_hash, url, parsed_json, parser_version, parser_upgrade_attempted_version, + cached_at, fetched_at, error_count, last_requested_at) + VALUES (?, ?, '{"title":"","items":[],"fetchedAt":0}', 0, 0, 0, 0, 0, ?) + ON CONFLICT(url_hash) DO UPDATE SET last_requested_at = excluded.last_requested_at` + ); + let registered = 0; + db.transaction(() => { + for (const url of feedUrls) { + if (!url) continue; + upsert.run(hashUrl(url), url, now); + registered++; + } + })(); + return registered; +} + +export interface CrawlSetResult { + registered: number; + error?: string; +} + +/** + * Pull the crawl set from the paired Worker and stamp every feed in it. + */ +export async function pullCrawlSet(db: Database, config: IngestConfig): Promise { + const headers: Record = {}; + if (config.secret) headers['X-Proxy-Secret'] = config.secret; + + try { + const response = await fetch(`${config.ingestUrl}/api/internal/crawl-set`, { + headers, + signal: AbortSignal.timeout(config.timeoutMs ?? DEFAULT_TIMEOUT_MS), + }); + if (!response.ok) { + return { registered: 0, error: `HTTP ${response.status}` }; + } + const body = (await response.json()) as { feeds?: Array<{ feedUrl: string }> }; + const urls = (body.feeds ?? []).map((f) => f.feedUrl).filter(Boolean); + return { registered: registerCrawlFeeds(db, urls, Date.now()) }; + } catch (error) { + return { + registered: 0, + error: error instanceof Error ? error.message : String(error), + }; + } +} diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 3d58f489..2d9cd820 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -41,9 +41,23 @@ All stores use Svelte 5 runes (`.svelte.ts` files): | Service | Purpose | | --------------- | -------------------------------------------------- | -| `api.ts` | HTTP client for backend API | -| `db.ts` | Dexie (IndexedDB) schema for offline storage | -| `sync-queue.ts` | Queue operations when offline, process when online | +| `api.ts` | HTTP client for backend API | +| `db.ts` | Dexie (IndexedDB) schema for offline storage | +| `feedFetcher.ts` | Feed refresh (timeline sync + legacy batch path) | +| `timelineSync.ts`| Pure helpers for the timeline sync (unit-tested) | +| `sync-queue.ts` | Queue operations when offline, process when online | + +### Feed refresh + +A refresh is **one** `GET /api/v2/timeline` request (plus drain pages while `hasMore`), served from +the backend's D1 archive with read state already stamped on each item. The client holds a single +global cursor in Dexie `metadata` (`timelineCursor` = `{cursor, generation}`), committed only after +a successful merge; a `generation` change cold-starts. New subscriptions are backfilled through +`fetchSingleFeed` → `GET /api/v2/feeds/fetch`, since their items sit below the global cursor. + +The legacy per-feed `/api/v2/feeds/batch` path (`fetchAllFeedsViaBatch`, with per-subscription +`feedCursors`) is kept for one release as a fallback: it runs when the timeline 404s or when a cold +start returns nothing for a subscribed user. See `docs/plans/D1_FEED_TIMELINE.md`. | `realtime.ts` | WebSocket connection management | ### Key Routes diff --git a/frontend/src/lib/components/ImportOPMLModal.svelte b/frontend/src/lib/components/ImportOPMLModal.svelte index 6054335a..60ab44bc 100644 --- a/frontend/src/lib/components/ImportOPMLModal.svelte +++ b/frontend/src/lib/components/ImportOPMLModal.svelte @@ -4,7 +4,7 @@ import { articlesStore } from '$lib/stores/articles.svelte'; import { auth } from '$lib/stores/auth.svelte'; import { liveDb } from '$lib/services/liveDb.svelte'; - import { fetchAllFeeds } from '$lib/services/feedFetcher'; + import { fetchNewSubscriptionFeeds } from '$lib/services/feedFetcher'; import Modal from '$lib/components/common/Modal.svelte'; interface Props { @@ -125,8 +125,10 @@ // Get the newly added subscriptions const newSubs = liveDb.subscriptions.filter((s) => result.added.includes(s.id!)); - // Fetch feeds in background - fetchAllFeeds(newSubs, articlesStore.savedGuids); + // Fetch each new feed directly in the background. A freshly imported feed's + // items sit below the global timeline cursor, so the per-feed endpoint (not + // the timeline) is what backfills its history. + fetchNewSubscriptionFeeds(newSubs, articlesStore.savedGuids); } } diff --git a/frontend/src/lib/services/api.ts b/frontend/src/lib/services/api.ts index 4ceb8329..f71eb325 100644 --- a/frontend/src/lib/services/api.ts +++ b/frontend/src/lib/services/api.ts @@ -53,6 +53,20 @@ export class UrlSaveLimitError extends Error { } } +// A non-2xx the client may want to branch on by status (the feed path uses it to +// detect a backend that predates /api/v2/timeline and fall back). `message` is +// unchanged from the generic error path, so existing `e.message` handling still +// works. +export class ApiError extends Error { + status: number; + + constructor(message: string, status: number) { + super(message); + this.name = 'ApiError'; + this.status = status; + } +} + export class OfflineError extends Error { constructor() { super('You are offline'); @@ -71,6 +85,23 @@ export class SessionRefreshError extends Error { } } +// One page of GET /api/v2/timeline. Items carry their archive `seq` and the feed +// they belong to; `cursor`/`generation` are stored and echoed on the next poll. +export interface TimelineResponse { + items: Array; + cursor: number; + generation: string; + hasMore: boolean; + // Server time (unix seconds) at annotation — seeds the forward read delta. + readCursor?: number; + // True when the server served a per-feed newest slice instead of draining from + // a cursor (no cursor sent, or the generation no longer matches). + coldStart: boolean; + // Feed-level metadata for the caller's subscriptions; present only on a page + // that carried items. + feeds?: Record; +} + export interface ExtractedArticle { title: string | null; author: string | null; @@ -237,7 +268,10 @@ class ApiClient { } const error = await response.json().catch(() => ({ error: 'Request failed' })); - throw new Error((error as { error: string }).error || `HTTP ${response.status}`); + throw new ApiError( + (error as { error: string }).error || `HTTP ${response.status}`, + response.status + ); } return response.json() as Promise; @@ -267,7 +301,14 @@ class ApiClient { } // Feeds (V2 - via Fly.io proxy) - async fetchFeedV2(url: string, sinceGuids?: string[], limit?: number): Promise { + // One feed's newest slice from the server-side archive. `refresh` forces the + // backend to re-fetch it through the crawler first (the per-feed retry action). + async fetchFeedV2( + url: string, + sinceGuids?: string[], + limit?: number, + refresh = false + ): Promise { const params = new URLSearchParams({ url }); if (sinceGuids && sinceGuids.length > 0) { params.set('since_guids', sinceGuids.join(',')); @@ -275,9 +316,31 @@ class ApiClient { if (limit) { params.set('limit', limit.toString()); } + if (refresh) { + params.set('refresh', '1'); + } return this.fetch(`/api/v2/feeds/fetch?${params}`); } + /** + * The whole feed refresh in one request: every new item across every + * subscription, with read state already stamped on. `since_seq` + `generation` + * are the client's global cursor into the server-side archive; `hasMore` drives + * the drain loop. Replaces the per-subscription batch calls below. + */ + async fetchTimeline(params: { + since_seq?: number; + generation?: string; + limit?: number; + }): Promise { + const search = new URLSearchParams(); + if (params.since_seq !== undefined) search.set('since_seq', String(params.since_seq)); + if (params.generation) search.set('generation', params.generation); + if (params.limit) search.set('limit', String(params.limit)); + const query = search.toString(); + return this.fetch(`/api/v2/timeline${query ? `?${query}` : ''}`); + } + async fetchFeedsBatchV2( feeds: Array<{ url: string; diff --git a/frontend/src/lib/services/feedFetcher.ts b/frontend/src/lib/services/feedFetcher.ts index 0bb96559..db0559ec 100644 --- a/frontend/src/lib/services/feedFetcher.ts +++ b/frontend/src/lib/services/feedFetcher.ts @@ -1,10 +1,18 @@ -import { api } from './api'; +import { api, ApiError } from './api'; import { liveDb } from './liveDb.svelte'; -import { db, type FeedCursorEntry } from './db'; +import { db, getMetadata, setMetadata, type FeedCursorEntry } from './db'; import { feedStatusStore, type V2FeedResult } from '$lib/stores/feedStatus.svelte'; import { socialStore } from '$lib/stores/social.svelte'; import { itemLabelsStore } from '$lib/stores/itemLabels.svelte'; import { buildDocumentRequests, collectDocumentBatches } from './documentSync'; +import { + buildSubscriptionIndex, + groupTimelineItems, + isRssSubscription, + shouldFallBackToBatch, + shouldUpdateTitle, + subscriptionMetaUpdate, +} from './timelineSync'; import { loadDigests, saveDigests, scopeKey } from './documentDigests'; import type { Subscription } from '$lib/types'; @@ -30,52 +38,182 @@ const COLD_START_LIMIT = 30; // it just continues on the next sync. Logged when hit (no silent truncation). const MAX_DRAIN_ROUNDS = 5; +export interface FetchResult { + totalFeeds: number; + successfulFeeds: number; + failedFeeds: number; + newArticles: number; +} + +// Dexie `metadata` key holding the global timeline cursor (one per client, not +// one per subscription — that's the whole point of the timeline). +const TIMELINE_CURSOR_KEY = 'timelineCursor'; + +// Items per timeline page (the server caps at 200 too). +const TIMELINE_PAGE_LIMIT = 200; + +interface TimelineCursor { + cursor: number; + generation: string; +} + +// Set for the session once the backend answers 404 for /api/v2/timeline (an old +// or rolled-back Worker), so we stop probing and stay on the legacy batch path. +let timelineUnavailable = false; + /** - * Check if a subscription's title should be updated from feed metadata. - * Returns true when the current title is a fallback (URL, hostname, etc.) - * and the feed provides a real title. + * The whole refresh in ONE request (plus drain pages): `GET /api/v2/timeline` + * returns every item newer than the client's global cursor across every + * subscription, with read state already joined in. + * + * Returns null when the caller should fall back to the legacy per-feed batch + * path — either the endpoint doesn't exist (old/rolled-back backend) or the + * server-side archive has nothing for this user yet (ingest not enabled in this + * environment). The cursor is never committed in that case, so a later switch to + * the timeline still cold-starts correctly. */ -function shouldUpdateTitle( - currentTitle: string, - feedUrl: string | undefined, - fetchedTitle: string -): boolean { - if (!fetchedTitle || fetchedTitle === 'Untitled Feed') return false; - if (currentTitle === fetchedTitle) return false; - - // Update if current title is the feed URL - if (feedUrl && currentTitle === feedUrl) return true; - - // Update if current title is just a hostname - try { - const hostname = feedUrl ? new URL(feedUrl).hostname : ''; - if (currentTitle === hostname) return true; - } catch { - // ignore invalid URL +async function fetchTimeline( + subscriptions: Subscription[], + savedGuids: Set +): Promise { + const rssSubs = subscriptions.filter(isRssSubscription); + const result: FetchResult = { + totalFeeds: rssSubs.length, + successfulFeeds: 0, + failedFeeds: 0, + newArticles: 0, + }; + if (rssSubs.length === 0) return result; + + // feedUrl → subscriptionId. Items for feeds we don't hold locally are skipped; + // races with an unsubscribe are benign. + const subIdByUrl = buildSubscriptionIndex(rssSubs); + + const stored = await getMetadata(TIMELINE_CURSOR_KEY); + let cursor = stored?.cursor; + let generation = stored?.generation; + + for (let round = 0; round < MAX_DRAIN_ROUNDS; round++) { + let page; + try { + page = await api.fetchTimeline({ + since_seq: cursor, + generation, + limit: TIMELINE_PAGE_LIMIT, + }); + } catch (e) { + if (e instanceof ApiError && e.status === 404) { + // Backend predates the timeline (or was rolled back): stay on /batch. + timelineUnavailable = true; + return null; + } + throw e; + } + + // A cold start that finds nothing while the user has subscriptions means the + // archive isn't populated for them yet (this environment's crawler isn't + // pushing). Fall back rather than showing an empty reader. + if (shouldFallBackToBatch(page, rssSubs.length)) return null; + + if (page.readCursor) await itemLabelsStore.seedReadCursor(page.readCursor); + + const { toMerge, readGuids, feedUrls } = groupTimelineItems(page.items, subIdByUrl); + + // One merge per page (rebuild + re-sort the in-memory array once), same + // discipline as the batch path. If it throws, the cursor below is NOT + // committed, so the next poll re-requests these items instead of skipping them. + if (toMerge.length > 0) { + result.newArticles += await liveDb.mergeArticlesBatch(toMerge, savedGuids); + } + if (readGuids.length > 0) { + await itemLabelsStore.applyAnnotatedReads(readGuids, 'article'); + } + + // Backfill subscription title/siteUrl from the archive's feed metadata. + if (page.feeds) { + for (const [feedUrl, meta] of Object.entries(page.feeds)) { + const subscriptionId = subIdByUrl.get(feedUrl); + if (!subscriptionId) continue; + const sub = liveDb.getSubscriptionById(subscriptionId); + if (!sub) continue; + const updates = subscriptionMetaUpdate(sub, meta); + if (updates) { + await liveDb.updateSubscription(subscriptionId, { + ...updates, + localUpdatedAt: Date.now(), + }); + } + } + } + + // Merge succeeded — commit the cursor. + cursor = page.cursor; + generation = page.generation; + await setMetadata(TIMELINE_CURSOR_KEY, { cursor, generation }); + + // A feed that just delivered items is demonstrably healthy; clear any stale + // error state. Feeds that delivered nothing are left alone — the archive + // carries no per-feed fetch status (that lives with the crawler). + for (const feedUrl of feedUrls) feedStatusStore.markReady(feedUrl); + + if (!page.hasMore) { + result.successfulFeeds = rssSubs.length; + return result; + } } - return false; + console.warn( + `[feedFetcher] Timeline drain cap (${MAX_DRAIN_ROUNDS} rounds) reached; the rest continues on the next sync.` + ); + result.successfulFeeds = rssSubs.length; + return result; } -export interface FetchResult { - totalFeeds: number; - successfulFeeds: number; - failedFeeds: number; - newArticles: number; +/** + * Fetch all subscribed feeds. + * + * Preferred path: one `GET /api/v2/timeline` request (plus drain pages) served + * from the server-side archive, with read state joined in. Falls back to the + * legacy per-feed `POST /api/v2/feeds/batch` fan-out when the timeline isn't + * available — an older backend, a rollback, or an environment whose crawler + * isn't pushing into D1 yet. + * + * @param subscriptions - Array of subscriptions to fetch + * @param savedGuids - Set of starred article GUIDs (to preserve during cleanup) + */ +export async function fetchAllFeeds( + subscriptions: Subscription[], + savedGuids: Set = new Set() +): Promise { + // Skip network requests when offline - cached articles are already loaded + if (typeof navigator !== 'undefined' && !navigator.onLine) { + return { totalFeeds: subscriptions.length, successfulFeeds: 0, failedFeeds: 0, newArticles: 0 }; + } + + if (!timelineUnavailable) { + try { + const timelineResult = await fetchTimeline(subscriptions, savedGuids); + if (timelineResult) return timelineResult; + } catch (e) { + console.error('[feedFetcher] Timeline sync failed, falling back to batch fetch:', e); + } + } + + return fetchAllFeedsViaBatch(subscriptions, savedGuids); } /** - * Fetch all subscribed feeds using V2 batch API + * Legacy path: per-feed batches against the proxy-backed `/api/v2/feeds/batch`, + * with per-subscription durable-log cursors. Kept for one release so a rollback + * (or a not-yet-ingesting environment) can't strand clients; removed in the + * timeline cleanup phase. * * - Chunks feeds into batches of 50 * - Uses GUID-based incremental sync (last 10 GUIDs per feed) * - Updates feedStatusStore with results * - Merges new articles into liveDb - * - * @param subscriptions - Array of subscriptions to fetch - * @param savedGuids - Set of starred article GUIDs (to preserve during cleanup) */ -export async function fetchAllFeeds( +async function fetchAllFeedsViaBatch( subscriptions: Subscription[], savedGuids: Set = new Set() ): Promise { @@ -401,10 +539,15 @@ export interface FetchSingleFeedResult { } /** - * Fetch a single feed using V2 API + * Fetch a single feed's newest slice from the server-side archive. + * + * This is how a brand-new subscription gets its history: its items sit BELOW the + * client's global timeline cursor, so the timeline alone would never deliver + * them. The backend pulls the feed through the crawler on a first-ever fetch. * * @param subscription - Subscription to fetch - * @param force - If true, fetch from source ignoring cache + * @param force - Skips the client-side circuit-breaker check (the archive read + * itself is always current; there is no cache to bypass any more) * @param savedGuids - Set of starred article GUIDs */ export async function fetchSingleFeed( @@ -424,7 +567,9 @@ export async function fetchSingleFeed( try { const recentGuids = force ? undefined : liveDb.getRecentGuids(subscription.id, GUIDS_PER_FEED); - const feed = await api.fetchFeedV2(subscription.feedUrl, recentGuids); + // `force` asks the backend to re-crawl the feed before serving the archive + // (a fresh subscription, or the user retrying a feed that looked broken). + const feed = await api.fetchFeedV2(subscription.feedUrl, recentGuids, undefined, force); // Mark as ready feedStatusStore.markReady(subscription.feedUrl); diff --git a/frontend/src/lib/services/timelineSync.test.ts b/frontend/src/lib/services/timelineSync.test.ts new file mode 100644 index 00000000..6c38e7eb --- /dev/null +++ b/frontend/src/lib/services/timelineSync.test.ts @@ -0,0 +1,154 @@ +import { describe, it, expect } from 'vitest'; +import { + buildSubscriptionIndex, + groupTimelineItems, + isRssSubscription, + shouldFallBackToBatch, + shouldUpdateTitle, + subscriptionMetaUpdate, + type TimelineItem, +} from './timelineSync'; +import type { Subscription } from '$lib/types'; + +const FEED_A = 'https://a.example/feed.xml'; +const FEED_B = 'https://b.example/feed.xml'; + +function sub(overrides: Partial = {}): Subscription { + return { + id: 1, + title: 'Feed A', + feedUrl: FEED_A, + createdAt: '2026-01-01T00:00:00.000Z', + ...overrides, + } as Subscription; +} + +function tItem(overrides: Partial = {}): TimelineItem { + return { + seq: 1, + feedUrl: FEED_A, + read: false, + guid: 'g1', + url: 'https://a.example/g1', + title: 'Item', + publishedAt: '2026-01-01T00:00:00.000Z', + ...overrides, + }; +} + +describe('isRssSubscription', () => { + it('accepts RSS subs and rejects every atproto source', () => { + expect(isRssSubscription(sub())).toBe(true); + expect(isRssSubscription(sub({ sourceType: 'rss' }))).toBe(true); + expect(isRssSubscription(sub({ sourceType: 'atproto.documents' }))).toBe(false); + expect(isRssSubscription(sub({ sourceType: 'atproto.collection' }))).toBe(false); + expect(isRssSubscription(sub({ feedUrl: '' }))).toBe(false); + }); +}); + +describe('buildSubscriptionIndex', () => { + it('maps feed URLs to subscription ids, skipping non-RSS and unsaved subs', () => { + const index = buildSubscriptionIndex([ + sub({ id: 1, feedUrl: FEED_A }), + sub({ id: 2, feedUrl: FEED_B }), + sub({ id: 3, feedUrl: 'at://did:plc:x/pub', sourceType: 'atproto.documents' }), + sub({ id: undefined, feedUrl: 'https://c.example/feed.xml' }), + ]); + expect([...index.entries()]).toEqual([ + [FEED_A, 1], + [FEED_B, 2], + ]); + }); +}); + +describe('groupTimelineItems', () => { + const index = new Map([ + [FEED_A, 1], + [FEED_B, 2], + ]); + + it('buckets items per subscription and collects read guids', () => { + const grouped = groupTimelineItems( + [ + tItem({ seq: 1, guid: 'a1', feedUrl: FEED_A, read: true }), + tItem({ seq: 2, guid: 'b1', feedUrl: FEED_B }), + tItem({ seq: 3, guid: 'a2', feedUrl: FEED_A }), + ], + index + ); + + expect(grouped.toMerge).toEqual([ + { + subscriptionId: 1, + items: [expect.objectContaining({ guid: 'a1' }), expect.objectContaining({ guid: 'a2' })], + }, + { subscriptionId: 2, items: [expect.objectContaining({ guid: 'b1' })] }, + ]); + expect(grouped.readGuids).toEqual(['a1']); + expect(grouped.feedUrls.sort()).toEqual([FEED_A, FEED_B]); + }); + + it('drops items for feeds this client no longer holds', () => { + const grouped = groupTimelineItems( + [tItem({ guid: 'gone', feedUrl: 'https://gone.example/feed.xml' })], + index + ); + expect(grouped.toMerge).toEqual([]); + expect(grouped.feedUrls).toEqual([]); + }); +}); + +describe('shouldFallBackToBatch', () => { + it('falls back when a cold start finds nothing for a subscribed user', () => { + expect(shouldFallBackToBatch({ items: [], coldStart: true }, 3)).toBe(true); + }); + + it('does not fall back for a user with no RSS subscriptions', () => { + expect(shouldFallBackToBatch({ items: [], coldStart: true }, 0)).toBe(false); + }); + + it('does not fall back once the archive delivers anything', () => { + expect(shouldFallBackToBatch({ items: [tItem()], coldStart: true }, 3)).toBe(false); + }); + + it('does not fall back on an empty incremental page (steady state)', () => { + expect(shouldFallBackToBatch({ items: [], coldStart: false }, 3)).toBe(false); + }); +}); + +describe('shouldUpdateTitle', () => { + it('replaces a URL or hostname placeholder with a real title', () => { + expect(shouldUpdateTitle(FEED_A, FEED_A, 'Real Title')).toBe(true); + expect(shouldUpdateTitle('a.example', FEED_A, 'Real Title')).toBe(true); + }); + + it('leaves a real title alone', () => { + expect(shouldUpdateTitle('Curated Name', FEED_A, 'Real Title')).toBe(false); + expect(shouldUpdateTitle(FEED_A, FEED_A, 'Untitled Feed')).toBe(false); + expect(shouldUpdateTitle(FEED_A, FEED_A, '')).toBe(false); + }); +}); + +describe('subscriptionMetaUpdate', () => { + it('backfills title and siteUrl independently', () => { + expect( + subscriptionMetaUpdate({ title: FEED_A, feedUrl: FEED_A }, { title: 'Real Title' }) + ).toEqual({ title: 'Real Title' }); + + expect( + subscriptionMetaUpdate( + { title: 'Curated Name', feedUrl: FEED_A }, + { title: 'Real Title', siteUrl: 'https://a.example' } + ) + ).toEqual({ siteUrl: 'https://a.example' }); + }); + + it('returns null when nothing changes', () => { + expect( + subscriptionMetaUpdate( + { title: 'Curated Name', feedUrl: FEED_A, siteUrl: 'https://a.example' }, + { title: 'Real Title', siteUrl: 'https://a.example' } + ) + ).toBeNull(); + }); +}); diff --git a/frontend/src/lib/services/timelineSync.ts b/frontend/src/lib/services/timelineSync.ts new file mode 100644 index 00000000..69c5d551 --- /dev/null +++ b/frontend/src/lib/services/timelineSync.ts @@ -0,0 +1,147 @@ +import type { FeedItem, Subscription } from '$lib/types'; + +/** + * Pure helpers for the D1-served timeline sync. Extracted from feedFetcher + * (which pulls in Svelte rune modules, untestable in plain vitest) so the + * grouping, fallback and metadata-backfill rules can be unit-tested directly. + * See timelineSync.test.ts. + */ + +/** One item as the timeline serves it: the feed it belongs to + its archive seq. */ +export type TimelineItem = FeedItem & { seq: number; feedUrl: string; read: boolean }; + +export interface TimelinePageShape { + items: TimelineItem[]; + coldStart: boolean; +} + +/** Feed metadata the timeline carries alongside a non-empty page. */ +export interface TimelineFeedMeta { + title?: string; + siteUrl?: string; + imageUrl?: string; +} + +/** RSS subscriptions are the only ones the timeline serves; atproto.* sources + * (standard.site documents, collections) ride their own digest sync. */ +export function isRssSubscription(sub: Subscription): boolean { + return !!sub.feedUrl && !(sub.sourceType && sub.sourceType.startsWith('atproto.')); +} + +/** feedUrl → subscriptionId, for mapping timeline items back onto local subs. */ +export function buildSubscriptionIndex(subscriptions: Subscription[]): Map { + const index = new Map(); + for (const sub of subscriptions) { + if (!sub.id || !isRssSubscription(sub)) continue; + index.set(sub.feedUrl!, sub.id); + } + return index; +} + +export interface GroupedTimelinePage { + // One entry per subscription that received items, ready for a single + // mergeArticlesBatch call. + toMerge: Array<{ subscriptionId: number; items: TimelineItem[] }>; + // GUIDs the server flagged as already read, applied additively after the merge. + readGuids: string[]; + // Feeds that delivered at least one item in this page. + feedUrls: string[]; +} + +/** + * Bucket a page by subscription. Items for feeds this client doesn't hold are + * dropped — a race with an unsubscribe elsewhere is benign. + */ +export function groupTimelineItems( + items: TimelineItem[], + index: Map +): GroupedTimelinePage { + const byFeed = new Map(); + const readGuids: string[] = []; + const feedUrls = new Set(); + + for (const item of items) { + const subscriptionId = index.get(item.feedUrl); + if (!subscriptionId) continue; + feedUrls.add(item.feedUrl); + const bucket = byFeed.get(subscriptionId); + if (bucket) bucket.push(item); + else byFeed.set(subscriptionId, [item]); + if (item.read) readGuids.push(item.guid); + } + + return { + toMerge: [...byFeed.entries()].map(([subscriptionId, bucket]) => ({ + subscriptionId, + items: bucket, + })), + readGuids, + feedUrls: [...feedUrls], + }; +} + +/** + * Whether to abandon the timeline for this sync and use the legacy per-feed + * batch path instead. + * + * A cold start that finds nothing while the user has subscriptions means the + * server-side archive isn't populated for them yet — the environment's crawler + * isn't pushing into D1. Showing an empty reader would be wrong; falling back + * (without committing a cursor) keeps the reader working until ingest catches up. + */ +export function shouldFallBackToBatch(page: TimelinePageShape, subscriptionCount: number): boolean { + return page.coldStart && page.items.length === 0 && subscriptionCount > 0; +} + +/** + * Whether a subscription's title should be updated from feed metadata. + * Returns true when the current title is a fallback (URL, hostname, etc.) + * and the feed provides a real title. + */ +export function shouldUpdateTitle( + currentTitle: string, + feedUrl: string | undefined, + fetchedTitle: string +): boolean { + if (!fetchedTitle || fetchedTitle === 'Untitled Feed') return false; + if (currentTitle === fetchedTitle) return false; + + // Update if current title is the feed URL + if (feedUrl && currentTitle === feedUrl) return true; + + // Update if current title is just a hostname + try { + const hostname = feedUrl ? new URL(feedUrl).hostname : ''; + if (currentTitle === hostname) return true; + } catch { + // ignore invalid URL + } + + return false; +} + +export interface SubscriptionMetaUpdate { + title?: string; + siteUrl?: string; +} + +/** + * The subscription fields worth backfilling from the archive's feed metadata, or + * null when nothing changes. Title and siteUrl are independent: a sub added + * before siteUrl tracking existed already has a real title, so gating the + * siteUrl write on a title change would leave it siteUrl-less forever (which + * hides it from cross-type duplicate detection). + */ +export function subscriptionMetaUpdate( + sub: Pick, + meta: TimelineFeedMeta +): SubscriptionMetaUpdate | null { + const updates: SubscriptionMetaUpdate = {}; + if (meta.title && shouldUpdateTitle(sub.title, sub.feedUrl, meta.title)) { + updates.title = meta.title; + } + if (meta.siteUrl && meta.siteUrl !== sub.siteUrl) { + updates.siteUrl = meta.siteUrl; + } + return updates.title !== undefined || updates.siteUrl !== undefined ? updates : null; +} diff --git a/frontend/src/lib/types/index.ts b/frontend/src/lib/types/index.ts index 6f3f0c65..6b9cf4ab 100644 --- a/frontend/src/lib/types/index.ts +++ b/frontend/src/lib/types/index.ts @@ -801,6 +801,9 @@ export interface FeedItem { // (inline read annotation). Consumed additively on merge, then discarded — it // is not an Article column. Absent on un-annotated responses. read?: boolean; + // The stored body exceeded the archive's per-item content cap and was dropped + // at ingest; the reader falls back to on-demand extraction for full text. + contentTruncated?: boolean; } // Combined feed item for unified "all" view diff --git a/scripts/dev-local.sh b/scripts/dev-local.sh index 88ec3768..dff0e774 100755 --- a/scripts/dev-local.sh +++ b/scripts/dev-local.sh @@ -28,14 +28,27 @@ cleanup() { trap cleanup SIGINT SIGTERM +# Local dev is its own environment "pair": the proxy pushes its item log into the +# local Worker's D1 and pulls the crawl set back from it. Ingest is fail-closed on +# the Worker side, so both halves must share this secret. +DEV_PROXY_SECRET="dev-proxy-secret" + # Check for required .dev.vars if [ ! -f "$BACKEND_DIR/.dev.vars" ]; then echo -e "${RED}Missing $BACKEND_DIR/.dev.vars${NC}" echo "Create it with:" echo " FRONTEND_URL=http://127.0.0.1:5173" + echo " FEED_PROXY_URL=http://127.0.0.1:3000" + echo " FEED_PROXY_SECRET=$DEV_PROXY_SECRET" exit 1 fi +if ! grep -q "^FEED_PROXY_SECRET=" "$BACKEND_DIR/.dev.vars"; then + echo -e "${YELLOW}Warning: $BACKEND_DIR/.dev.vars has no FEED_PROXY_SECRET.${NC}" + echo -e "${YELLOW}Feed ingest is fail-closed, so the proxy's pushes will 401 and the${NC}" + echo -e "${YELLOW}reader will stay empty. Add: FEED_PROXY_SECRET=$DEV_PROXY_SECRET${NC}" +fi + echo -e "${GREEN}Starting local development environment...${NC}\n" # Run D1 migrations @@ -47,11 +60,11 @@ if ! echo "y" | npx wrangler d1 migrations apply skyreader --local; then fi echo -e "${GREEN}Migrations applied.${NC}\n" -# Start feed proxy (no auth needed locally) +# Start feed proxy (crawler + ingest pusher pointed at the local Worker) echo -e "${YELLOW}[1/4] Starting feed proxy...${NC}" cd "$FEED_PROXY_DIR" bun install --frozen-lockfile 2>/dev/null || bun install -bun run dev & +INGEST_URL=http://127.0.0.1:8787 PROXY_SECRET="$DEV_PROXY_SECRET" bun run dev & FEED_PROXY_PID=$! sleep 2 From 6281d0ca8bebe9cea94e5df540cbeb851367b589 Mon Sep 17 00:00:00 2001 From: "claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla)" Date: Wed, 12 Aug 2026 23:37:08 +0000 Subject: [PATCH 2/7] Address the timeline review: ingest order, a server-authoritative rollout gate, and paged cold starts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ten findings from the review of the D1 timeline change, most consequential first. The subscribe-time pull-through ingested a proxy feed FORWARD. A proxy feed is newest-first and seq is assigned in insert order, so the newest item got the lowest seq — and since a re-push of an unchanged item is a no-op, that inversion never healed: every later per-feed cold start served the feed's oldest entries, and the sanity-cap trim would have deleted its newest. `ingestProxyFeed` now walks the array backwards, as the proxy's own `writeFeedItems` does. The rollout gate was inferred from an empty archive, which one pull-through write falsified: a client could commit a cursor against a D1 nothing was crawling and stop refreshing 40 feeds in silence. Both internal endpoints now stamp `sync_state.crawler_heartbeat_at` (the crawl-set pull runs every 5 minutes regardless of feed activity), the timeline reports `ingestActive` from it, and the client stays on the batch path until the crawler is demonstrably alive. Cold starts were bounded only by 500 feeds x 30 items — up to 15,000 rows buffered in one Worker response, for everyone at once after a generation bump. They now page: feeds in a stable order, an item budget per page, continuation via `cold_offset`/`nextColdOffset`. The client keeps the FIRST page's cursor (the head read before the slices, so concurrent ingest can't be skipped) and commits it only once the last page merges. Also: - A cursor above the archive head now cold-starts that client, the D1 twin of the proxy's snapshot-restore guard — a Time Travel restore no longer stalls every client forever pending a manual `UPDATE sync_state`. - `contentTruncated` is no longer inert: it rides into the Article row and ArticleCard extracts the original when such an article is expanded, so a full-text feed doesn't quietly degrade to a two-sentence RSS summary. - Subscriptions that arrive from another device sit below the global cursor, so each is backfilled once through the per-feed endpoint (<= 10 per sync, tracked in Dexie). - New-subscription backfill no longer forces a crawl per feed and is paced (3 at a time, 1 s apart); subscribe time crawls straight into the archive (`warmFeedIntoArchive` replaces the old warm-and-discard), and the endpoint gets its own light rate limit. A 250-feed OPML import stops 429ing halfway. - The pull-through requires the caller to subscribe to the feed, so the shared never-pruned archive isn't an open write surface. - Staging keeps pointing at the prod proxy until Phase 3 provisions its own Fly app, and the staging deploy job skips itself with a notice until then, instead of taking staging's extract/discovery down on merge. - The incremental drain's scan cost is documented where it lives. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) --- .github/workflows/feed-proxy-deploy.yml | 16 ++ backend/CLAUDE.md | 8 +- backend/src/routes/feeds-v2.ts | 62 ++++-- backend/src/routes/ingest.ts | 60 +++++- backend/src/routes/subscriptions.ts | 20 +- backend/src/routes/timeline.ts | 170 +++++++++++++---- backend/src/services/rate-limit.ts | 5 + backend/test/feed-timeline.spec.ts | 144 +++++++++++++- backend/test/feeds-v2-fetch.spec.ts | 142 ++++++++++++++ backend/wrangler.toml | 16 +- docs/plans/D1_FEED_TIMELINE.md | 73 +++++-- e2e/seed.ts | 5 + frontend/CLAUDE.md | 28 +-- .../src/lib/components/ArticleCard.svelte | 13 ++ .../src/lib/components/ImportOPMLModal.svelte | 7 +- frontend/src/lib/services/api.ts | 7 + frontend/src/lib/services/articleMerge.ts | 3 + frontend/src/lib/services/feedFetcher.ts | 179 ++++++++++++++---- .../src/lib/services/timelineSync.test.ts | 55 +++++- frontend/src/lib/services/timelineSync.ts | 62 +++++- frontend/src/lib/types/index.ts | 4 + 21 files changed, 933 insertions(+), 146 deletions(-) create mode 100644 backend/test/feeds-v2-fetch.spec.ts diff --git a/.github/workflows/feed-proxy-deploy.yml b/.github/workflows/feed-proxy-deploy.yml index 42df642f..50e65fef 100644 --- a/.github/workflows/feed-proxy-deploy.yml +++ b/.github/workflows/feed-proxy-deploy.yml @@ -70,7 +70,23 @@ jobs: - name: Setup flyctl uses: superfly/flyctl-actions/setup-flyctl@ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1 # master + # The app is provisioned by hand (Phase 3 of docs/plans/D1_FEED_TIMELINE.md: + # app + volume + PROXY_SECRET). Until that happens this job would fail on + # every push to main, so skip it with a notice instead. + - name: Check staging app exists + id: staging_app + run: | + if flyctl status -a skyreader-feed-proxy-staging >/dev/null 2>&1; then + echo "exists=true" >> "$GITHUB_OUTPUT" + else + echo "exists=false" >> "$GITHUB_OUTPUT" + echo "::notice::skyreader-feed-proxy-staging not provisioned yet; skipping staging deploy (see docs/plans/D1_FEED_TIMELINE.md, Phase 3)." + fi + env: + FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }} + - name: Deploy + if: steps.staging_app.outputs.exists == 'true' run: flyctl deploy --remote-only --config fly.staging.toml env: FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }} diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 424a7bef..9978a004 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -10,8 +10,12 @@ Skyreader backend is a Cloudflare Workers API that serves as a gateway between t and edited items into `feed_items` (`POST /api/internal/ingest`) and pulls the set of feeds to crawl (`GET /api/internal/crawl-set`), both authenticated with the shared `FEED_PROXY_SECRET` (fail-closed when unset). A client refresh is one `GET /api/v2/timeline` — a single query joining -subscriptions and read state. See `docs/plans/D1_FEED_TIMELINE.md`. The proxy is still on the path -for `/api/extract`, feed discovery, standard.site documents, and social context. +subscriptions and read state. Both internal endpoints stamp `sync_state.crawler_heartbeat_at`, and +the timeline reports `ingestActive` from it: an environment whose proxy has no `INGEST_URL` tells +clients to stay on the legacy batch path instead of reading an archive nothing fills. See +`docs/plans/D1_FEED_TIMELINE.md`. The proxy is still on the path for `/api/extract`, feed +discovery, standard.site documents, and social context — plus the one crawl per new subscription +(`warmFeedIntoArchive`) and the subscription-gated pull-through in `/api/v2/feeds/fetch`. ## Key Concepts diff --git a/backend/src/routes/feeds-v2.ts b/backend/src/routes/feeds-v2.ts index 2b327064..64f04d95 100644 --- a/backend/src/routes/feeds-v2.ts +++ b/backend/src/routes/feeds-v2.ts @@ -53,6 +53,21 @@ interface V2BatchResponse { const SINGLE_FEED_LIMIT = 30; const SINGLE_FEED_MAX_LIMIT = 200; +/** + * Does this user hold a subscription to this feed? The gate on every write into + * the shared archive that a user request can trigger. Parked (`active = 0`) subs + * count — the user owns the feed either way, and re-activating it shouldn't need + * a different code path. + */ +async function callerSubscribes(env: Env, userDid: string, feedUrl: string): Promise { + const row = await env.DB.prepare( + `SELECT 1 AS ok FROM subscriptions_cache WHERE user_did = ? AND feed_url = ? LIMIT 1` + ) + .bind(userDid, feedUrl) + .first<{ ok: number }>(); + return !!row; +} + /** * GET /api/v2/feeds/fetch * @@ -65,6 +80,13 @@ const SINGLE_FEED_MAX_LIMIT = 200; * pushed it), we PULL THROUGH: fetch it from the proxy once, ingest the result, * then serve from D1. Steady state never touches Fly. * + * The pull-through only runs for a feed the CALLER actually subscribes to. The + * archive is shared and (by design) never pruned, so an open ingest surface would + * let any authenticated user write arbitrary feeds into it forever; requiring a + * subscription bounds writes to each user's own feed list. Subscriptions are + * written to `subscriptions_cache` synchronously by POST /api/subscriptions + * before the client fetches, so the add-feed path is unaffected. + * * Query params: * - url: Feed URL (required) * - limit: Max items to return (optional, default 30) @@ -113,9 +135,11 @@ export async function handleV2FeedFetch( // Pull-through: the archive has nothing for this feed (first subscriber, so // the crawler never pushed it), or the caller explicitly asked for a fresh // fetch. One synchronous proxy call, ingested so every later read — and - // every other user — comes from D1. + // every other user — comes from D1. Gated on the caller's own subscription + // (see the note above); a non-subscriber just reads whatever D1 already holds. const archiveEmpty = items.length === 0; - if (archiveEmpty || forceRefresh) { + const wantsPullThrough = archiveEmpty || forceRefresh; + if (wantsPullThrough && (await callerSubscribes(env, session.did, feedUrl))) { try { const client = new FeedProxyClient(env); const feed = await client.fetchFeed(feedUrl); @@ -713,26 +737,31 @@ export interface WarmCacheResult { } /** - * Warm up the proxy cache for a single feed. - * Just fetches via proxy to ensure it's cached - no D1 storage. + * Fetch a newly subscribed feed through the proxy and INGEST it into the archive. + * + * This used to only warm the proxy's cache and throw the parse away. Now that D1 + * is the read path, the same one fetch we were already paying for at subscribe + * time populates the archive — so the client's first per-feed read is a plain D1 + * query instead of another synchronous crawl through the pull-through. */ -export async function warmProxyCache(env: Env, feedUrl: string): Promise { +export async function warmFeedIntoArchive(env: Env, feedUrl: string): Promise { try { const client = new FeedProxyClient(env); const feed = await client.fetchFeed(feedUrl); + await ingestProxyFeed(env, feedUrl, feed); return { success: true, itemCount: feed.items.length }; } catch (error) { const errorMessage = error instanceof Error ? error.message : 'Failed to fetch feed'; - console.error(`[warmProxyCache] Error fetching ${feedUrl}:`, errorMessage); + console.error(`[warmFeedIntoArchive] Error fetching ${feedUrl}:`, errorMessage); return { success: false, error: errorMessage }; } } /** - * Warm up the proxy cache for multiple feeds in a single batch request. - * Just fetches via proxy to ensure they're cached - no D1 storage. + * The same, for several feeds in one proxy batch request (bulk / OPML import). + * A feed whose ingest fails is reported as an error but never fails the others. */ -export async function warmProxyCacheBatch( +export async function warmFeedsIntoArchive( env: Env, feedUrls: string[] ): Promise> { @@ -757,15 +786,20 @@ export async function warmProxyCacheBatch( } else if (feedResult.status === 'error') { results[feedUrl] = { success: false, error: feedResult.error }; } else { - results[feedUrl] = { - success: true, - itemCount: feedResult.items.length, - }; + try { + await ingestProxyFeed(env, feedUrl, feedResult); + results[feedUrl] = { success: true, itemCount: feedResult.items.length }; + } catch (error) { + results[feedUrl] = { + success: false, + error: error instanceof Error ? error.message : 'Ingest failed', + }; + } } } } catch (error) { const errorMessage = error instanceof Error ? error.message : 'Batch fetch failed'; - console.error(`[warmProxyCacheBatch] Batch error:`, errorMessage); + console.error(`[warmFeedsIntoArchive] Batch error:`, errorMessage); for (const feedUrl of feedUrls) { results[feedUrl] = { success: false, error: errorMessage }; diff --git a/backend/src/routes/ingest.ts b/backend/src/routes/ingest.ts index c668c54f..d29d1d90 100644 --- a/backend/src/routes/ingest.ts +++ b/backend/src/routes/ingest.ts @@ -18,9 +18,19 @@ export const SANITY_CAP = 5000; // Stored-content cap per item. Unbounded retention makes this mandatory rather // than optional: D1 has a hard 10 GB database ceiling, so storage grows with // ingest velocity × item size × time. Oversized bodies are dropped at ingest -// (summary/title/url/image kept) and the reader falls back to /extract on demand. +// (summary/title/url/image kept) and `contentTruncated` is set, which the reader +// acts on: ArticleCard auto-extracts the full text via /api/extract when a +// truncated article is opened, so the body the user sees is still the whole +// article (see frontend/src/lib/components/ArticleCard.svelte). export const MAX_ITEM_CONTENT_BYTES = 8 * 1024; +// How long a crawler heartbeat stays "fresh". The proxy pulls the crawl set every +// 5 minutes whenever INGEST_URL is set, so a stamp older than this means this +// environment has no crawler pushing into its D1 — the timeline says so and +// clients stay on the legacy batch path instead of serving an empty archive. +export const CRAWLER_HEARTBEAT_KEY = 'crawler_heartbeat_at'; +export const CRAWLER_HEARTBEAT_FRESH_SECONDS = 30 * 60; + // Bounds on one ingest call. The pusher chunks to ~100 items; these are abuse // guards, not tuning knobs. const MAX_INGEST_ITEMS = 1000; @@ -235,9 +245,18 @@ export async function trimFeedsToSanityCap( } /** - * Ingest a feed fetched straight from the proxy (the subscribe-time pull-through - * in feeds-v2.ts). Hashes match what the crawler will push later, so the first - * real push over these rows is a no-op rather than a phantom edit. + * Ingest a feed fetched straight from the proxy (the subscribe-time warm/ingest + * and the pull-through in feeds-v2.ts). Hashes match what the crawler will push + * later, so the first real push over these rows is a no-op rather than a phantom + * edit. + * + * Order matters: a proxy feed is newest-first, and `seq` is assigned in insert + * order, so we walk the array BACKWARDS — oldest first — exactly as the proxy's + * own `writeFeedItems` does. Ingesting forward would give the newest item the + * lowest seq, and since a re-push of an unchanged item is a no-op, that + * inversion would never heal: every later per-feed cold start (`ORDER BY seq + * DESC`) would serve the feed's OLDEST items, and the sanity-cap trim (deletes + * the lowest seqs) would delete its newest. */ export async function ingestProxyFeed( env: Env, @@ -251,8 +270,9 @@ export async function ingestProxyFeed( } ): Promise { const nowMs = Date.now(); + const oldestFirst = [...feed.items].reverse(); const items: IngestItem[] = await Promise.all( - feed.items.map(async (item) => { + oldestFirst.map(async (item) => { const publishedMs = new Date(item.publishedAt).getTime(); return { feedUrl, @@ -280,6 +300,31 @@ export async function ingestProxyFeed( ); } +/** + * Record that this environment's crawler just talked to us. Both internal + * endpoints stamp it, so the signal survives a quiet period with no new items + * (the crawl-set pull runs every 5 minutes regardless of what the feeds do). + * + * This is what makes "is ingest live here?" server-authoritative instead of + * inferred from an empty archive: a Worker whose proxy has no INGEST_URL never + * gets a stamp, so `/api/v2/timeline` reports `ingestActive: false` and clients + * keep using the legacy batch path rather than committing a cursor against an + * archive nothing is filling. + */ +export async function stampCrawlerHeartbeat(env: Env): Promise { + try { + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at` + ) + .bind(CRAWLER_HEARTBEAT_KEY, String(Math.floor(Date.now() / 1000))) + .run(); + } catch (error) { + // Observability only — never fail an ingest because the stamp didn't land. + console.error('[ingest] Failed to stamp crawler heartbeat:', error); + } +} + /** * POST /api/internal/ingest * @@ -307,6 +352,7 @@ export async function handleIngest(request: Request, env: Env): Promise 0) { - const cacheResults = await warmProxyCacheBatch(env, feedsToWarmNow); + const cacheResults = await warmFeedsIntoArchive(env, feedsToWarmNow); for (const [feedUrl, result] of Object.entries(cacheResults)) { if (result.success) { - console.log(`Warmed proxy cache: ${feedUrl} (${result.itemCount} items)`); + console.log(`Ingested new feed: ${feedUrl} (${result.itemCount} items)`); } else { - console.error(`Failed to warm cache for ${feedUrl}: ${result.error}`); + console.error(`Failed to ingest ${feedUrl}: ${result.error}`); } } } diff --git a/backend/src/routes/timeline.ts b/backend/src/routes/timeline.ts index ecd5d40a..cb54d000 100644 --- a/backend/src/routes/timeline.ts +++ b/backend/src/routes/timeline.ts @@ -1,5 +1,9 @@ import type { Env, FeedItem, Session } from '../types'; -import { rssSubscriptionPredicate } from './ingest'; +import { + CRAWLER_HEARTBEAT_KEY, + CRAWLER_HEARTBEAT_FRESH_SECONDS, + rssSubscriptionPredicate, +} from './ingest'; /** * GET /api/v2/timeline — the whole feed refresh, in one request. @@ -26,6 +30,13 @@ const COLD_START_CHUNK = 25; // A cold start touches every subscribed feed; bound the work (and log if hit — // no silent truncation). const COLD_START_MAX_FEEDS = 500; +// Rows one cold-start PAGE may accumulate. A cold start walks feeds in a stable +// order and stops once it passes this budget, handing back `nextColdOffset` for +// the next page — so a 150-feed reader (or everyone at once after a generation +// bump) can't ask the Worker to buffer thousands of content-bearing items in a +// single response. Checked per chunk, so a page can overshoot by at most +// COLD_START_CHUNK × COLD_START_PER_FEED. +const COLD_START_MAX_ITEMS = 750; export interface TimelineItem extends FeedItem { seq: number; @@ -102,19 +113,55 @@ export async function readFeedMetadata(env: Env, feedUrl: string): Promise(); } -export async function getItemsGeneration(env: Env): Promise { - const row = await env.DB.prepare( - `SELECT value FROM sync_state WHERE key = 'items_generation'` - ).first<{ value: string }>(); - return row?.value ?? ''; +export interface ArchiveState { + generation: string; + // True when this environment's crawler has checked in recently. False means + // nothing is filling this D1 (no INGEST_URL on the paired proxy, or the proxy + // is down), so the client must not treat an empty/partial archive as the truth. + ingestActive: boolean; } +/** + * Generation token + crawler liveness in one `sync_state` read (the timeline + * needs both on every request, and they live one row apart). + */ +export async function readArchiveState(env: Env): Promise { + const rows = await env.DB.prepare( + `SELECT key, value FROM sync_state WHERE key IN ('items_generation', ?)` + ) + .bind(CRAWLER_HEARTBEAT_KEY) + .all<{ key: string; value: string }>(); + + let generation = ''; + let heartbeat = 0; + for (const row of rows.results) { + if (row.key === 'items_generation') generation = row.value; + else if (row.key === CRAWLER_HEARTBEAT_KEY) heartbeat = parseInt(row.value, 10) || 0; + } + + const age = Math.floor(Date.now() / 1000) - heartbeat; + return { generation, ingestActive: heartbeat > 0 && age <= CRAWLER_HEARTBEAT_FRESH_SECONDS }; +} + +/** The archive's current head; 0 when nothing has ever been ingested. */ +async function archiveHead(env: Env): Promise { + const row = await env.DB.prepare('SELECT MAX(seq) AS max_seq FROM feed_items').first<{ + max_seq: number | null; + }>(); + return row?.max_seq ?? 0; +} + +/** + * Subscribed RSS feed URLs in a STABLE order — the cold start pages through this + * list by index, so the ordering has to be the same from one page to the next. + */ async function subscribedFeedUrls(env: Env, userDid: string): Promise { const rows = await env.DB.prepare( `SELECT DISTINCT feed_url FROM subscriptions_cache WHERE user_did = ? AND active = 1 AND feed_url IS NOT NULL AND feed_url <> '' - AND ${rssSubscriptionPredicate()}` + AND ${rssSubscriptionPredicate()} + ORDER BY feed_url` ) .bind(userDid) .all<{ feed_url: string }>(); @@ -173,6 +220,7 @@ export async function handleTimeline( const sinceSeqParam = url.searchParams.get('since_seq'); const generationParam = url.searchParams.get('generation'); const limitParam = url.searchParams.get('limit'); + const coldOffsetParam = url.searchParams.get('cold_offset'); const parsedLimit = limitParam ? parseInt(limitParam, 10) : DEFAULT_LIMIT; const limit = Number.isInteger(parsedLimit) @@ -182,18 +230,35 @@ export async function handleTimeline( const parsedSince = sinceSeqParam !== null ? parseInt(sinceSeqParam, 10) : NaN; const sinceSeq = Number.isInteger(parsedSince) && parsedSince >= 0 ? parsedSince : undefined; - const generation = await getItemsGeneration(env); + // Continuation index into the caller's (stably ordered) subscribed-feed list. + const parsedColdOffset = coldOffsetParam !== null ? parseInt(coldOffsetParam, 10) : NaN; + const coldOffset = + Number.isInteger(parsedColdOffset) && parsedColdOffset > 0 ? parsedColdOffset : 0; + + const { generation, ingestActive } = await readArchiveState(env); // Server time (unix seconds) at annotation. The client seeds its forward // read-delta cursor from this, exactly as /batch does today, so the delta // starts from bootstrap with no client/server clock skew. const readCursor = Math.floor(Date.now() / 1000); - const incremental = sinceSeq !== undefined && generationParam === generation && generation !== ''; + const incremental = + sinceSeq !== undefined && + coldOffset === 0 && + generationParam === generation && + generation !== ''; try { if (incremental) { // Drain oldest-unseen first so a backlog larger than one page is paged // across polls, never skipped. limit+1 probes hasMore without a second query. + // + // Scaling note: this walks the `feed_items` rowid range above the cursor and + // probes the subscription set per row, so the cost tracks GLOBAL ingest above + // the cursor rather than the caller's own new items. That is the accepted + // fan-out-on-read trade at ~1,300 feeds; if D1 row-reads ever become the + // constraint, the fix is to bound the scan with per-feed `(feed_url, seq)` + // seeks (idx_feed_items_feed_seq already supports them), not to materialize + // per-user timelines. const rows = await env.DB.prepare( `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} FROM feed_items fi @@ -211,23 +276,39 @@ export async function handleTimeline( const hasMore = rows.results.length > limit; const page = hasMore ? rows.results.slice(0, limit) : rows.results; - // Cursor comes from the returned rows, never a separate MAX(seq): the - // latter races ingest and would skip everything written in between. - const cursor = page.length > 0 ? page[page.length - 1].seq : sinceSeq; - const items = toTimelineItems(page); - - return json({ - items, - cursor, - generation, - hasMore, - readCursor, - coldStart: false, - feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined, - }); + + // Rewound-archive guard, the D1 twin of the proxy's snapshot-restore check. + // A Time Travel restore (or a rebuild from export) rewinds `feed_items` seqs + // while `items_generation` comes back unchanged, leaving every client cursor + // above the head: `seq > ?` then returns nothing on every poll, forever. A + // cursor can never legitimately exceed the head, so treat that as a cold + // start instead of silently starving the client. Only costs a MAX(seq) on + // an otherwise empty page — never on a page that carried items. + if (page.length === 0 && sinceSeq > (await archiveHead(env))) { + console.warn( + `[timeline] Cursor ${sinceSeq} is above the archive head; cold-starting (archive rewound?).` + ); + } else { + // Cursor comes from the returned rows, never a separate MAX(seq): the + // latter races ingest and would skip everything written in between. + const cursor = page.length > 0 ? page[page.length - 1].seq : sinceSeq; + const items = toTimelineItems(page); + + return json({ + items, + cursor, + generation, + ingestActive, + hasMore, + readCursor, + coldStart: false, + feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined, + }); + } } - // Cold start: no cursor, or a generation mismatch (D1 recreated / restored). + // Cold start: no cursor, a generation mismatch (D1 recreated / restored), a + // rewound archive, or a continuation page of one already in progress. const allFeedUrls = await subscribedFeedUrls(env, session.did); const feedUrls = allFeedUrls.slice(0, COLD_START_MAX_FEEDS); if (allFeedUrls.length > feedUrls.length) { @@ -236,9 +317,25 @@ export async function handleTimeline( ); } + // The cold-start cursor is the archive head read BEFORE the per-feed slices, + // and the client keeps the first page's value across a paged cold start. + // Reading it first is what makes it safe: an item ingested while we page gets + // a seq above this head, so it arrives on the first incremental poll instead + // of being skipped. (Re-delivering a handful of items is harmless — the merge + // dedupes by GUID.) The incremental path still derives its cursor from the + // rows it returned. + const cursor = await archiveHead(env); + + // Walk feeds from the continuation point until the page budget is spent. The + // client re-requests with `cold_offset=nextColdOffset` until hasMore is false. + // A subscription added or removed mid-cold-start shifts the offsets by one, so + // a feed can be missed by this bootstrap; the client's per-feed backfill picks + // up any subscription that ends up with no articles. const rows: ItemRow[] = []; - for (let i = 0; i < feedUrls.length; i += COLD_START_CHUNK) { - const statements = feedUrls.slice(i, i + COLD_START_CHUNK).map((feedUrl) => + let nextIndex = Math.min(coldOffset, feedUrls.length); + while (nextIndex < feedUrls.length && rows.length < COLD_START_MAX_ITEMS) { + const chunk = feedUrls.slice(nextIndex, nextIndex + COLD_START_CHUNK); + const statements = chunk.map((feedUrl) => env.DB.prepare( `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} FROM feed_items fi @@ -247,31 +344,20 @@ export async function handleTimeline( LIMIT ?3` ).bind(session.did, feedUrl, COLD_START_PER_FEED) ); - if (statements.length === 0) continue; const results = await env.DB.batch(statements); for (const result of results) rows.push(...(result.results ?? [])); + nextIndex += chunk.length; } - - let cursor = 0; - for (const row of rows) if (row.seq > cursor) cursor = row.seq; - if (cursor === 0) { - // Nothing to deliver (fresh account, or the feeds aren't ingested yet). - // Start the client at the current head rather than 0, so its first - // incremental poll doesn't scan the archive from the beginning. Anything - // ingested from here on is above this cursor; history for a brand-new - // subscription arrives via the single-feed endpoint's pull-through. - const head = await env.DB.prepare('SELECT MAX(seq) AS max_seq FROM feed_items').first<{ - max_seq: number | null; - }>(); - cursor = head?.max_seq ?? 0; - } + const hasMore = nextIndex < feedUrls.length; const items = toTimelineItems(rows); return json({ items, cursor, generation, - hasMore: false, + ingestActive, + hasMore, + nextColdOffset: hasMore ? nextIndex : undefined, readCursor, coldStart: true, feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined, diff --git a/backend/src/services/rate-limit.ts b/backend/src/services/rate-limit.ts index ed711ef5..31dd5ee8 100644 --- a/backend/src/services/rate-limit.ts +++ b/backend/src/services/rate-limit.ts @@ -57,6 +57,11 @@ const RATE_LIMITS: Record = { // The whole feed refresh is one D1 query per page now, and a returning reader // drains several pages in a burst — a light limit, not a standard one. '/api/v2/timeline': LIGHT_LIMIT, + // Single-feed reads are D1 queries now (the archive), and an OPML import fires + // one per imported feed to backfill it. The one expensive case — the + // pull-through crawl — only happens for a feed the caller subscribes to and + // only when the archive has nothing for it yet. + '/api/v2/feeds/fetch': LIGHT_LIMIT, '/api/feeds/cached': LIGHT_LIMIT, '/api/feeds/batch': LIGHT_LIMIT, '/api/feeds/status': LIGHT_LIMIT, diff --git a/backend/test/feed-timeline.spec.ts b/backend/test/feed-timeline.spec.ts index 06141867..7008439f 100644 --- a/backend/test/feed-timeline.spec.ts +++ b/backend/test/feed-timeline.spec.ts @@ -1,7 +1,13 @@ import { env } from 'cloudflare:test'; import { describe, it, expect, beforeEach, afterEach } from 'vitest'; -import { handleIngest, handleCrawlSet, trimFeedsToSanityCap } from '../src/routes/ingest'; -import { handleTimeline } from '../src/routes/timeline'; +import { + handleIngest, + handleCrawlSet, + trimFeedsToSanityCap, + ingestProxyFeed, + CRAWLER_HEARTBEAT_KEY, +} from '../src/routes/ingest'; +import { handleTimeline, readFeedSlice } from '../src/routes/timeline'; import type { Env, FeedItem, Session } from '../src/types'; const TEST_DID = 'did:plc:timeline123'; @@ -103,12 +109,18 @@ async function timeline(params: Record = {}) { cursor: number; generation: string; hasMore: boolean; + nextColdOffset?: number; + ingestActive: boolean; readCursor: number; coldStart: boolean; feeds?: Record; }; } +async function clearCrawlerHeartbeat() { + await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(CRAWLER_HEARTBEAT_KEY).run(); +} + describe('feed timeline (D1 ingest + serve)', () => { let savedSecret: string | undefined; @@ -131,6 +143,7 @@ describe('feed timeline (D1 ingest + serve)', () => { await env.DB.prepare('DELETE FROM feeds').run(); await env.DB.prepare('DELETE FROM subscriptions_cache').run(); await env.DB.prepare('DELETE FROM item_labels_cache').run(); + await clearCrawlerHeartbeat(); }); describe('ingest auth', () => { @@ -276,6 +289,41 @@ describe('feed timeline (D1 ingest + serve)', () => { }); }); + describe('pull-through ingest (proxy feed → archive)', () => { + it('assigns seq oldest→newest, so the newest entries are what a per-feed read serves', async () => { + // A proxy feed is newest-first. Ingesting it forward would give the newest + // item the lowest seq, and every later `ORDER BY seq DESC` read would then + // serve the feed's OLDEST items. + const newestFirst = Array.from({ length: 5 }, (_, i) => item(`p${5 - i}`)); + await ingestProxyFeed(env, FEED_A, { title: 'Pulled', items: newestFirst }); + + const rows = await env.DB.prepare( + 'SELECT guid, seq FROM feed_items WHERE feed_url = ? ORDER BY seq ASC' + ) + .bind(FEED_A) + .all<{ guid: string; seq: number }>(); + expect(rows.results.map((r) => r.guid)).toEqual(['p1', 'p2', 'p3', 'p4', 'p5']); + + const slice = await readFeedSlice(env, TEST_DID, FEED_A, 2); + expect(slice.map((i) => i.guid)).toEqual(['p5', 'p4']); + }); + + it('hashes identically to a later crawler push, so the push is a no-op', async () => { + await ingestProxyFeed(env, FEED_A, { title: 'Pulled', items: [item('same')] }); + const before = await env.DB.prepare('SELECT seq, content_hash FROM feed_items WHERE guid = ?') + .bind('same') + .first<{ seq: number; content_hash: string }>(); + + await ingestProxyFeed(env, FEED_A, { title: 'Pulled', items: [item('same')] }); + const rows = await env.DB.prepare('SELECT seq, content_hash FROM feed_items WHERE guid = ?') + .bind('same') + .all<{ seq: number; content_hash: string }>(); + expect(rows.results.length).toBe(1); + expect(rows.results[0].seq).toBe(before?.seq); + expect(rows.results[0].content_hash).toBe(before?.content_hash); + }); + }); + describe('crawl set', () => { it('requires the shared secret', async () => { const res = await handleCrawlSet( @@ -408,5 +456,97 @@ describe('feed timeline (D1 ingest + serve)', () => { expect(body.items).toEqual([]); expect(body.cursor).toBeGreaterThan(0); }); + + it('cold-starts a client whose cursor sits above the head (rewound archive)', async () => { + // The generation survives a Time Travel restore while the seqs rewind, so a + // cursor above the head is the only symptom — and `seq > cursor` would + // otherwise return nothing on every poll, forever. + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('r1'), contentHash: 'h1' }]); + const cold = await timeline(); + + const stalled = await timeline({ + since_seq: String(cold.cursor + 5000), + generation: cold.generation, + }); + expect(stalled.coldStart).toBe(true); + expect(stalled.items.map((i) => i.guid)).toEqual(['r1']); + expect(stalled.cursor).toBeLessThan(cold.cursor + 5000); + }); + + it('pages a large cold start and continues from nextColdOffset', async () => { + // 26 feeds × 30 items is past the per-page item budget, so the first page + // stops early and hands back a continuation index. + const feedUrls = Array.from({ length: 26 }, (_, i) => `https://example.com/paged${i}.xml`); + for (const feedUrl of feedUrls) { + await addSubscription(TEST_DID, feedUrl); + await ingest( + feedUrl, + Array.from({ length: 30 }, (_, i) => ({ + item: item(`${feedUrl}#${i}`), + contentHash: `${feedUrl}-${i}`, + })) + ); + } + + const first = await timeline(); + expect(first.coldStart).toBe(true); + expect(first.hasMore).toBe(true); + expect(first.nextColdOffset).toBeGreaterThan(0); + expect(first.items.length).toBeLessThan(26 * 30); + + const second = await timeline({ cold_offset: String(first.nextColdOffset) }); + expect(second.coldStart).toBe(true); + expect(second.hasMore).toBe(false); + expect(second.items.length).toBeGreaterThan(0); + + // Every feed is covered across the two pages, each with its newest slice. + const seen = new Set([...first.items, ...second.items].map((i) => i.feedUrl)); + expect(seen.size).toBe(26); + expect(first.items.length + second.items.length).toBe(26 * 30); + }); + }); + + describe('crawler liveness (ingestActive)', () => { + it('is false until the crawler checks in', async () => { + await addSubscription(TEST_DID, FEED_A); + await clearCrawlerHeartbeat(); + const body = await timeline(); + expect(body.ingestActive).toBe(false); + }); + + it('is true after an ingest push', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('h1'), contentHash: 'h1' }]); + expect((await timeline()).ingestActive).toBe(true); + }); + + it('is true after a crawl-set pull, even with nothing ingested', async () => { + await addSubscription(TEST_DID, FEED_A); + await clearCrawlerHeartbeat(); + const res = await handleCrawlSet( + new Request('https://api.example/api/internal/crawl-set', { + headers: { 'X-Proxy-Secret': SECRET }, + }), + env + ); + expect(res.status).toBe(200); + + const body = await timeline(); + expect(body.ingestActive).toBe(true); + expect(body.items).toEqual([]); + }); + + it('goes stale when the last heartbeat is old', async () => { + await addSubscription(TEST_DID, FEED_A); + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value` + ) + .bind(CRAWLER_HEARTBEAT_KEY, String(Math.floor(Date.now() / 1000) - 4 * 3600)) + .run(); + + expect((await timeline()).ingestActive).toBe(false); + }); }); }); diff --git a/backend/test/feeds-v2-fetch.spec.ts b/backend/test/feeds-v2-fetch.spec.ts new file mode 100644 index 00000000..1cb9f9c4 --- /dev/null +++ b/backend/test/feeds-v2-fetch.spec.ts @@ -0,0 +1,142 @@ +import { env } from 'cloudflare:test'; +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { handleV2FeedFetch } from '../src/routes/feeds-v2'; +import type { Env, Session } from '../src/types'; + +/** + * `GET /api/v2/feeds/fetch` reads the D1 archive, and pulls a feed through the + * crawler when the archive holds nothing for it. That pull-through is the one + * user-triggered write into a shared, never-pruned archive, so it is gated on the + * caller's own subscription. + */ + +const TEST_DID = 'did:plc:fetchtester'; +const FEED_URL = 'https://example.com/fetch-feed.xml'; + +const SESSION: Session = { + did: TEST_DID, + handle: 'fetch.bsky.social', + pdsUrl: 'https://test.pds.example', + accessToken: 'token', + refreshToken: 'refresh', + dpopPrivateKey: '{}', + expiresAt: Date.now() + 3600000, +}; + +function fetchRequest(params: Record): Request { + const search = new URLSearchParams(params).toString(); + return new Request(`https://api.example/api/v2/feeds/fetch?${search}`); +} + +function proxyFeed(guids: string[]): Response { + return new Response( + JSON.stringify({ + feed: { + title: 'Pulled Feed', + siteUrl: 'https://example.com', + // The proxy serves newest-first. + items: guids.map((guid) => ({ + guid, + url: `https://example.com/${guid}`, + title: `Title ${guid}`, + publishedAt: '2026-01-01T00:00:00.000Z', + })), + }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } } + ); +} + +async function subscribe(feedUrl: string) { + await env.DB.prepare( + `INSERT OR IGNORE INTO users (did, handle, pds_url, tier, created_at) + VALUES (?, 'fetch.test', 'https://test.pds.example', 'free', unixepoch())` + ) + .bind(TEST_DID) + .run(); + await env.DB.prepare( + `INSERT INTO subscriptions_cache (user_did, record_uri, feed_url, title, active) + VALUES (?, ?, ?, 'Feed', 1)` + ) + .bind(TEST_DID, `at://${TEST_DID}/app.skyreader.feed.subscription/${Date.now()}`, feedUrl) + .run(); +} + +describe('handleV2FeedFetch (archive + gated pull-through)', () => { + let originalFetch: typeof fetch; + let proxyCalls: string[]; + + beforeEach(() => { + originalFetch = globalThis.fetch; + proxyCalls = []; + (env as Env).FEED_PROXY_URL = 'https://proxy.example'; + (env as Env).FEED_PROXY_SECRET = 'test-secret'; + globalThis.fetch = vi.fn(async (input: RequestInfo | URL) => { + proxyCalls.push(typeof input === 'string' ? input : input.toString()); + return proxyFeed(['newest', 'middle', 'oldest']); + }) as unknown as typeof fetch; + }); + + afterEach(async () => { + globalThis.fetch = originalFetch; + vi.restoreAllMocks(); + await env.DB.prepare('DELETE FROM feed_items').run(); + await env.DB.prepare('DELETE FROM feeds').run(); + await env.DB.prepare('DELETE FROM subscriptions_cache').run(); + }); + + it('pulls a subscribed feed through the crawler and serves it newest-first', async () => { + await subscribe(FEED_URL); + + const res = await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION); + expect(res.status).toBe(200); + const body = (await res.json()) as { title: string; items: Array<{ guid: string }> }; + + expect(proxyCalls.length).toBe(1); + expect(body.title).toBe('Pulled Feed'); + expect(body.items.map((i) => i.guid)).toEqual(['newest', 'middle', 'oldest']); + + // Ingested oldest→newest, so the archive's seq order matches feed order. + const rows = await env.DB.prepare( + 'SELECT guid FROM feed_items WHERE feed_url = ? ORDER BY seq ASC' + ) + .bind(FEED_URL) + .all<{ guid: string }>(); + expect(rows.results.map((r) => r.guid)).toEqual(['oldest', 'middle', 'newest']); + }); + + it('does not write the shared archive for a feed the caller does not subscribe to', async () => { + const res = await handleV2FeedFetch( + fetchRequest({ url: FEED_URL, refresh: '1' }), + env, + SESSION + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { items: unknown[] }; + + expect(proxyCalls).toEqual([]); + expect(body.items).toEqual([]); + const count = await env.DB.prepare('SELECT COUNT(*) AS c FROM feed_items').first<{ + c: number; + }>(); + expect(count?.c).toBe(0); + }); + + it('serves the archive without a crawl once the feed is ingested', async () => { + await subscribe(FEED_URL); + await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION); + expect(proxyCalls.length).toBe(1); + + const res = await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION); + const body = (await res.json()) as { items: Array<{ guid: string }> }; + expect(proxyCalls.length).toBe(1); + expect(body.items.map((i) => i.guid)).toEqual(['newest', 'middle', 'oldest']); + }); + + it('re-crawls a subscribed feed on refresh=1', async () => { + await subscribe(FEED_URL); + await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION); + await handleV2FeedFetch(fetchRequest({ url: FEED_URL, refresh: '1' }), env, SESSION); + expect(proxyCalls.length).toBe(2); + }); +}); diff --git a/backend/wrangler.toml b/backend/wrangler.toml index 3243036f..892dbea1 100644 --- a/backend/wrangler.toml +++ b/backend/wrangler.toml @@ -69,10 +69,18 @@ vars = { FRONTEND_URL = "http://127.0.0.1:5173" } # Staging environment [env.staging] name = "skyreader-api-staging" -# Staging talks to its OWN Fly proxy (skyreader-feed-proxy-staging), which pushes -# only into the staging D1. Prod and staging are two fully isolated pairs — no -# cross-links, distinct PROXY_SECRET/FEED_PROXY_SECRET values. -vars = { FRONTEND_URL = "https://staging.skyreader.app", ALLOWED_ORIGINS = "https://staging.skyreader.app,https://staging-linkblogs.skyreader.app", FEED_PROXY_URL = "https://skyreader-feed-proxy-staging.fly.dev", LINKBLOG_PUBLIC_URL = "https://staging-linkblogs.skyreader.app" } +# Staging is MEANT to talk to its own Fly proxy (skyreader-feed-proxy-staging), +# which pushes only into the staging D1 — prod and staging as two fully isolated +# pairs, no cross-links, distinct PROXY_SECRET/FEED_PROXY_SECRET values. +# +# The URL below still points at the prod proxy on purpose: staging deploys on +# every push to main, so flipping it before the app exists would take /extract, +# discovery, standard.site documents and social context down on staging. Flip it +# to https://skyreader-feed-proxy-staging.fly.dev in the SAME deploy that rotates +# FEED_PROXY_SECRET, once Phase 3 of docs/plans/D1_FEED_TIMELINE.md has actually +# provisioned the app. Until then staging clients stay on the legacy batch path +# by themselves (the timeline reports ingestActive: false). +vars = { FRONTEND_URL = "https://staging.skyreader.app", ALLOWED_ORIGINS = "https://staging.skyreader.app,https://staging-linkblogs.skyreader.app", FEED_PROXY_URL = "https://skyreader-feed-proxy.fly.dev", LINKBLOG_PUBLIC_URL = "https://staging-linkblogs.skyreader.app" } routes = [{ pattern = "api-staging.skyreader.app", custom_domain = true }] [env.staging.durable_objects] diff --git a/docs/plans/D1_FEED_TIMELINE.md b/docs/plans/D1_FEED_TIMELINE.md index e1dcd7e2..ade987ba 100644 --- a/docs/plans/D1_FEED_TIMELINE.md +++ b/docs/plans/D1_FEED_TIMELINE.md @@ -46,12 +46,21 @@ read state (`getReadKeys`). Now: authenticated by a constant-time compare against `FEED_PROXY_SECRET` and **fail-closed** when it is unset. Idempotent upsert (edit-in-place keeps the seq), per-item content cap (`MAX_ITEM_CONTENT_BYTES = 8 KB`, drops `content` and sets `contentTruncated`), and the per-feed - `SANITY_CAP = 5000` trim — the only pruning that ever runs. -- `routes/timeline.ts`: `GET /api/v2/timeline?since_seq=&generation=&limit=` — incremental drain - (cursor derived from returned rows, `hasMore` via `limit+1`) and a per-feed newest-30 cold start. - Read state is an `EXISTS` probe in the same query; `getReadKeys` is never called on the feed path. + `SANITY_CAP = 5000` trim — the only pruning that ever runs. Both endpoints stamp a **crawler + heartbeat** (`sync_state.crawler_heartbeat_at`); the crawl-set pull runs every 5 minutes, so the + stamp is fresh even when no feed produced an item. +- `routes/timeline.ts`: `GET /api/v2/timeline?since_seq=&generation=&limit=&cold_offset=` — + incremental drain (cursor derived from returned rows, `hasMore` via `limit+1`) and a **paged** + per-feed newest-30 cold start (feeds walked in a stable order, `COLD_START_MAX_ITEMS` per page, + continuation via `nextColdOffset`). Read state is an `EXISTS` probe in the same query; + `getReadKeys` is never called on the feed path. Every response carries `ingestActive`, derived + from the heartbeat: false means this deployment has no crawler filling D1, and clients stay on + the legacy batch path. A cursor above the archive head cold-starts (rewound-archive guard). - `GET /api/v2/feeds/fetch` re-backed with D1 + **pull-through**: if a feed isn't in the archive - yet (first subscriber), fetch it from the proxy once, ingest it, then serve. + yet (first subscriber), fetch it from the proxy once, ingest it, then serve. The pull-through is + gated on the caller's own subscription, so the shared never-pruned archive can't be written with + arbitrary feeds. Subscribe time already crawls + ingests the feed (`warmFeedIntoArchive`, which + replaced the old warm-and-discard), so the pull-through is normally not needed at all. **Proxy** (`feed-proxy/`) @@ -69,10 +78,18 @@ read state (`getReadKeys`). Now: - `feedFetcher.fetchAllFeeds` prefers one `GET /api/v2/timeline` (plus drain pages) with a single global cursor in Dexie `metadata` (`timelineCursor`). Pure helpers live in `timelineSync.ts`. - Falls back to the legacy `/api/v2/feeds/batch` path when the timeline 404s (old or rolled-back - backend) **or** when a cold start returns nothing for a subscribed user (the environment's - crawler isn't pushing yet). The cursor is never committed in that case. -- OPML import now backfills via the per-feed endpoint: a freshly imported feed's items sit below - the global cursor, so only the single-feed path (with its pull-through) can deliver them. + backend) **or** whenever the server reports `ingestActive: false`. The cursor is never committed + in that case. (An `ingestActive`-less backend keeps the old empty-cold-start heuristic.) +- A paged cold start commits only the FIRST page's cursor, and only after its last page merges, so + an interrupted bootstrap starts over instead of skipping feeds it never delivered. +- Subscriptions that arrive from another device sit below the global cursor, so each is backfilled + once through the per-feed endpoint (`backfillMissingSubscriptions`, ≤ 10 per sync, attempts + recorded in Dexie `metadata.timelineBackfilledFeeds`). +- OPML import backfills via the per-feed endpoint: a freshly imported feed's items sit below the + global cursor, so only the single-feed path can deliver them. The requests are paced (3 at a + time, 1 s apart) and no longer force a crawl, so a 250-feed import stays inside the rate limit. +- An article whose body was dropped at ingest (`contentTruncated`) is extracted automatically when + its card opens, so the reader shows the whole article rather than an RSS summary. **Admin** (`admin/`) — feed health is re-pointed at `feeds`/`feed_items`: crawled feeds, subscribed feeds not ingesting (the R1 alarm), archived item count, estimated archive size with a 6 GB alert, @@ -109,8 +126,10 @@ cd feed-proxy && fly deploy --remote-only --config fly.staging.toml ``` Sequencing matters: today's staging Worker authenticates to the **prod** proxy with prod's secret, -so provision + deploy the staging proxy first, then flip `FEED_PROXY_URL` (already committed) and -rotate `FEED_PROXY_SECRET` together in one staging Worker deploy. +so provision + deploy the staging proxy first, then flip `FEED_PROXY_URL` in +`backend/wrangler.toml` (`[env.staging]`, still pointing at the prod proxy on purpose) and rotate +`FEED_PROXY_SECRET` together in one staging Worker deploy. The CI staging Fly job skips itself with +a notice until the app exists, so nothing breaks in the meantime. Verify: `fly status -a skyreader-feed-proxy-staging` shows exactly **one** machine (the singleton invariant applies to this app too — never `fly scale count`); `/stats` answers and its @@ -119,8 +138,11 @@ logs show no staging-origin traffic. **Then enable prod:** uncomment `INGEST_URL` in `feed-proxy/fly.toml` and cut a release. Prod's backfill drains through the normal pusher loop (≤ 200 × active feeds at 100 items / 15 s). Until -that happens, prod clients keep using the legacy batch path automatically — the frontend's -empty-archive fallback covers exactly this window. +that happens, prod clients keep using the legacy batch path automatically: with no crawler pushing, +nothing stamps `crawler_heartbeat_at`, so `/api/v2/timeline` answers `ingestActive: false` and no +client commits a cursor. That signal is server-side on purpose — subscribe-time ingest and the +pull-through both write to the archive, so "the archive is empty for this user" would stop being +true long before the crawler existed. ### Phase 5 — cleanup (a later release, once no legacy traffic remains) @@ -132,10 +154,31 @@ step deleting `feeds`/`feed_items` rows whose feed has had **zero active subscri ## Invariants worth keeping -- **Cursor from returned rows, never `MAX(seq)`** — the latter races ingest and silently skips rows. +- **The incremental cursor comes from returned rows, never `MAX(seq)`** — the latter races ingest + and silently skips rows. A cold start is the one exception, and only because it reads the head + BEFORE its per-feed slices: anything ingested while it pages lands above that head and arrives on + the next poll. - **Any D1 restore bumps `items_generation`** (one `UPDATE sync_state`): Time Travel rewinds seqs - while the token would otherwise stay the same, so clients would sit above the head forever. + while the token would otherwise stay the same. The timeline also self-heals a cursor that sits + above the head by cold-starting that client, so a forgotten bump degrades to one extra cold start + rather than a silent, permanent stall. +- **Ingest order is oldest→newest.** A proxy feed is newest-first; seq is assigned in insert order, + so the pull-through walks it backwards (as the proxy's `writeFeedItems` does). Inverting it would + never heal — a re-push of an unchanged item is a no-op. +- **Writes to the archive need a subscription.** Ordinary ingest deletes nothing, so every + user-triggered write path (subscribe-time ingest, the pull-through) is gated on the caller's own + subscription list. - **The pusher sends `cache.url`**, the registered URL, never a post-redirect one: the timeline joins on that exact string. - **Ordinary ingest deletes nothing.** A feed at the sanity cap is a bug signal (GUID churn), not steady state — investigate the feed rather than letting it rotate. + +## Known scaling knob + +The incremental drain scans the `feed_items` rowid range above the caller's cursor and probes the +subscription set per row, so its cost tracks **global** ingest above the cursor rather than the +caller's own new items: a 5-feed reader returning after a week pays for everything the whole system +ingested that week. That is the accepted fan-out-on-read trade at ~1,330 feeds. If D1 row-reads or +CPU ever become the constraint, bound the scan with per-feed `(feed_url, seq)` seeks against the +subscription set (`idx_feed_items_feed_seq` already supports them) rather than materializing +per-user timelines. diff --git a/e2e/seed.ts b/e2e/seed.ts index dac8afbd..7056f54a 100644 --- a/e2e/seed.ts +++ b/e2e/seed.ts @@ -159,6 +159,10 @@ export async function seedFeedItems( const statements = [ `INSERT OR REPLACE INTO feeds (feed_url, title, site_url, last_ingest_at, created_at) VALUES (${sqlString(feedUrl)}, ${sqlNullableString(opts.title ?? null)}, ${sqlNullableString(opts.siteUrl ?? null)}, ${nowSeconds}, ${nowSeconds})`, + // Stand in for the crawler's check-in. Without a fresh heartbeat the timeline + // reports `ingestActive: false` and the client (correctly) stays on the legacy + // batch path, which is not what these tests are exercising. + `INSERT INTO sync_state (key, value, updated_at) VALUES ('crawler_heartbeat_at', ${sqlString(String(nowSeconds))}, ${nowSeconds}) ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`, ]; items.forEach((item, index) => { @@ -182,6 +186,7 @@ export async function cleanupFeedItems(feedUrl: string): Promise { await execD1([ `DELETE FROM feed_items WHERE feed_url = ${sqlString(feedUrl)}`, `DELETE FROM feeds WHERE feed_url = ${sqlString(feedUrl)}`, + `DELETE FROM sync_state WHERE key = 'crawler_heartbeat_at'`, ]); } diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 2d9cd820..4395264f 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -39,26 +39,30 @@ All stores use Svelte 5 runes (`.svelte.ts` files): ### Services -| Service | Purpose | -| --------------- | -------------------------------------------------- | -| `api.ts` | HTTP client for backend API | -| `db.ts` | Dexie (IndexedDB) schema for offline storage | -| `feedFetcher.ts` | Feed refresh (timeline sync + legacy batch path) | -| `timelineSync.ts`| Pure helpers for the timeline sync (unit-tested) | -| `sync-queue.ts` | Queue operations when offline, process when online | +| Service | Purpose | +| ----------------- | -------------------------------------------------- | +| `api.ts` | HTTP client for backend API | +| `db.ts` | Dexie (IndexedDB) schema for offline storage | +| `feedFetcher.ts` | Feed refresh (timeline sync + legacy batch path) | +| `timelineSync.ts` | Pure helpers for the timeline sync (unit-tested) | +| `sync-queue.ts` | Queue operations when offline, process when online | +| `realtime.ts` | WebSocket connection management | ### Feed refresh A refresh is **one** `GET /api/v2/timeline` request (plus drain pages while `hasMore`), served from the backend's D1 archive with read state already stamped on each item. The client holds a single global cursor in Dexie `metadata` (`timelineCursor` = `{cursor, generation}`), committed only after -a successful merge; a `generation` change cold-starts. New subscriptions are backfilled through -`fetchSingleFeed` → `GET /api/v2/feeds/fetch`, since their items sit below the global cursor. +a successful merge; a `generation` change cold-starts. A cold start is paged (`cold_offset`), and +commits the first page's cursor only once the last page has merged. New subscriptions are +backfilled through `fetchSingleFeed` → `GET /api/v2/feeds/fetch`, since their items sit below the +global cursor — including ones that arrive from another device (`backfillMissingSubscriptions`, +once per feed, ≤ 10 per sync). The legacy per-feed `/api/v2/feeds/batch` path (`fetchAllFeedsViaBatch`, with per-subscription -`feedCursors`) is kept for one release as a fallback: it runs when the timeline 404s or when a cold -start returns nothing for a subscribed user. See `docs/plans/D1_FEED_TIMELINE.md`. -| `realtime.ts` | WebSocket connection management | +`feedCursors`) is kept for one release as a fallback: it runs when the timeline 404s and whenever +the server reports `ingestActive: false` (this environment's crawler isn't pushing into D1). See +`docs/plans/D1_FEED_TIMELINE.md`. ### Key Routes diff --git a/frontend/src/lib/components/ArticleCard.svelte b/frontend/src/lib/components/ArticleCard.svelte index 7070db1c..51d6d033 100644 --- a/frontend/src/lib/components/ArticleCard.svelte +++ b/frontend/src/lib/components/ArticleCard.svelte @@ -594,6 +594,19 @@ }; }); + // The archive drops oversized bodies at ingest (per-item content cap) and marks + // the item `contentTruncated`. Without this the reader would silently show the + // RSS summary — often two sentences — in place of a full-text post. So when + // such an article opens, extract the original automatically; the result flows + // into displayContent above (fetchedOriginal wins) and the extract cache means + // it happens once per URL. + // Gated on `expanded` rather than `isOpen`: a keyboard cursor moving across the + // list shouldn't fire an extraction per card it passes over. + $effect(() => { + if (!expanded || !article?.contentTruncated || !itemUrl) return; + linkPostContentStore.fetch(itemUrl); + }); + // Same lazy-load for a document's flat text (stripped from memory). Only // fetched when the document carries no in-memory textContent — i.e. a // stripped social-feed doc — and read back by recordUri. diff --git a/frontend/src/lib/components/ImportOPMLModal.svelte b/frontend/src/lib/components/ImportOPMLModal.svelte index 60ab44bc..f473f987 100644 --- a/frontend/src/lib/components/ImportOPMLModal.svelte +++ b/frontend/src/lib/components/ImportOPMLModal.svelte @@ -125,9 +125,10 @@ // Get the newly added subscriptions const newSubs = liveDb.subscriptions.filter((s) => result.added.includes(s.id!)); - // Fetch each new feed directly in the background. A freshly imported feed's - // items sit below the global timeline cursor, so the per-feed endpoint (not - // the timeline) is what backfills its history. + // Backfill each new feed in the background. A freshly imported feed's items + // sit below the global timeline cursor, so the per-feed endpoint (not the + // timeline) is what delivers its history. fetchNewSubscriptionFeeds paces + // the requests, so a large import stays inside the endpoint's rate limit. fetchNewSubscriptionFeeds(newSubs, articlesStore.savedGuids); } } diff --git a/frontend/src/lib/services/api.ts b/frontend/src/lib/services/api.ts index f71eb325..af2adcaa 100644 --- a/frontend/src/lib/services/api.ts +++ b/frontend/src/lib/services/api.ts @@ -97,6 +97,11 @@ export interface TimelineResponse { // True when the server served a per-feed newest slice instead of draining from // a cursor (no cursor sent, or the generation no longer matches). coldStart: boolean; + // Continuation index for a paged cold start; echo it back as `cold_offset`. + nextColdOffset?: number; + // Whether this deployment's crawler is actually pushing into the archive. + // Absent on a backend that predates the flag. + ingestActive?: boolean; // Feed-level metadata for the caller's subscriptions; present only on a page // that carried items. feeds?: Record; @@ -332,11 +337,13 @@ class ApiClient { since_seq?: number; generation?: string; limit?: number; + cold_offset?: number; }): Promise { const search = new URLSearchParams(); if (params.since_seq !== undefined) search.set('since_seq', String(params.since_seq)); if (params.generation) search.set('generation', params.generation); if (params.limit) search.set('limit', String(params.limit)); + if (params.cold_offset) search.set('cold_offset', String(params.cold_offset)); const query = search.toString(); return this.fetch(`/api/v2/timeline${query ? `?${query}` : ''}`); } diff --git a/frontend/src/lib/services/articleMerge.ts b/frontend/src/lib/services/articleMerge.ts index e619cd35..51ab282c 100644 --- a/frontend/src/lib/services/articleMerge.ts +++ b/frontend/src/lib/services/articleMerge.ts @@ -84,6 +84,9 @@ export function selectNewArticles( imageUrl: item.imageUrl, publishedAt: item.publishedAt, fetchedAt: now, + // Carried through so the reader knows to extract the full text on open + // (the archive dropped an oversized body at ingest). + contentTruncated: item.contentTruncated || undefined, }); } } diff --git a/frontend/src/lib/services/feedFetcher.ts b/frontend/src/lib/services/feedFetcher.ts index db0559ec..9b005cfc 100644 --- a/frontend/src/lib/services/feedFetcher.ts +++ b/frontend/src/lib/services/feedFetcher.ts @@ -9,6 +9,8 @@ import { buildSubscriptionIndex, groupTimelineItems, isRssSubscription, + pruneAttemptedBackfills, + selectBackfillTargets, shouldFallBackToBatch, shouldUpdateTitle, subscriptionMetaUpdate, @@ -49,9 +51,22 @@ export interface FetchResult { // one per subscription — that's the whole point of the timeline). const TIMELINE_CURSOR_KEY = 'timelineCursor'; +// Dexie `metadata` key holding the feed URLs we've already tried to backfill +// through the per-feed endpoint (see backfillMissingSubscriptions). +const TIMELINE_BACKFILL_KEY = 'timelineBackfilledFeeds'; + // Items per timeline page (the server caps at 200 too). const TIMELINE_PAGE_LIMIT = 200; +// Cold-start pages are per-feed slices bounded by a server-side item budget, so a +// large subscription list takes several of them. A cold start is a one-time +// bootstrap, so it gets a much larger round budget than an incremental drain. +const MAX_COLD_START_ROUNDS = 25; + +// Per-sync cap on the one-off per-feed backfills for subscriptions that arrived +// from another device. The rest continue on the next sync. +const MAX_BACKFILLS_PER_SYNC = 10; + interface TimelineCursor { cursor: number; generation: string; @@ -68,9 +83,9 @@ let timelineUnavailable = false; * * Returns null when the caller should fall back to the legacy per-feed batch * path — either the endpoint doesn't exist (old/rolled-back backend) or the - * server-side archive has nothing for this user yet (ingest not enabled in this - * environment). The cursor is never committed in that case, so a later switch to - * the timeline still cold-starts correctly. + * server says its crawler isn't filling the archive (`ingestActive: false`, the + * whole pre-rollout window). The cursor is never committed in that case, so a + * later switch to the timeline still cold-starts correctly. */ async function fetchTimeline( subscriptions: Subscription[], @@ -93,14 +108,37 @@ async function fetchTimeline( let cursor = stored?.cursor; let generation = stored?.generation; - for (let round = 0; round < MAX_DRAIN_ROUNDS; round++) { + // Cold-start paging state. The server hands back a continuation index into the + // caller's feed list; the cursor to keep is the FIRST page's (the archive head + // as of the moment the cold start began), committed only once the last page has + // merged — a cold start abandoned halfway simply starts over next sync rather + // than committing a cursor past feeds it never delivered. + let coldOffset: number | undefined; + let coldCursor: TimelineCursor | undefined; + let incrementalRounds = 0; + let coldRounds = 0; + // Set when we stop because of a round cap rather than because we're done. + let cappedOut = false; + + for (;;) { + if ( + coldOffset === undefined + ? incrementalRounds >= MAX_DRAIN_ROUNDS + : coldRounds >= MAX_COLD_START_ROUNDS + ) { + cappedOut = true; + break; + } + if (coldOffset === undefined) incrementalRounds++; + else coldRounds++; + let page; try { - page = await api.fetchTimeline({ - since_seq: cursor, - generation, - limit: TIMELINE_PAGE_LIMIT, - }); + page = await api.fetchTimeline( + coldOffset === undefined + ? { since_seq: cursor, generation, limit: TIMELINE_PAGE_LIMIT } + : { cold_offset: coldOffset, limit: TIMELINE_PAGE_LIMIT } + ); } catch (e) { if (e instanceof ApiError && e.status === 404) { // Backend predates the timeline (or was rolled back): stay on /batch. @@ -110,9 +148,8 @@ async function fetchTimeline( throw e; } - // A cold start that finds nothing while the user has subscriptions means the - // archive isn't populated for them yet (this environment's crawler isn't - // pushing). Fall back rather than showing an empty reader. + // The server says whether this environment's crawler is actually filling the + // archive. Until it is, use the legacy batch path and commit no cursor. if (shouldFallBackToBatch(page, rssSubs.length)) return null; if (page.readCursor) await itemLabelsStore.seedReadCursor(page.readCursor); @@ -146,29 +183,84 @@ async function fetchTimeline( } } - // Merge succeeded — commit the cursor. - cursor = page.cursor; - generation = page.generation; - await setMetadata(TIMELINE_CURSOR_KEY, { cursor, generation }); - // A feed that just delivered items is demonstrably healthy; clear any stale // error state. Feeds that delivered nothing are left alone — the archive // carries no per-feed fetch status (that lives with the crawler). for (const feedUrl of feedUrls) feedStatusStore.markReady(feedUrl); - if (!page.hasMore) { - result.successfulFeeds = rssSubs.length; - return result; + if (page.coldStart) { + // Keep the first cold page's cursor; page through the rest before committing. + if (!coldCursor) coldCursor = { cursor: page.cursor, generation: page.generation }; + if (page.hasMore && page.nextColdOffset != null) { + coldOffset = page.nextColdOffset; + continue; + } + await setMetadata(TIMELINE_CURSOR_KEY, coldCursor); + coldOffset = undefined; + break; } + + // Merge succeeded — commit the cursor. + cursor = page.cursor; + generation = page.generation; + await setMetadata(TIMELINE_CURSOR_KEY, { cursor, generation }); + + if (!page.hasMore) break; } - console.warn( - `[feedFetcher] Timeline drain cap (${MAX_DRAIN_ROUNDS} rounds) reached; the rest continues on the next sync.` - ); + if (cappedOut) { + console.warn( + coldOffset !== undefined + ? `[feedFetcher] Cold-start page cap (${MAX_COLD_START_ROUNDS} rounds) reached; the rest continues on the next sync.` + : `[feedFetcher] Timeline drain cap (${MAX_DRAIN_ROUNDS} rounds) reached; the rest continues on the next sync.` + ); + } + + // Subscriptions that arrived from another device sit below the global cursor, + // so only the per-feed endpoint can deliver their history. + result.newArticles += await backfillMissingSubscriptions(rssSubs, savedGuids); + result.successfulFeeds = rssSubs.length; return result; } +/** + * One-off per-feed backfill for subscriptions the timeline can't reach. + * + * A subscription synced in from another device is already below the global + * cursor, so the drain will never deliver its existing items — the legacy path + * got this for free because every subscription carried its own cursor. Each feed + * is attempted once (the attempt is recorded in Dexie), and only a handful per + * sync, so this can't turn into a request storm after a large sync. + */ +async function backfillMissingSubscriptions( + rssSubs: Subscription[], + savedGuids: Set +): Promise { + const hasArticles = (sub: Subscription) => liveDb.getRecentGuids(sub.id!, 1).length > 0; + // Steady state: every subscription holds articles, so this costs one in-memory + // scan and never touches Dexie metadata. + if (!rssSubs.some((sub) => sub.id && !hasArticles(sub))) return 0; + + const attempted = new Set((await getMetadata(TIMELINE_BACKFILL_KEY)) ?? []); + const targets = selectBackfillTargets(rssSubs, attempted, hasArticles, MAX_BACKFILLS_PER_SYNC); + if (targets.length === 0) return 0; + + let newArticles = 0; + for (const sub of targets) { + // Not `force`: the archive read is already current, and the backend still + // pulls the feed through the crawler if it holds nothing for it. + const fetched = await fetchSingleFeed(sub, false, savedGuids); + newArticles += fetched.newArticles; + // Recorded either way — a feed that legitimately has no items must not be + // re-requested every sync. The sidebar's retry action covers a real failure. + attempted.add(sub.feedUrl!); + } + + await setMetadata(TIMELINE_BACKFILL_KEY, pruneAttemptedBackfills(attempted, rssSubs)); + return newArticles; +} + /** * Fetch all subscribed feeds. * @@ -602,9 +694,21 @@ export async function fetchSingleFeed( } } +// Pacing for the new-subscription backfill (OPML import is the big case: one +// per-feed request each). Small concurrent groups with a pause between them keep +// a 250-feed import comfortably under the endpoint's per-minute limit — the old +// unpaced loop would 429 partway through and finish with a screen of "broken" +// feeds that were only rate-limited. +const NEW_SUBSCRIPTION_CONCURRENCY = 3; +const NEW_SUBSCRIPTION_DELAY_MS = 1000; + /** - * Fetch feeds for newly added subscriptions - * These are fetched one by one since they don't have any cached content yet + * Fetch feeds for newly added subscriptions. + * + * These need the per-feed endpoint: their items sit below the global timeline + * cursor, so the timeline alone would never deliver them. The backend already + * crawled and ingested the first feeds at subscribe time, so this is normally a + * plain archive read — no `force`, which would make each one a synchronous crawl. * * @param subscriptions - New subscriptions to fetch * @param savedGuids - Set of starred article GUIDs @@ -622,16 +726,25 @@ export async function fetchNewSubscriptionFeeds( newArticles: 0, }; - for (let i = 0; i < subscriptions.length; i++) { - const sub = subscriptions[i]; + for (let i = 0; i < subscriptions.length; i += NEW_SUBSCRIPTION_CONCURRENCY) { onProgress?.(i, subscriptions.length); + const group = subscriptions.slice(i, i + NEW_SUBSCRIPTION_CONCURRENCY); - const fetchResult = await fetchSingleFeed(sub, true, savedGuids); - if (fetchResult.success) { - result.successfulFeeds++; - result.newArticles += fetchResult.newArticles; - } else { - result.failedFeeds++; + const groupResults = await Promise.allSettled( + group.map((sub) => fetchSingleFeed(sub, false, savedGuids)) + ); + + for (const groupResult of groupResults) { + if (groupResult.status === 'fulfilled' && groupResult.value.success) { + result.successfulFeeds++; + result.newArticles += groupResult.value.newArticles; + } else { + result.failedFeeds++; + } + } + + if (i + NEW_SUBSCRIPTION_CONCURRENCY < subscriptions.length) { + await new Promise((resolve) => setTimeout(resolve, NEW_SUBSCRIPTION_DELAY_MS)); } } diff --git a/frontend/src/lib/services/timelineSync.test.ts b/frontend/src/lib/services/timelineSync.test.ts index 6c38e7eb..68a1a9fe 100644 --- a/frontend/src/lib/services/timelineSync.test.ts +++ b/frontend/src/lib/services/timelineSync.test.ts @@ -3,6 +3,8 @@ import { buildSubscriptionIndex, groupTimelineItems, isRssSubscription, + pruneAttemptedBackfills, + selectBackfillTargets, shouldFallBackToBatch, shouldUpdateTitle, subscriptionMetaUpdate, @@ -99,7 +101,27 @@ describe('groupTimelineItems', () => { }); describe('shouldFallBackToBatch', () => { - it('falls back when a cold start finds nothing for a subscribed user', () => { + it('falls back whenever the server says nothing is ingesting, however full the page', () => { + // The rollout window: one subscribe-time ingest is enough to make a cold + // start non-empty while nothing crawls the user's other feeds. + expect( + shouldFallBackToBatch({ items: [tItem()], coldStart: true, ingestActive: false }, 3) + ).toBe(true); + expect( + shouldFallBackToBatch({ items: [tItem()], coldStart: false, ingestActive: false }, 3) + ).toBe(true); + }); + + it('stays on the timeline once the crawler is live, even on an empty page', () => { + expect(shouldFallBackToBatch({ items: [], coldStart: true, ingestActive: true }, 3)).toBe( + false + ); + expect(shouldFallBackToBatch({ items: [], coldStart: false, ingestActive: true }, 3)).toBe( + false + ); + }); + + it('falls back when a cold start finds nothing for a subscribed user (pre-flag backend)', () => { expect(shouldFallBackToBatch({ items: [], coldStart: true }, 3)).toBe(true); }); @@ -116,6 +138,37 @@ describe('shouldFallBackToBatch', () => { }); }); +describe('selectBackfillTargets', () => { + const subA = sub({ id: 1, feedUrl: FEED_A }); + const subB = sub({ id: 2, feedUrl: FEED_B, title: 'Feed B' }); + const none = () => false; + + it('picks subscriptions that hold no articles yet', () => { + expect(selectBackfillTargets([subA, subB], new Set(), none, 10)).toEqual([subA, subB]); + }); + + it('skips feeds already tried, feeds with articles, and non-RSS sources', () => { + const atproto = sub({ id: 3, feedUrl: 'at://did:plc:x/pub', sourceType: 'atproto.documents' }); + const targets = selectBackfillTargets( + [subA, subB, atproto], + new Set([FEED_A]), + (s) => s.id === 2, + 10 + ); + expect(targets).toEqual([]); + }); + + it('caps how many it returns per sync', () => { + expect(selectBackfillTargets([subA, subB], new Set(), none, 1)).toEqual([subA]); + }); +}); + +describe('pruneAttemptedBackfills', () => { + it('keeps only feeds still subscribed', () => { + expect(pruneAttemptedBackfills([FEED_A, FEED_B], [sub({ feedUrl: FEED_A })])).toEqual([FEED_A]); + }); +}); + describe('shouldUpdateTitle', () => { it('replaces a URL or hostname placeholder with a real title', () => { expect(shouldUpdateTitle(FEED_A, FEED_A, 'Real Title')).toBe(true); diff --git a/frontend/src/lib/services/timelineSync.ts b/frontend/src/lib/services/timelineSync.ts index 69c5d551..9e5e8c87 100644 --- a/frontend/src/lib/services/timelineSync.ts +++ b/frontend/src/lib/services/timelineSync.ts @@ -13,6 +13,9 @@ export type TimelineItem = FeedItem & { seq: number; feedUrl: string; read: bool export interface TimelinePageShape { items: TimelineItem[]; coldStart: boolean; + // Server-authoritative: is this deployment's crawler pushing into the archive? + // Absent on a backend that predates the flag. + ingestActive?: boolean; } /** Feed metadata the timeline carries alongside a non-empty page. */ @@ -84,15 +87,66 @@ export function groupTimelineItems( * Whether to abandon the timeline for this sync and use the legacy per-feed * batch path instead. * - * A cold start that finds nothing while the user has subscriptions means the - * server-side archive isn't populated for them yet — the environment's crawler - * isn't pushing into D1. Showing an empty reader would be wrong; falling back - * (without committing a cursor) keeps the reader working until ingest catches up. + * The decisive signal is the server's own `ingestActive`: it is false until this + * deployment's crawler has actually checked in, so a Worker whose proxy has no + * `INGEST_URL` (the whole pre-rollout window) tells every client to stay on the + * batch path — no cursor is committed, and nothing infers "healthy" from a page + * that happened to carry items. Emptiness alone can't decide that: one + * subscribe-time ingest is enough to make a cold start non-empty while the + * user's other feeds are never crawled at all. + * + * The emptiness heuristic remains only for a backend that predates the flag. */ export function shouldFallBackToBatch(page: TimelinePageShape, subscriptionCount: number): boolean { + if (page.ingestActive === false) return true; + if (page.ingestActive === true) return false; return page.coldStart && page.items.length === 0 && subscriptionCount > 0; } +/** + * Subscriptions that need a one-off per-feed backfill. + * + * The timeline's cursor is global, so a subscription that arrives from ANOTHER + * device (synced in from the backend/PDS) is already below it: the drain will + * never deliver its existing items, and the reader would show it empty until the + * crawler happens to publish something new. The add-feed and OPML paths call the + * per-feed endpoint explicitly; this covers the remote-sync path. + * + * `attempted` holds the feed URLs already tried (persisted), so a genuinely empty + * feed is fetched once rather than on every sync. Bounded per sync so a large + * incoming subscription list doesn't turn into a request storm. + */ +export function selectBackfillTargets( + subscriptions: Subscription[], + attempted: Set, + hasArticles: (sub: Subscription) => boolean, + max: number +): Subscription[] { + const targets: Subscription[] = []; + for (const sub of subscriptions) { + if (targets.length >= max) break; + if (!sub.id || !isRssSubscription(sub)) continue; + if (attempted.has(sub.feedUrl!)) continue; + if (hasArticles(sub)) continue; + targets.push(sub); + } + return targets; +} + +/** + * Keep the attempted-backfill record to feeds the user still subscribes to, so + * it can't grow without bound as feeds come and go. + */ +export function pruneAttemptedBackfills( + attempted: Iterable, + subscriptions: Subscription[] +): string[] { + const live = new Set( + subscriptions.filter((s) => isRssSubscription(s)).map((s) => s.feedUrl as string) + ); + return [...attempted].filter((url) => live.has(url)); +} + /** * Whether a subscription's title should be updated from feed metadata. * Returns true when the current title is a fallback (URL, hostname, etc.) diff --git a/frontend/src/lib/types/index.ts b/frontend/src/lib/types/index.ts index 6b9cf4ab..d468f879 100644 --- a/frontend/src/lib/types/index.ts +++ b/frontend/src/lib/types/index.ts @@ -47,6 +47,10 @@ export interface Article { imageUrl?: string; publishedAt: string; fetchedAt: number; + // The archive dropped this item's body at ingest (over the per-item content + // cap), so `content` here is absent or just the RSS summary. The reader + // extracts the full text on demand when the card opens. + contentTruncated?: boolean; // Precomputed body stats. The full `content` HTML is dropped from the // in-memory copy of an article (see toLightArticle) to keep the heap small — // it stays in IndexedDB and is lazy-loaded on expand. These numbers let the From b2205a4e6afc8544084f56a2c549a64724454026 Mon Sep 17 00:00:00 2001 From: "codexbot.disnetdev.com (did:plc:hbonvqr5ysrscg5wdyb5klie)" Date: Thu, 13 Aug 2026 00:07:02 +0000 Subject: [PATCH 3/7] Fix timeline rollout edge cases Co-Authored-By: codexbot.disnetdev.com (did:plc:hbonvqr5ysrscg5wdyb5klie) --- backend/migrations/0061_feed_timeline.sql | 2 ++ backend/src/routes/timeline.ts | 4 +-- backend/test/feed-timeline.spec.ts | 30 +++++++++++++++++++ docs/plans/D1_FEED_TIMELINE.md | 12 ++++++-- .../src/lib/components/ArticleCard.svelte | 6 +++- frontend/src/lib/services/feedFetcher.ts | 11 +++---- 6 files changed, 54 insertions(+), 11 deletions(-) diff --git a/backend/migrations/0061_feed_timeline.sql b/backend/migrations/0061_feed_timeline.sql index 7baf881a..620c26d1 100644 --- a/backend/migrations/0061_feed_timeline.sql +++ b/backend/migrations/0061_feed_timeline.sql @@ -43,6 +43,8 @@ CREATE TABLE feed_items ( ); CREATE INDEX idx_feed_items_feed_seq ON feed_items(feed_url, seq); +CREATE INDEX idx_feed_items_feed_published_seq + ON feed_items(feed_url, published_at DESC, seq DESC); -- The timeline join probes subscriptions by (user_did, feed_url). CREATE INDEX IF NOT EXISTS idx_subscriptions_cache_user_feed diff --git a/backend/src/routes/timeline.ts b/backend/src/routes/timeline.ts index cb54d000..9b644221 100644 --- a/backend/src/routes/timeline.ts +++ b/backend/src/routes/timeline.ts @@ -89,7 +89,7 @@ export async function readFeedSlice( `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} FROM feed_items fi WHERE fi.feed_url = ?2 - ORDER BY fi.seq DESC + ORDER BY fi.published_at DESC, fi.seq DESC LIMIT ?3` ) .bind(userDid, feedUrl, limit) @@ -340,7 +340,7 @@ export async function handleTimeline( `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} FROM feed_items fi WHERE fi.feed_url = ?2 - ORDER BY fi.seq DESC + ORDER BY fi.published_at DESC, fi.seq DESC LIMIT ?3` ).bind(session.did, feedUrl, COLD_START_PER_FEED) ); diff --git a/backend/test/feed-timeline.spec.ts b/backend/test/feed-timeline.spec.ts index 7008439f..2540c8d3 100644 --- a/backend/test/feed-timeline.spec.ts +++ b/backend/test/feed-timeline.spec.ts @@ -322,6 +322,36 @@ describe('feed timeline (D1 ingest + serve)', () => { expect(rows.results[0].seq).toBe(before?.seq); expect(rows.results[0].content_hash).toBe(before?.content_hash); }); + + it('serves newest publications when pull-through and crawler ingest interleave', async () => { + // Subscribe-time pull-through can write the newest proxy window before + // the crawler's initial backlog reaches this feed. The older backlog then + // has higher seq values, so per-feed reads must not use seq as recency. + await ingestProxyFeed(env, FEED_A, { + title: 'Pulled', + items: [ + item('newest', { publishedAt: '2026-03-01T00:00:00.000Z' }), + item('newer', { publishedAt: '2026-02-01T00:00:00.000Z' }), + ], + }); + await ingest(FEED_A, [ + { + item: item('oldest', { publishedAt: '2025-12-01T00:00:00.000Z' }), + contentHash: 'oldest-hash', + }, + { + item: item('older', { publishedAt: '2026-01-01T00:00:00.000Z' }), + contentHash: 'older-hash', + }, + ]); + + const slice = await readFeedSlice(env, TEST_DID, FEED_A, 2); + expect(slice.map((i) => i.guid)).toEqual(['newest', 'newer']); + + await addSubscription(TEST_DID, FEED_A); + const page = await timeline(); + expect(page.items.map((i) => i.guid)).toEqual(['newest', 'newer', 'older', 'oldest']); + }); }); describe('crawl set', () => { diff --git a/docs/plans/D1_FEED_TIMELINE.md b/docs/plans/D1_FEED_TIMELINE.md index ade987ba..88fa2ed2 100644 --- a/docs/plans/D1_FEED_TIMELINE.md +++ b/docs/plans/D1_FEED_TIMELINE.md @@ -144,6 +144,11 @@ client commits a cursor. That signal is server-side on purpose — subscribe-tim pull-through both write to the archive, so "the archive is empty for this user" would stop being true long before the crawler existed. +The heartbeat means a crawler is attached; it does **not** mean the initial archive backfill is +complete. After enabling prod, watch `/stats` and wait for `ingest.pending` to trend to ~0 before +announcing the rollout. New or cleared clients remain correct while it drains, but their first cold +start can be sparse and will fill in over subsequent syncs. + ### Phase 5 — cleanup (a later release, once no legacy traffic remains) Remove `/api/v2/feeds/batch`'s proxy passthrough + its `getReadKeys` call (keep it for documents), @@ -162,9 +167,10 @@ step deleting `feeds`/`feed_items` rows whose feed has had **zero active subscri while the token would otherwise stay the same. The timeline also self-heals a cursor that sits above the head by cold-starting that client, so a forgotten bump degrades to one extra cold start rather than a silent, permanent stall. -- **Ingest order is oldest→newest.** A proxy feed is newest-first; seq is assigned in insert order, - so the pull-through walks it backwards (as the proxy's `writeFeedItems` does). Inverting it would - never heal — a re-push of an unchanged item is a no-op. +- **Per-feed recency comes from `published_at`, then `seq`.** A proxy feed is newest-first, so each + ingest call still writes oldest→newest. But subscribe-time pull-through and the crawler backlog + can interleave, making archive `seq` different from publication order; per-feed slices must remain + correct in that case. Incremental delivery continues to use `seq` as its cursor. - **Writes to the archive need a subscription.** Ordinary ingest deletes nothing, so every user-triggered write path (subscribe-time ingest, the pull-through) is gated on the caller's own subscription list. diff --git a/frontend/src/lib/components/ArticleCard.svelte b/frontend/src/lib/components/ArticleCard.svelte index 51d6d033..640563c8 100644 --- a/frontend/src/lib/components/ArticleCard.svelte +++ b/frontend/src/lib/components/ArticleCard.svelte @@ -1,4 +1,5 @@ -{labels[status]} +{label ?? labels[status]} diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 411ff1fa..4ba0eb56 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -7,10 +7,14 @@ Skyreader backend is a Cloudflare Workers API that serves as a gateway between the frontend and the AT Protocol ecosystem. It handles authentication, the feed timeline, social features, saved articles, labels, and background Jetstream polling. **Feed reads are served from D1, not the proxy.** The Fly.io proxy is the crawler: it pushes new -and edited items into `feed_items` (`POST /api/internal/ingest`) and pulls the set of feeds to -crawl (`GET /api/internal/crawl-set`), both authenticated with the shared `FEED_PROXY_SECRET` +and edited items into `feed_items` (`POST /api/internal/ingest`), pulls the set of feeds to +crawl (`GET /api/internal/crawl-set`), and reports which feeds are failing to crawl +(`POST /api/internal/feed-health`) — all authenticated with the shared `FEED_PROXY_SECRET` (fail-closed when unset). A client refresh is one `GET /api/v2/timeline` — a single query joining -subscriptions and read state. Both internal endpoints stamp `sync_state.crawler_heartbeat_at`, and +subscriptions and read state. Because reads never touch the crawler, a broken feed just goes quiet; +the health report is the only thing that tells a reader its feed is dead rather than idle, and its +payload is the COMPLETE unhealthy set (recovery = absence from the next report). All three internal +endpoints stamp `sync_state.crawler_heartbeat_at`, and the timeline reports `ingestActive` from it: an environment whose proxy has no `INGEST_URL` tells clients to stay on the legacy batch path instead of reading an archive nothing fills. See `docs/plans/D1_FEED_TIMELINE.md`. The proxy is still on the path for `/api/extract`, feed @@ -54,24 +58,24 @@ See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed documentation. ### Routes -| File | Purpose | -| ----------------------------- | ----------------------------------------------------- | -| `src/routes/auth.ts` | OAuth flow (login, callback, logout, client metadata) | -| `src/routes/timeline.ts` | `GET /api/v2/timeline` — the whole refresh, one query | -| `src/routes/ingest.ts` | Crawler endpoints: item ingest + crawl set | -| `src/routes/feeds-v2.ts` | Single-feed read (D1 + pull-through), discover, docs | -| `src/routes/social.ts` | Social feed, popular, grouped, detect-content | -| `src/routes/shares.ts` | User shares CRUD (with PDS sync) | -| `src/routes/subscriptions.ts` | Subscription CRUD (with PDS sync) | -| `src/routes/records.ts` | PDS record listing | -| `src/routes/reading.ts` | Article + document read positions (forward delta) | -| `src/routes/labels.ts` | Unified item labels (read/starred/archived/tags) | -| `src/routes/saved.ts` | Saved articles CRUD | -| `src/routes/settings.ts` | User settings | -| `src/routes/sync.ts` | PDS full sync, subscription sync, sync status | -| `src/routes/lexicons.ts` | Serve lexicon schemas at /.well-known/lexicons | -| `src/routes/health.ts` | `/api/health` (shallow) + `/api/health/deep` (gated) | -| `src/routes/telemetry.ts` | `/api/telemetry/error` — sampled client error reports | +| File | Purpose | +| ----------------------------- | ------------------------------------------------------ | +| `src/routes/auth.ts` | OAuth flow (login, callback, logout, client metadata) | +| `src/routes/timeline.ts` | `GET /api/v2/timeline` — the whole refresh, one query | +| `src/routes/ingest.ts` | Crawler endpoints: item ingest, crawl set, feed health | +| `src/routes/feeds-v2.ts` | Single-feed read (D1 + pull-through), discover, docs | +| `src/routes/social.ts` | Social feed, popular, grouped, detect-content | +| `src/routes/shares.ts` | User shares CRUD (with PDS sync) | +| `src/routes/subscriptions.ts` | Subscription CRUD (with PDS sync) | +| `src/routes/records.ts` | PDS record listing | +| `src/routes/reading.ts` | Article + document read positions (forward delta) | +| `src/routes/labels.ts` | Unified item labels (read/starred/archived/tags) | +| `src/routes/saved.ts` | Saved articles CRUD | +| `src/routes/settings.ts` | User settings | +| `src/routes/sync.ts` | PDS full sync, subscription sync, sync status | +| `src/routes/lexicons.ts` | Serve lexicon schemas at /.well-known/lexicons | +| `src/routes/health.ts` | `/api/health` (shallow) + `/api/health/deep` (gated) | +| `src/routes/telemetry.ts` | `/api/telemetry/error` — sampled client error reports | ### Services @@ -147,7 +151,11 @@ Key tables: - `sessions` - Server-side sessions (tokens, DPoP key, expiry) - `subscriptions_cache` - Cached feed subscriptions from PDS - `shares` - Aggregated share data from Jetstream -- `feeds` - One row per crawled feed (title/site/image + `last_ingest_at`) +- `feeds` - One row per crawled feed (title/site/image + `last_ingest_at`), plus the crawler's + health verdict: `error_count`/`last_error`/`next_retry_at`/`last_fetch_at` (unix seconds), which + the timeline serves to readers as `feedHealth`, and `crawl_stale` for a feed the crawler isn't + reaching at all (operator-only; the admin alarms on it). `last_ingest_at` is publishing cadence, + NOT health — it only moves when a fetch yields a new item - `feed_items` - The feed archive the timeline serves: every item the crawler has ever pushed, keyed `(feed_url, guid)` with a monotonic `seq`. Never pruned in ordinary operation — see `docs/plans/D1_FEED_TIMELINE.md` diff --git a/backend/migrations/0070_feed_health.sql b/backend/migrations/0070_feed_health.sql new file mode 100644 index 00000000..f108cb9b --- /dev/null +++ b/backend/migrations/0070_feed_health.sql @@ -0,0 +1,46 @@ +-- Per-feed crawl health in the archive. +-- +-- On the legacy batch path the client learned a feed was broken from the proxy's +-- own response (`status: 'error'`, errorCount, nextRetryAt). The timeline path +-- never touches the proxy, so that signal disappeared: a feed failing to crawl +-- simply delivered no items, which is indistinguishable from a quiet feed. The +-- crawler now reports health here (POST /api/internal/feed-health) and the +-- timeline hands it to the client alongside its items. +-- +-- Units: unix SECONDS, like `last_ingest_at` and the rest of the backend. The +-- proxy stores these as milliseconds and converts on send; the timeline converts +-- back to milliseconds on the way out, because that is what the client's +-- FeedStatus contract has always used. + +ALTER TABLE feeds ADD COLUMN error_count INTEGER NOT NULL DEFAULT 0; +ALTER TABLE feeds ADD COLUMN last_error TEXT; +ALTER TABLE feeds ADD COLUMN last_error_at INTEGER; +ALTER TABLE feeds ADD COLUMN next_retry_at INTEGER; +-- Last time the crawler successfully FETCHED the feed. Distinct from +-- `last_ingest_at`, which only moves when a fetch produced a new or edited item: +-- a healthy feed that hasn't published in a month still gets fetched hourly. +ALTER TABLE feeds ADD COLUMN last_fetch_at INTEGER; +-- The crawler has this feed in its crawl set but hasn't managed to fetch it in +-- hours — starved by a saturated warm loop rather than failing outright. Distinct +-- from `error_count`: nothing is erroring, the crawler simply never gets to it. +-- +-- This is the difference between an alarm and noise. `last_ingest_at` only moves +-- when a fetch yields a NEW item, so "no ingest in an hour" describes every +-- weekly blog in the archive; the admin's old stale-feed metric was therefore +-- warning permanently and meant nothing. Being in the crawl set and un-fetched is +-- a genuine fault, and it is the one the current warm-loop capacity work needs. +ALTER TABLE feeds ADD COLUMN crawl_stale INTEGER NOT NULL DEFAULT 0; + +-- The trouble set is a handful of rows out of the whole archive, so let the +-- per-user health lookup, the recovery sweep and the admin's counts start from it +-- rather than scanning `feeds`. Recovery is "currently flagged, and absent from +-- the latest report", which this index makes cheap to enumerate — the +-- alternative, a NOT IN list of every healthy feed, would blow the +-- bound-parameter limit at ~1,300. +CREATE INDEX idx_feeds_unhealthy ON feeds(feed_url) WHERE error_count > 0 OR crawl_stale = 1; + +-- Bumped whenever a health report actually changes something. Clients echo the +-- revision they last saw and the timeline only re-sends the health payload when +-- it differs, so the steady-state poll stays at one query. Only `error_count` +-- feeds into it: `crawl_stale` is an operator signal, invisible to readers. +INSERT OR IGNORE INTO sync_state (key, value) VALUES ('feed_health_rev', '0'); diff --git a/backend/src/index.ts b/backend/src/index.ts index b67ace5f..63601262 100644 --- a/backend/src/index.ts +++ b/backend/src/index.ts @@ -16,7 +16,7 @@ import { handleV2Mentions, handleV2MentionLane, } from './routes/feeds-v2'; -import { handleIngest, handleCrawlSet } from './routes/ingest'; +import { handleIngest, handleCrawlSet, handleFeedHealth } from './routes/ingest'; import { handleTimeline } from './routes/timeline'; import { handleDetectContent } from './routes/social'; import { @@ -325,6 +325,9 @@ async function route( case url.pathname === '/api/internal/crawl-set': response = await handleCrawlSet(request, env); break; + case url.pathname === '/api/internal/feed-health': + response = await handleFeedHealth(request, env); + break; // The reader's whole refresh, served from the D1 archive in one query. case url.pathname === '/api/v2/timeline': diff --git a/backend/src/routes/feeds-v2.ts b/backend/src/routes/feeds-v2.ts index 39ec8999..6dfbbf13 100644 --- a/backend/src/routes/feeds-v2.ts +++ b/backend/src/routes/feeds-v2.ts @@ -8,8 +8,8 @@ import type { } from '../services/feed-proxy-client'; import { resolveStandardSite } from '../utils/canonical-url'; import { chunkArray, getReadKeys } from './reading'; -import { ingestProxyFeed } from './ingest'; -import { readFeedMetadata, readFeedSlice } from './timeline'; +import { clearFeedHealth, ingestProxyFeed } from './ingest'; +import { readFeedMetadata, readFeedSlice, type FeedHealth } from './timeline'; import { getLinkblogTargets, publicationUri as linkblogPublicationUri, @@ -24,6 +24,8 @@ interface V2FeedResponse { // Unix ms of the last ingest for this feed (freshness, for the client's UI) — // no longer a live upstream fetch time, since reads never touch the proxy. fetchedAt: number; + // Present only when the crawler currently considers this feed broken. + health?: FeedHealth; } interface V2BatchFeedResult { @@ -148,6 +150,10 @@ export async function handleV2FeedFetch( const client = new FeedProxyClient(env); const feed = await client.fetchFeed(feedUrl); await ingestProxyFeed(env, feedUrl, feed); + // We just fetched it, so whatever the crawler last recorded is stale. + // Clearing here is what makes the user's "retry this feed" action show a + // result now rather than after the next health report. + await clearFeedHealth(env, feedUrl); items = await readFeedSlice(env, session.did, feedUrl, limit); metadata = await readFeedMetadata(env, feedUrl); } catch (error) { @@ -164,6 +170,19 @@ export async function handleV2FeedFetch( imageUrl: metadata?.image_url ?? undefined, items, fetchedAt: (metadata?.last_ingest_at ?? Math.floor(Date.now() / 1000)) * 1000, + // The crawler's verdict on this feed, so a per-feed read reports a broken + // feed even when the archive still has old items to serve. Timestamps in + // ms, matching the timeline's health payload. + health: + metadata && metadata.error_count > 0 + ? { + errorCount: metadata.error_count, + error: metadata.last_error ?? undefined, + lastErrorAt: metadata.last_error_at ? metadata.last_error_at * 1000 : undefined, + nextRetryAt: metadata.next_retry_at ? metadata.next_retry_at * 1000 : undefined, + lastFetchedAt: metadata.last_fetch_at ? metadata.last_fetch_at * 1000 : undefined, + } + : undefined, }; return new Response(JSON.stringify(response), { diff --git a/backend/src/routes/ingest.ts b/backend/src/routes/ingest.ts index d29d1d90..20644d1e 100644 --- a/backend/src/routes/ingest.ts +++ b/backend/src/routes/ingest.ts @@ -31,6 +31,11 @@ export const MAX_ITEM_CONTENT_BYTES = 8 * 1024; export const CRAWLER_HEARTBEAT_KEY = 'crawler_heartbeat_at'; export const CRAWLER_HEARTBEAT_FRESH_SECONDS = 30 * 60; +// Revision token for the set of feeds the crawler currently considers broken. +// The timeline sends the per-feed health payload only when the client's echoed +// revision differs from this, so a steady-state poll costs no extra query. +export const FEED_HEALTH_REV_KEY = 'feed_health_rev'; + // Bounds on one ingest call. The pusher chunks to ~100 items; these are abuse // guards, not tuning knobs. const MAX_INGEST_ITEMS = 1000; @@ -325,6 +330,210 @@ export async function stampCrawlerHeartbeat(env: Env): Promise { } } +/** + * Per-feed crawl health, as the crawler reports it. Unix SECONDS on this wire + * (the proxy keeps milliseconds internally and converts on send); the timeline + * converts back to milliseconds for the client. + */ +export interface FeedHealthReport { + feedUrl: string; + errorCount: number; + lastError?: string | null; + lastErrorAt?: number | null; + nextRetryAt?: number | null; + lastFetchAt?: number | null; + // In the crawl set but going unfetched — starved, not failing. Independent of + // `errorCount`, which can be 0 while this is true. + crawlStale?: boolean; +} + +// A report carries only the unhealthy feeds, so this is far above any plausible +// real number — an abuse guard, not a tuning knob. +const MAX_HEALTH_FEEDS = 2000; +// Error strings are rendered in a popover and matched against substrings; the +// crawler's messages are short, so anything longer is a runaway. +const MAX_HEALTH_ERROR_CHARS = 500; +// Feed URLs per recovery statement. Well under D1's per-statement bind limit, +// and the recovered set is normally a handful anyway. +const HEALTH_CLEAR_CHUNK = 50; + +/** + * A cheap fingerprint of the whole unhealthy set, used as the `feed_health_rev` + * token. Clients echo the revision they last saw and the timeline re-sends the + * health payload only when it differs, which keeps a steady-state poll at the + * one query the architecture promises. + * + * Aggregates rather than a real hash: the partial index makes this a scan of the + * handful of broken feeds, not of the archive. Two genuinely different sets can + * alias only if they share a count, an error-count sum, and both timestamps — and + * the cost of that is one poll showing yesterday's error, self-healing on the + * next change. + */ +async function readFeedHealthRev(env: Env): Promise { + const row = await env.DB.prepare( + `SELECT COUNT(*) AS n, COALESCE(SUM(error_count), 0) AS errors, + COALESCE(MAX(last_error_at), 0) AS newest, COALESCE(MAX(next_retry_at), 0) AS retry + FROM feeds WHERE error_count > 0` + ).first<{ n: number; errors: number; newest: number; retry: number }>(); + return `${row?.n ?? 0}:${row?.errors ?? 0}:${row?.newest ?? 0}:${row?.retry ?? 0}`; +} + +/** + * Clear one feed's error state because we just fetched it successfully. + * + * The pull-through in `/api/v2/feeds/fetch` is the user's explicit "retry this + * feed" action, and it is proof the feed works — but the crawler's next health + * report is up to five minutes away, so without this the reader would keep + * showing the error the user just cleared. Refreshes the revision so every other + * client picks the recovery up too. + */ +export async function clearFeedHealth(env: Env, feedUrl: string): Promise { + try { + const result = await env.DB.prepare( + `UPDATE feeds + SET error_count = 0, last_error = NULL, last_error_at = NULL, next_retry_at = NULL, + crawl_stale = 0, last_fetch_at = ? + WHERE feed_url = ? AND (error_count > 0 OR crawl_stale = 1)` + ) + .bind(Math.floor(Date.now() / 1000), feedUrl) + .run(); + if (!result.meta?.changes) return; + + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at` + ) + .bind(FEED_HEALTH_REV_KEY, await readFeedHealthRev(env)) + .run(); + } catch (error) { + // Health is observability, not the read itself — never fail a fetch over it. + console.error('[ingest] Failed to clear feed health:', error); + } +} + +/** + * POST /api/internal/feed-health + * + * The crawler's periodic report of every feed it currently considers broken. + * + * This is the timeline path's replacement for what the legacy batch response + * carried inline: on `/api/v2/feeds/batch` a failing feed came back with + * `status: 'error'` plus its error, count and retry time, which is what fed the + * reader's per-feed error badge and popover. Reads no longer touch the proxy, and + * a feed that fails to crawl pushes no items, so without this a broken feed is + * indistinguishable from a quiet one and the user gets no signal at all. + * + * The payload is the COMPLETE unhealthy set, not a delta: recovery is inferred + * from absence (see the sweep below), so a feed that starts working again clears + * without the crawler having to say anything about it. + */ +export async function handleFeedHealth(request: Request, env: Env): Promise { + if (request.method !== 'POST') return badRequest('Method not allowed', 405); + if (!isAuthorizedProxyRequest(request, env)) return unauthorized(); + + const declaredLength = Number(request.headers.get('Content-Length') ?? '0'); + if (declaredLength > MAX_INGEST_BODY_BYTES) return badRequest('Payload too large', 413); + + let body: { feeds?: FeedHealthReport[] }; + try { + body = await request.json(); + } catch { + return badRequest('Invalid JSON body'); + } + + // A report entry means "something is wrong with this feed" — it is erroring, + // or the crawler isn't reaching it, or both. An entry claiming neither is + // noise and would keep the feed flagged forever. + const reports = (Array.isArray(body.feeds) ? body.feeds : []).filter( + (f) => f?.feedUrl && ((Number.isFinite(f.errorCount) && f.errorCount > 0) || f.crawlStale) + ); + if (reports.length > MAX_HEALTH_FEEDS) return badRequest('Too many feeds'); + + try { + const before = await readFeedHealthRev(env); + + // Which feeds are flagged right now. Read BEFORE the upserts so the + // difference against this report is exactly "was in trouble, isn't any more". + // Small by construction (the partial index covers only flagged feeds), so the + // chunked IN list below can never approach the bound-parameter limit. + const flagged = await env.DB.prepare( + `SELECT feed_url FROM feeds WHERE error_count > 0 OR crawl_stale = 1` + ).all<{ feed_url: string }>(); + + const stillTroubled = new Set(reports.map((r) => r.feedUrl)); + const recovered = flagged.results + .map((r) => r.feed_url) + .filter((url) => !stillTroubled.has(url)); + + // A feed can be broken from its very first crawl, in which case it has never + // been ingested and has no `feeds` row at all — so this inserts rather than + // assuming one exists. Such a row carries health only (NULL title/site_url), + // which the metadata backfill already skips. + const upserts = reports.map((report) => + env.DB.prepare( + `INSERT INTO feeds (feed_url, error_count, last_error, last_error_at, next_retry_at, + last_fetch_at, crawl_stale) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7) + ON CONFLICT(feed_url) DO UPDATE SET + error_count = excluded.error_count, + last_error = excluded.last_error, + last_error_at = excluded.last_error_at, + next_retry_at = excluded.next_retry_at, + last_fetch_at = COALESCE(excluded.last_fetch_at, feeds.last_fetch_at), + crawl_stale = excluded.crawl_stale` + ).bind( + report.feedUrl, + Number.isFinite(report.errorCount) ? Math.max(report.errorCount, 0) : 0, + report.lastError ? report.lastError.slice(0, MAX_HEALTH_ERROR_CHARS) : null, + report.lastErrorAt ?? null, + report.nextRetryAt ?? null, + report.lastFetchAt ?? null, + report.crawlStale ? 1 : 0 + ) + ); + for (let i = 0; i < upserts.length; i += INGEST_BATCH_SIZE) { + await env.DB.batch(upserts.slice(i, i + INGEST_BATCH_SIZE)); + } + + // Recovery: a feed the crawler stopped listing is fine again. Absence is the + // entire signal, which is why the report has to be the complete set. + // Chunked well under D1's per-statement bind limit — the same limit a + // subscription-sized IN list walked into once already. + for (let i = 0; i < recovered.length; i += HEALTH_CLEAR_CHUNK) { + const chunk = recovered.slice(i, i + HEALTH_CLEAR_CHUNK); + await env.DB.prepare( + `UPDATE feeds + SET error_count = 0, last_error = NULL, last_error_at = NULL, next_retry_at = NULL, + crawl_stale = 0 + WHERE feed_url IN (${chunk.map(() => '?').join(',')})` + ) + .bind(...chunk) + .run(); + } + + const after = await readFeedHealthRev(env); + if (after !== before) { + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at` + ) + .bind(FEED_HEALTH_REV_KEY, after) + .run(); + } + + await stampCrawlerHeartbeat(env); + return new Response(JSON.stringify({ ok: true, unhealthy: reports.length, rev: after }), { + headers: { 'Content-Type': 'application/json' }, + }); + } catch (error) { + console.error('[ingest] Feed-health write error:', error); + return new Response(JSON.stringify({ error: 'Feed health update failed' }), { + status: 500, + headers: { 'Content-Type': 'application/json' }, + }); + } +} + /** * POST /api/internal/ingest * diff --git a/backend/src/routes/timeline.ts b/backend/src/routes/timeline.ts index 9b644221..63cd2a05 100644 --- a/backend/src/routes/timeline.ts +++ b/backend/src/routes/timeline.ts @@ -2,6 +2,7 @@ import type { Env, FeedItem, Session } from '../types'; import { CRAWLER_HEARTBEAT_KEY, CRAWLER_HEARTBEAT_FRESH_SECONDS, + FEED_HEALTH_REV_KEY, rssSubscriptionPredicate, } from './ingest'; @@ -103,11 +104,20 @@ export interface FeedMetadataRow { description: string | null; image_url: string | null; last_ingest_at: number | null; + // Crawl health (unix seconds), so a single-feed read can report a broken feed + // even while it serves the archived items the feed still has. + error_count: number; + last_error: string | null; + last_error_at: number | null; + next_retry_at: number | null; + last_fetch_at: number | null; } export async function readFeedMetadata(env: Env, feedUrl: string): Promise { return env.DB.prepare( - `SELECT title, site_url, description, image_url, last_ingest_at FROM feeds WHERE feed_url = ?` + `SELECT title, site_url, description, image_url, last_ingest_at, + error_count, last_error, last_error_at, next_retry_at, last_fetch_at + FROM feeds WHERE feed_url = ?` ) .bind(feedUrl) .first(); @@ -119,28 +129,98 @@ export interface ArchiveState { // nothing is filling this D1 (no INGEST_URL on the paired proxy, or the proxy // is down), so the client must not treat an empty/partial archive as the truth. ingestActive: boolean; + // Revision of the unhealthy-feed set. A client that echoes this back unchanged + // already holds current health and is sent no health payload. + healthRev: string; } /** - * Generation token + crawler liveness in one `sync_state` read (the timeline - * needs both on every request, and they live one row apart). + * Generation token, crawler liveness and the feed-health revision in one + * `sync_state` read (the timeline needs all three on every request, and they are + * three rows of the same small table). */ export async function readArchiveState(env: Env): Promise { const rows = await env.DB.prepare( - `SELECT key, value FROM sync_state WHERE key IN ('items_generation', ?)` + `SELECT key, value FROM sync_state WHERE key IN ('items_generation', ?, ?)` ) - .bind(CRAWLER_HEARTBEAT_KEY) + .bind(CRAWLER_HEARTBEAT_KEY, FEED_HEALTH_REV_KEY) .all<{ key: string; value: string }>(); let generation = ''; let heartbeat = 0; + let healthRev = ''; for (const row of rows.results) { if (row.key === 'items_generation') generation = row.value; else if (row.key === CRAWLER_HEARTBEAT_KEY) heartbeat = parseInt(row.value, 10) || 0; + else if (row.key === FEED_HEALTH_REV_KEY) healthRev = row.value; } const age = Math.floor(Date.now() / 1000) - heartbeat; - return { generation, ingestActive: heartbeat > 0 && age <= CRAWLER_HEARTBEAT_FRESH_SECONDS }; + return { + generation, + ingestActive: heartbeat > 0 && age <= CRAWLER_HEARTBEAT_FRESH_SECONDS, + healthRev, + }; +} + +/** + * Per-feed crawl health for the caller's subscriptions — the timeline path's + * replacement for the per-feed `status: 'error'` the legacy batch response + * carried inline. + * + * Only broken feeds are returned: the client treats absence as healthy, which is + * what makes recovery work without sending a row per subscription. Driven from + * the (small, partially indexed) unhealthy set rather than from the caller's + * subscriptions, because on a healthy system that set is empty and this costs + * nothing. + * + * Timestamps go out in MILLISECONDS — D1 stores seconds like the rest of the + * backend, but the client's FeedStatus contract has always been ms. + */ +export async function subscribedFeedHealth( + env: Env, + userDid: string +): Promise> { + const rows = await env.DB.prepare( + `SELECT f.feed_url, f.error_count, f.last_error, f.last_error_at, f.next_retry_at, f.last_fetch_at + FROM feeds f + WHERE f.error_count > 0 + AND EXISTS ( + SELECT 1 FROM subscriptions_cache sc + WHERE sc.user_did = ? AND sc.feed_url = f.feed_url AND sc.active = 1 + AND ${rssSubscriptionPredicate('sc')} + )` + ) + .bind(userDid) + .all<{ + feed_url: string; + error_count: number; + last_error: string | null; + last_error_at: number | null; + next_retry_at: number | null; + last_fetch_at: number | null; + }>(); + + const health: Record = {}; + for (const row of rows.results) { + health[row.feed_url] = { + errorCount: row.error_count, + error: row.last_error ?? undefined, + lastErrorAt: row.last_error_at ? row.last_error_at * 1000 : undefined, + nextRetryAt: row.next_retry_at ? row.next_retry_at * 1000 : undefined, + lastFetchedAt: row.last_fetch_at ? row.last_fetch_at * 1000 : undefined, + }; + } + return health; +} + +/** One broken feed as the timeline reports it. Timestamps are unix ms. */ +export interface FeedHealth { + errorCount: number; + error?: string; + lastErrorAt?: number; + nextRetryAt?: number; + lastFetchedAt?: number; } /** The archive's current head; 0 when nothing has ever been ingested. */ @@ -221,6 +301,9 @@ export async function handleTimeline( const generationParam = url.searchParams.get('generation'); const limitParam = url.searchParams.get('limit'); const coldOffsetParam = url.searchParams.get('cold_offset'); + // The feed-health revision this client already holds. Absent (a fresh page + // load, whose status store is empty) means "send it". + const healthRevParam = url.searchParams.get('health_rev'); const parsedLimit = limitParam ? parseInt(limitParam, 10) : DEFAULT_LIMIT; const limit = Number.isInteger(parsedLimit) @@ -235,7 +318,12 @@ export async function handleTimeline( const coldOffset = Number.isInteger(parsedColdOffset) && parsedColdOffset > 0 ? parsedColdOffset : 0; - const { generation, ingestActive } = await readArchiveState(env); + const { generation, ingestActive, healthRev } = await readArchiveState(env); + // Send health when the client's copy is stale. A cold start always gets it: + // it is the one page that delivers already-archived items for a feed that may + // have broken since, and its blanket "these feeds delivered, so they're fine" + // pass would otherwise clear a live error. + const healthStale = healthRevParam !== healthRev; // Server time (unix seconds) at annotation. The client seeds its forward // read-delta cursor from this, exactly as /batch does today, so the delta // starts from bootstrap with no client/server clock skew. @@ -303,6 +391,8 @@ export async function handleTimeline( readCursor, coldStart: false, feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined, + healthRev, + feedHealth: healthStale ? await subscribedFeedHealth(env, session.did) : undefined, }); } } @@ -361,6 +451,8 @@ export async function handleTimeline( readCursor, coldStart: true, feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined, + healthRev, + feedHealth: await subscribedFeedHealth(env, session.did), }); } catch (error) { console.error('[timeline] Query error:', error); diff --git a/backend/test/feed-timeline.spec.ts b/backend/test/feed-timeline.spec.ts index 2540c8d3..211e346d 100644 --- a/backend/test/feed-timeline.spec.ts +++ b/backend/test/feed-timeline.spec.ts @@ -3,9 +3,12 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import { handleIngest, handleCrawlSet, + handleFeedHealth, + clearFeedHealth, trimFeedsToSanityCap, ingestProxyFeed, CRAWLER_HEARTBEAT_KEY, + FEED_HEALTH_REV_KEY, } from '../src/routes/ingest'; import { handleTimeline, readFeedSlice } from '../src/routes/timeline'; import type { Env, FeedItem, Session } from '../src/types'; @@ -114,6 +117,17 @@ async function timeline(params: Record = {}) { readCursor: number; coldStart: boolean; feeds?: Record; + healthRev?: string; + feedHealth?: Record< + string, + { + errorCount: number; + error?: string; + lastErrorAt?: number; + nextRetryAt?: number; + lastFetchedAt?: number; + } + >; }; } @@ -121,6 +135,32 @@ async function clearCrawlerHeartbeat() { await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(CRAWLER_HEARTBEAT_KEY).run(); } +async function reportHealth(feeds: unknown[], secret: string | null = SECRET) { + const headers: Record = { 'Content-Type': 'application/json' }; + if (secret !== null) headers['X-Proxy-Secret'] = secret; + return handleFeedHealth( + new Request('https://api.example/api/internal/feed-health', { + method: 'POST', + headers, + body: JSON.stringify({ feeds }), + }), + env + ); +} + +function brokenFeed(feedUrl: string, overrides: Record = {}) { + const nowSeconds = Math.floor(Date.now() / 1000); + return { + feedUrl, + errorCount: 3, + lastError: 'Failed to fetch (HTTP 404)', + lastErrorAt: nowSeconds, + nextRetryAt: nowSeconds + 600, + lastFetchAt: nowSeconds - 7200, + ...overrides, + }; +} + describe('feed timeline (D1 ingest + serve)', () => { let savedSecret: string | undefined; @@ -144,6 +184,7 @@ describe('feed timeline (D1 ingest + serve)', () => { await env.DB.prepare('DELETE FROM subscriptions_cache').run(); await env.DB.prepare('DELETE FROM item_labels_cache').run(); await clearCrawlerHeartbeat(); + await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(FEED_HEALTH_REV_KEY).run(); }); describe('ingest auth', () => { @@ -579,4 +620,211 @@ describe('feed timeline (D1 ingest + serve)', () => { expect((await timeline()).ingestActive).toBe(false); }); }); + + describe('feed health', () => { + it('requires the shared secret', async () => { + expect((await reportHealth([brokenFeed(FEED_A)], null)).status).toBe(401); + expect((await reportHealth([brokenFeed(FEED_A)], 'nope')).status).toBe(401); + }); + + it('stamps the crawler heartbeat like the other internal endpoints', async () => { + await addSubscription(TEST_DID, FEED_A); + await clearCrawlerHeartbeat(); + expect((await reportHealth([])).status).toBe(200); + expect((await timeline()).ingestActive).toBe(true); + }); + + it('serves a broken feed to its subscribers, in milliseconds', async () => { + await addSubscription(TEST_DID, FEED_A); + const report = brokenFeed(FEED_A); + await reportHealth([report]); + + const body = await timeline(); + expect(body.feedHealth?.[FEED_A]).toEqual({ + errorCount: 3, + error: 'Failed to fetch (HTTP 404)', + lastErrorAt: report.lastErrorAt * 1000, + nextRetryAt: report.nextRetryAt * 1000, + lastFetchedAt: report.lastFetchAt * 1000, + }); + }); + + it('records a feed that has never ingested a single item', async () => { + // Broken from its first crawl: no items were ever pushed, so there is no + // `feeds` row for the health report to update. + await addSubscription(TEST_DID, FEED_B); + await reportHealth([brokenFeed(FEED_B)]); + expect((await timeline()).feedHealth?.[FEED_B]?.errorCount).toBe(3); + }); + + it('does not leak another user’s broken feeds', async () => { + await addSubscription(OTHER_DID, FEED_B); + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B)]); + + const body = await timeline(); + expect(Object.keys(body.feedHealth ?? {})).toEqual([FEED_A]); + }); + + it('clears a feed that recovers, by absence from the next report', async () => { + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A)]); + expect((await timeline()).feedHealth?.[FEED_A]).toBeDefined(); + + // The crawler no longer lists it — that is the whole recovery signal. + await reportHealth([]); + expect((await timeline()).feedHealth?.[FEED_A]).toBeUndefined(); + }); + + it('recovers one feed without disturbing another that is still broken', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B); + await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B)]); + await reportHealth([brokenFeed(FEED_B)]); + + const body = await timeline(); + expect(Object.keys(body.feedHealth ?? {})).toEqual([FEED_B]); + }); + + it('omits the payload when the client already holds the current revision', async () => { + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A)]); + + const first = await timeline(); + expect(first.feedHealth).toBeDefined(); + expect(first.healthRev).toBeTruthy(); + + // Steady state: the client echoes the revision back and pays nothing. + const second = await timeline({ + since_seq: String(first.cursor), + generation: first.generation, + health_rev: first.healthRev!, + }); + expect(second.feedHealth).toBeUndefined(); + expect(second.healthRev).toBe(first.healthRev); + }); + + it('re-sends the payload once the unhealthy set changes', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B); + await reportHealth([brokenFeed(FEED_A)]); + const first = await timeline(); + + await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B, { errorCount: 1 })]); + const second = await timeline({ + since_seq: String(first.cursor), + generation: first.generation, + health_rev: first.healthRev!, + }); + + expect(second.healthRev).not.toBe(first.healthRev); + expect(Object.keys(second.feedHealth ?? {}).sort()).toEqual([FEED_A, FEED_B].sort()); + }); + + it('always sends health on a cold start, whatever revision the client claims', async () => { + // A cold start replays already-archived items, including from a feed that + // has broken since — so its blanket "these delivered, they're fine" pass + // must be corrected in the same response. + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('c1'), contentHash: 'c1' }]); + await reportHealth([brokenFeed(FEED_A)]); + + const rev = (await timeline()).healthRev!; + const cold = await timeline({ health_rev: rev }); + expect(cold.coldStart).toBe(true); + expect(cold.items).toHaveLength(1); + expect(cold.feedHealth?.[FEED_A]).toBeDefined(); + }); + + it('clearFeedHealth clears one feed and moves the revision', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B); + await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B)]); + const before = (await timeline()).healthRev; + + await clearFeedHealth(env as Env, FEED_A); + + const body = await timeline(); + expect(Object.keys(body.feedHealth ?? {})).toEqual([FEED_B]); + expect(body.healthRev).not.toBe(before); + }); + + it('flags a starved feed without telling readers anything', async () => { + // `crawl_stale` is an operator signal: the crawler isn't reaching the feed, + // but nothing is erroring, so the reader has no error to show. + await addSubscription(TEST_DID, FEED_A); + await reportHealth([{ feedUrl: FEED_A, errorCount: 0, crawlStale: true }]); + + const body = await timeline(); + expect(body.feedHealth?.[FEED_A]).toBeUndefined(); + + const row = await env.DB.prepare( + 'SELECT error_count, crawl_stale FROM feeds WHERE feed_url = ?' + ) + .bind(FEED_A) + .first<{ error_count: number; crawl_stale: number }>(); + expect(row).toMatchObject({ error_count: 0, crawl_stale: 1 }); + }); + + it('clears a starved flag once the crawler catches up', async () => { + await addSubscription(TEST_DID, FEED_A); + await reportHealth([{ feedUrl: FEED_A, errorCount: 0, crawlStale: true }]); + await reportHealth([]); + + const row = await env.DB.prepare('SELECT crawl_stale FROM feeds WHERE feed_url = ?') + .bind(FEED_A) + .first<{ crawl_stale: number }>(); + expect(row?.crawl_stale).toBe(0); + }); + + it('carries both flags for a feed that is erroring and starved', async () => { + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A, { crawlStale: true })]); + + const row = await env.DB.prepare( + 'SELECT error_count, crawl_stale FROM feeds WHERE feed_url = ?' + ) + .bind(FEED_A) + .first<{ error_count: number; crawl_stale: number }>(); + expect(row).toMatchObject({ error_count: 3, crawl_stale: 1 }); + // Readers still see the error, which is the part they can act on. + expect((await timeline()).feedHealth?.[FEED_A]?.errorCount).toBe(3); + }); + + it('ignores an entry that claims no fault at all', async () => { + // Otherwise a healthy feed listed by mistake would stay flagged forever. + await addSubscription(TEST_DID, FEED_A); + await reportHealth([{ feedUrl: FEED_A, errorCount: 0, crawlStale: false }]); + + const row = await env.DB.prepare('SELECT feed_url FROM feeds WHERE feed_url = ?') + .bind(FEED_A) + .first(); + expect(row).toBeNull(); + }); + + it('leaves the revision alone when only the starved flag moves', async () => { + // The reader payload holds erroring feeds only, so a crawl-capacity change + // must not make every client re-download it. + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A)]); + const before = (await timeline()).healthRev; + + await addSubscription(TEST_DID, FEED_B); + await reportHealth([ + brokenFeed(FEED_A), + { feedUrl: FEED_B, errorCount: 0, crawlStale: true }, + ]); + expect((await timeline()).healthRev).toBe(before); + }); + + it('leaves the revision alone when a report changes nothing', async () => { + await addSubscription(TEST_DID, FEED_A); + const report = brokenFeed(FEED_A); + await reportHealth([report]); + const first = (await timeline()).healthRev; + + await reportHealth([report]); + expect((await timeline()).healthRev).toBe(first); + }); + }); }); diff --git a/docs/plans/D1_FEED_TIMELINE.md b/docs/plans/D1_FEED_TIMELINE.md index d5e60334..87a4dd7a 100644 --- a/docs/plans/D1_FEED_TIMELINE.md +++ b/docs/plans/D1_FEED_TIMELINE.md @@ -56,11 +56,26 @@ read state (`getReadKeys`). Now: `getReadKeys` is never called on the feed path. Every response carries `ingestActive`, derived from the heartbeat: false means this deployment has no crawler filling D1, and clients stay on the legacy batch path. A cursor above the archive head cold-starts (rewound-archive guard). +- `routes/ingest.ts` also serves `POST /api/internal/feed-health`: the crawler's periodic report of + every feed it currently considers broken, which the timeline hands to readers. On the batch path + a failing feed came back with `status: 'error'` inline; reads no longer touch the proxy, and a + broken feed pushes no items, so without this a dead feed and a quiet feed look identical to the + client. The payload is the **complete trouble set, not a delta** — recovery is inferred from a + feed's absence, so nothing has to announce that it started working. `feeds` carries the state + (`error_count`, `last_error`, `last_error_at`, `next_retry_at`, `last_fetch_at`, `crawl_stale`, + migration `0070`), and a report inserts rather than assumes a row, because a feed broken from its + first crawl has never ingested an item. Two distinct faults ride the same report: `error_count` + (the fetch fails — readers see this) and `crawl_stale` (the feed is in the crawl set but going + unfetched for hours — an operator signal only, invisible to readers and deliberately excluded from + `feed_health_rev` so crawl-capacity churn can't make every client re-download the payload). - `GET /api/v2/feeds/fetch` re-backed with D1 + **pull-through**: if a feed isn't in the archive yet (first subscriber), fetch it from the proxy once, ingest it, then serve. The pull-through is gated on the caller's own subscription, so the shared never-pruned archive can't be written with arbitrary feeds. Subscribe time already crawls + ingests the feed (`warmFeedIntoArchive`, which - replaced the old warm-and-discard), so the pull-through is normally not needed at all. + replaced the old warm-and-discard), so the pull-through is normally not needed at all. The + response carries the feed's `health` when it has any, and a successful pull-through clears it + (`clearFeedHealth`) — that path *is* the user's "retry this feed" action, so it must show a result + now rather than after the crawler's next report. **Proxy** (`feed-proxy/`) @@ -70,6 +85,11 @@ read state (`getReadKeys`). Now: - Crawl-set pull every 5 min: registers each feed's `cache` row and stamps `last_requested_at`, so the existing warm loop / active window / eviction machinery keeps working now that read traffic no longer stamps anything. +- A feed-health report rides the same 5-minute timer, sent right **after** the crawl-set pull so a + feed registered this cycle is already in the set. `selectFeedHealth` reads the crawl-set `cache` + rows that are either erroring (`error_count > 0`) or unfetched for `CRAWL_STALE_MS` (2 h — the + warm loop works on a minutes-long cadence, so hours means the feed is losing its turn every tick), + converting its millisecond timestamps to seconds once, at this boundary. - `INGEST_URL` unset ⇒ both loops disabled. `push_state` cascades on the K = 200 cap trim, the redelivery delete, and `cleanupCache` eviction. `/stats` reports `ingest.{items,pushed,pending}`. @@ -90,10 +110,27 @@ read state (`getReadKeys`). Now: time, 1 s apart) and no longer force a crawl, so a 250-feed import stays inside the rate limit. - An article whose body was dropped at ingest (`contentTruncated`) is extracted automatically when its card opens, so the reader shows the whole article rather than an RSS summary. - -**Admin** (`admin/`) — feed health is re-pointed at `feeds`/`feed_items`: crawled feeds, subscribed -feeds not ingesting (the R1 alarm), archived item count, estimated archive size with a 6 GB alert, -and a churn detector for feeds nearing the sanity cap. +- Per-feed error badges come from the response's `feedHealth` (`reconcileFeedHealth` → + `feedStatusStore.applyHealthSnapshot`). Absence from that map is what CLEARS an error, so the + reconcile runs after the "these feeds delivered items, so they're fine" pass and overrides it — a + cold start replays archived items from feeds that may have broken since. To keep the steady-state + poll at one query, the payload is only sent when the client's echoed `health_rev` is stale, plus + unconditionally on a cold start (whose status store may be empty). + +**Admin** (`admin/`) — feed health is re-pointed at `feeds`/`feed_items`: crawled feeds, archived +item count, estimated archive size with a 6 GB alert, and a churn detector for feeds nearing the +sanity cap. Health itself is the **crawler's verdict**, not an inference: + +- **Subscribed Feeds Erroring** (`error_count > 0`) and **Subscribed Feeds Not Being Crawled** + (`crawl_stale = 1`) replace the single "Subscribed Feeds Not Ingesting" tile. That tile keyed off + `last_ingest_at`, which only moves when a fetch yields a NEW item — so it counted every feed that + simply hadn't published in an hour, warned permanently, and told an operator nothing. +- The Feeds page filters on All / Erroring / Not Crawled / OK, sorts by `error_count`, and shows the + crawler's actual message, failure count, retry time and last good fetch on the row. `last_ingest_at` + stays, relabelled "Last Item", as what it really is: publishing cadence. +- The existing "Proxy Feeds in Error" tile and its trend series are untouched. They count the + proxy's whole cache, orphaned feeds included, and stay sourced from `proxy_stats` so the series + keeps matching its tile; the new tiles are subscriber-scoped and read D1. ## Operator steps (not code) @@ -178,6 +215,18 @@ step deleting `feeds`/`feed_items` rows whose feed has had **zero active subscri joins on that exact string. - **Ordinary ingest deletes nothing.** A feed at the sanity cap is a bug signal (GUID churn), not steady state — investigate the feed rather than letting it rotate. +- **A health report is the whole trouble set, never a delta.** Recovery is inferred from absence, + so a partial report silently marks feeds healthy. The recovery sweep is a set difference against + the currently flagged feeds (small, partially indexed) rather than a `NOT IN` list of every + healthy feed, which at ~1,300 feeds is the bound-parameter wall `/batch` already hit once. +- **`last_ingest_at` is publishing cadence, not health.** It only moves when a fetch produces a new + or edited item, so a healthy monthly newsletter looks identical to a feed that 404s. Anything + asking "is this feed alive?" has to read `error_count` / `crawl_stale`, which is exactly what the + admin's old stale-ingest alarm got wrong. +- **`nextRetryAt` is milliseconds everywhere the client sees it.** The crawler computes it as + `Date.now() + backoff`; D1 stores seconds like the rest of the backend and converts back on the + way out. Rescaling it a second time is what put every retry ~50,000 years out and made + `canFetch` retire a feed permanently after one transient error. ## Known scaling knob diff --git a/feed-proxy/README.md b/feed-proxy/README.md index 964123b7..bc6668a8 100644 --- a/feed-proxy/README.md +++ b/feed-proxy/README.md @@ -317,6 +317,15 @@ for exactly ONE Worker + D1 pair (prod proxy → prod Worker, staging proxy → - **Pull:** every `CRAWL_SET_INTERVAL_SECONDS` the proxy fetches `GET {INGEST_URL}/api/internal/crawl-set` and stamps `last_requested_at` on each feed, which is what keeps the warm loop working now that reads no longer touch this box. +- **Health:** immediately after each crawl-set pull, the proxy posts every crawl-set feed with + something wrong with it to `POST {INGEST_URL}/api/internal/feed-health` — erroring + (`error_count > 0`), starved (not fetched in `CRAWL_STALE_MS`, 2 h), or both. The erroring ones + are how a reader learns its feed is dead rather than idle, since reads no longer come through here + and a failing feed is otherwise indistinguishable from one that hasn't published; the starved ones + are the admin's alarm that this box can't keep up with its crawl set. The payload is the + **complete** trouble set: the Worker infers recovery from a feed's absence, so a partial report + silently marks feeds healthy. Timestamps are converted from this box's milliseconds to seconds + here, once. - The per-feed cap (`FEED_ITEMS_CAP = 200`) bounds the **outbox**, not the archive: D1 retains everything it has ingested. `push_state` cascades with every `feed_items` delete. diff --git a/feed-proxy/src/index.ts b/feed-proxy/src/index.ts index c0b26b6b..1c8d398f 100644 --- a/feed-proxy/src/index.ts +++ b/feed-proxy/src/index.ts @@ -6,7 +6,7 @@ import { mkdirSync } from 'fs'; import { createApp, initDatabase, cleanupCache } from './app'; import { DocumentFirehose } from './jetstream'; import { pingHeartbeat } from './heartbeat'; -import { pushDirtyItems, pullCrawlSet, type IngestConfig } from './ingest-push'; +import { pushDirtyItems, pullCrawlSet, reportFeedHealth, type IngestConfig } from './ingest-push'; // Config const PROXY_SECRET = process.env.PROXY_SECRET; @@ -215,6 +215,14 @@ if (INGEST_ENABLED) { .then((result) => { if (result.error) console.error(`[Proxy] Crawl-set pull failed: ${result.error}`); else console.log(`[Proxy] Crawl set: ${result.registered} feed(s) registered`); + // Report health AFTER the pull, so a feed registered for the first time + // this cycle is already in the crawl set and its errors are reportable. + // Reads no longer pass through here, so this is the only way a broken + // feed reaches the reader's error badge. + return reportFeedHealth(db, ingestConfig).then((health) => { + if (health.error) console.error(`[Proxy] Feed-health report failed: ${health.error}`); + else console.log(`[Proxy] Feed health: ${health.reported} feed(s) in error`); + }); }) .catch((error) => { console.error('[Proxy] Crawl-set pull error:', error); diff --git a/feed-proxy/src/ingest-push.test.ts b/feed-proxy/src/ingest-push.test.ts index a72a227b..c3ca58e3 100644 --- a/feed-proxy/src/ingest-push.test.ts +++ b/feed-proxy/src/ingest-push.test.ts @@ -12,7 +12,10 @@ import { pushDirtyItems, pullCrawlSet, registerCrawlFeeds, + reportFeedHealth, + CRAWL_STALE_MS, selectDirtyRows, + selectFeedHealth, countDirtyRows, type IngestConfig, } from './ingest-push'; @@ -333,3 +336,126 @@ describe('crawl-set registration', () => { expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM cache').get()?.c).toBe(0); }); }); + +describe('feed health reporting', () => { + let db: Database; + + beforeEach(() => { + db = new Database(':memory:'); + initDatabase(db); + seedCache(db); + }); + + afterEach(() => { + db.close(); + }); + + function breakFeed( + url: string, + opts: { errorCount?: number; error?: string; at?: number; retryAt?: number } = {} + ): void { + const at = opts.at ?? Date.now(); + db.run( + 'UPDATE cache SET error_count = ?, last_error = ?, last_error_at = ?, next_retry_at = ? WHERE url_hash = ?', + [ + opts.errorCount ?? 3, + opts.error ?? 'Failed to fetch (HTTP 404)', + at, + opts.retryAt ?? at + 600_000, + hashUrl(url), + ] + ); + } + + it('reports only broken feeds, converting milliseconds to seconds', () => { + const at = 1_770_000_000_000; + breakFeed(FEED_URL, { errorCount: 4, at, retryAt: at + 600_000 }); + + const health = selectFeedHealth(db); + expect(health).toHaveLength(1); + expect(health[0]).toMatchObject({ + feedUrl: FEED_URL, + errorCount: 4, + lastError: 'Failed to fetch (HTTP 404)', + lastErrorAt: Math.floor(at / 1000), + nextRetryAt: Math.floor((at + 600_000) / 1000), + }); + }); + + it('leaves healthy feeds out entirely — absence is the recovery signal', () => { + seedCache(db, 'https://other.example/feed.xml', 'Other'); + breakFeed(FEED_URL); + + expect(selectFeedHealth(db).map((f) => f.feedUrl)).toEqual([FEED_URL]); + }); + + it('flags a crawl-set feed the warm loop has not fetched in hours', () => { + // Nothing is erroring; the feed is simply losing its turn every tick, which + // is what a capped warm batch does and what the admin needs to see. + const now = Date.now(); + db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [ + now - CRAWL_STALE_MS - 60_000, + URL_HASH, + ]); + + const health = selectFeedHealth(db, now); + expect(health).toHaveLength(1); + expect(health[0]).toMatchObject({ feedUrl: FEED_URL, errorCount: 0, crawlStale: true }); + }); + + it('does not flag a feed fetched within the window', () => { + const now = Date.now(); + db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [now - 60_000, URL_HASH]); + expect(selectFeedHealth(db, now)).toHaveLength(0); + }); + + it('reports a feed that is both erroring and starved', () => { + const now = Date.now(); + breakFeed(FEED_URL); + db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [ + now - CRAWL_STALE_MS - 60_000, + URL_HASH, + ]); + + const health = selectFeedHealth(db, now); + expect(health[0]).toMatchObject({ errorCount: 3, crawlStale: true }); + }); + + it('skips feeds that have dropped out of the crawl set', () => { + // last_requested_at NULL = evicted / never registered, so nobody is + // subscribed and its errors are not the reader's problem. + breakFeed(FEED_URL); + db.run('UPDATE cache SET last_requested_at = NULL WHERE url_hash = ?', [URL_HASH]); + + expect(selectFeedHealth(db)).toHaveLength(0); + }); + + it('posts the set to the Worker, and posts an empty set too', async () => { + const endpoint = mockIngestEndpoint(); + await reportFeedHealth(db, CONFIG); + + expect(endpoint.calls[0].url).toBe('https://api.example/api/internal/feed-health'); + expect(endpoint.calls[0].headers['X-Proxy-Secret']).toBe('test-secret'); + // Everything is healthy: the empty list is what clears the Worker's flags. + expect((endpoint.calls[0].body as unknown as { feeds: unknown[] }).feeds).toEqual([]); + + breakFeed(FEED_URL); + const result = await reportFeedHealth(db, CONFIG); + endpoint.restore(); + + expect(result.reported).toBe(1); + expect( + (endpoint.calls[1].body as unknown as { feeds: Array<{ feedUrl: string }> }).feeds[0].feedUrl + ).toBe(FEED_URL); + }); + + it('surfaces a rejected report instead of pretending it landed', async () => { + breakFeed(FEED_URL); + const endpoint = mockIngestEndpoint(503); + const result = await reportFeedHealth(db, CONFIG); + endpoint.restore(); + + expect(result.reported).toBe(0); + expect(result.error).toContain('503'); + }); +}); diff --git a/feed-proxy/src/ingest-push.ts b/feed-proxy/src/ingest-push.ts index a0293d8c..64260a1e 100644 --- a/feed-proxy/src/ingest-push.ts +++ b/feed-proxy/src/ingest-push.ts @@ -215,6 +215,112 @@ export interface CrawlSetResult { error?: string; } +/** + * One broken feed, as reported to the Worker. Timestamps are unix SECONDS on the + * wire — the cache stores milliseconds (everything here compares against + * `Date.now()`), and the Worker's `feeds` table is in seconds like the rest of + * the backend, so the conversion happens once, here. + */ +export interface FeedHealthReport { + feedUrl: string; + errorCount: number; + lastError: string | null; + lastErrorAt: number | null; + nextRetryAt: number | null; + lastFetchAt: number | null; + // In the crawl set but not fetched in CRAWL_STALE_MS — starved by a saturated + // warm loop rather than failing. `errorCount` can be 0 while this is true. + crawlStale: boolean; +} + +// How long a crawl-set feed may go unfetched before it counts as starved. The +// warm loop works on a minutes-long cadence (a 300s cache TTL refreshed at +// ~180s), so hours without a fetch means this feed is losing its turn every +// tick — the failure mode a capped warm batch produces. +export const CRAWL_STALE_MS = 2 * 60 * 60 * 1000; + +function toSeconds(ms: number | null | undefined): number | null { + return ms ? Math.floor(ms / 1000) : null; +} + +/** + * Every crawl-set feed with something wrong with it: failing to fetch, starved + * of fetches, or both. + * + * Deliberately the whole trouble set rather than a delta: the Worker infers + * recovery from a feed's ABSENCE here, so a feed that starts working again needs + * no message of its own. Only rows still in the crawl set count — a feed evicted + * by `cleanupCache` is nobody's problem any more. + */ +export function selectFeedHealth(db: Database, now = Date.now()): FeedHealthReport[] { + const staleBefore = now - CRAWL_STALE_MS; + return db + .query< + { + url: string; + error_count: number; + last_error: string | null; + last_error_at: number | null; + next_retry_at: number | null; + fetched_at: number | null; + }, + [number] + >( + `SELECT url, error_count, last_error, last_error_at, next_retry_at, fetched_at + FROM cache + WHERE last_requested_at IS NOT NULL + AND (error_count > 0 OR COALESCE(fetched_at, 0) < ?)` + ) + .all(staleBefore) + .map((row) => ({ + feedUrl: row.url, + errorCount: row.error_count, + lastError: row.last_error, + lastErrorAt: toSeconds(row.last_error_at), + nextRetryAt: toSeconds(row.next_retry_at), + // 0 means never fetched — an error placeholder from a first-crawl failure, + // or a row the crawl set just registered. Not a timestamp. + lastFetchAt: toSeconds(row.fetched_at), + crawlStale: (row.fetched_at ?? 0) < staleBefore, + })); +} + +export interface FeedHealthResult { + reported: number; + error?: string; +} + +/** + * Push the trouble set to the paired Worker: the erroring feeds it serves to + * readers, and the starved ones the admin alarms on. An empty list is a + * meaningful report — it is how "everything recovered" is communicated — so this + * always posts. + */ +export async function reportFeedHealth( + db: Database, + config: IngestConfig +): Promise { + const feeds = selectFeedHealth(db); + const headers: Record = { 'Content-Type': 'application/json' }; + if (config.secret) headers['X-Proxy-Secret'] = config.secret; + + try { + const response = await fetch(`${config.ingestUrl}/api/internal/feed-health`, { + method: 'POST', + headers, + body: JSON.stringify({ feeds }), + signal: AbortSignal.timeout(config.timeoutMs ?? DEFAULT_TIMEOUT_MS), + }); + if (!response.ok) return { reported: 0, error: `HTTP ${response.status}` }; + return { reported: feeds.length }; + } catch (error) { + return { + reported: 0, + error: error instanceof Error ? error.message : String(error), + }; + } +} + /** * Pull the crawl set from the paired Worker and stamp every feed in it. */ diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index df7fe5fc..7fabdb0e 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -60,6 +60,14 @@ backfilled through `fetchSingleFeed` → `GET /api/v2/feeds/fetch`, since their global cursor — including ones that arrive from another device (`backfillMissingSubscriptions`, once per feed, ≤ 10 per sync). +Per-feed error state (the sidebar badge and the Manage Sources popover) comes from the response's +`feedHealth`: the set of feeds the crawler currently considers broken. Reads are served from the +archive, so "this feed returned nothing" means nothing — a dead feed and a quiet feed look +identical, and only the server's health verdict distinguishes them. **Absence from that map is what +clears an error**, so `applyHealthSnapshot` runs after the per-feed `markReady` pass and overrides +it. The payload is sent on every cold start and whenever the echoed `health_rev` is stale, so a +steady-state poll pays nothing for it. + The legacy per-feed `/api/v2/feeds/batch` path (`fetchAllFeedsViaBatch`, with per-subscription `feedCursors`) is kept for one release as a fallback: it runs when the timeline 404s and whenever the server reports `ingestActive: false` (this environment's crawler isn't pushing into D1). See diff --git a/frontend/src/lib/services/api.ts b/frontend/src/lib/services/api.ts index a79c5104..0209cfb4 100644 --- a/frontend/src/lib/services/api.ts +++ b/frontend/src/lib/services/api.ts @@ -105,6 +105,23 @@ export interface TimelineResponse { // Feed-level metadata for the caller's subscriptions; present only on a page // that carried items. feeds?: Record; + // Revision of the server's unhealthy-feed set. Echoed back as `health_rev` so + // the payload below is only re-sent when it actually changed. + healthRev?: string; + // Per-feed crawl health for the caller's subscriptions — ONLY the broken ones. + // A subscribed feed missing from this map is healthy, which is how recovery is + // communicated. Present on every cold start and whenever `healthRev` moved. + // Absent entirely on a backend that predates feed-health reporting. + feedHealth?: Record; +} + +/** One broken feed as the timeline reports it. Timestamps are unix ms. */ +export interface TimelineFeedHealth { + errorCount: number; + error?: string; + lastErrorAt?: number; + nextRetryAt?: number; + lastFetchedAt?: number; } export interface ExtractedArticle { @@ -338,12 +355,14 @@ class ApiClient { generation?: string; limit?: number; cold_offset?: number; + health_rev?: string; }): Promise { const search = new URLSearchParams(); if (params.since_seq !== undefined) search.set('since_seq', String(params.since_seq)); if (params.generation) search.set('generation', params.generation); if (params.limit) search.set('limit', String(params.limit)); if (params.cold_offset) search.set('cold_offset', String(params.cold_offset)); + if (params.health_rev) search.set('health_rev', params.health_rev); const query = search.toString(); return this.fetch(`/api/v2/timeline${query ? `?${query}` : ''}`); } diff --git a/frontend/src/lib/services/feedFetcher.ts b/frontend/src/lib/services/feedFetcher.ts index 6fefa742..cfcd5b4a 100644 --- a/frontend/src/lib/services/feedFetcher.ts +++ b/frontend/src/lib/services/feedFetcher.ts @@ -79,6 +79,12 @@ interface TimelineCursor { // or rolled-back Worker), so we stop probing and stay on the legacy batch path. let timelineUnavailable = false; +// Revision of the feed-health set this client has already applied. In memory on +// purpose: feedStatusStore is in-memory too, so a fresh page load starts with no +// revision and the server sends the full health payload — exactly what an empty +// status store needs. +let feedHealthRev: string | undefined; + /** * The whole refresh in ONE request (plus drain pages): `GET /api/v2/timeline` * returns every item newer than the client's global cursor across every @@ -139,8 +145,13 @@ async function fetchTimeline( try { page = await api.fetchTimeline( coldOffset === undefined - ? { since_seq: cursor, generation, limit: TIMELINE_PAGE_LIMIT } - : { cold_offset: coldOffset, limit: TIMELINE_PAGE_LIMIT } + ? { + since_seq: cursor, + generation, + limit: TIMELINE_PAGE_LIMIT, + health_rev: feedHealthRev, + } + : { cold_offset: coldOffset, limit: TIMELINE_PAGE_LIMIT, health_rev: feedHealthRev } ); } catch (e) { if (e instanceof ApiError && e.status === 404) { @@ -186,11 +197,23 @@ async function fetchTimeline( } } - // A feed that just delivered items is demonstrably healthy; clear any stale - // error state. Feeds that delivered nothing are left alone — the archive - // carries no per-feed fetch status (that lives with the crawler). + // A feed that just delivered NEW items is demonstrably healthy. This can't + // stand on its own, though: a broken feed produces nothing, and the archive + // keeps serving its old items, so "delivered nothing" says nothing at all. + // The health payload below is the authoritative signal — applied second, so + // it wins over a cold start replaying archived items from a feed that has + // since broken. for (const feedUrl of feedUrls) feedStatusStore.markReady(feedUrl); + // Per-feed crawl health, as the crawler reports it. Sent on every cold start + // and whenever the unhealthy set changed since our last poll; absent + // otherwise (and on a backend that predates health reporting), in which case + // what we already hold is still current. + if (page.feedHealth) { + feedStatusStore.applyHealthSnapshot(page.feedHealth, subIdByUrl.keys()); + } + if (page.healthRev !== undefined) feedHealthRev = page.healthRev; + if (page.coldStart) { // Keep the first cold page's cursor; page through the rest before committing. if (!coldCursor) coldCursor = { cursor: page.cursor, generation: page.generation }; @@ -675,8 +698,16 @@ export async function fetchSingleFeed( // (a fresh subscription, or the user retrying a feed that looked broken). const feed = await api.fetchFeedV2(subscription.feedUrl, recentGuids, undefined, force); - // Mark as ready - feedStatusStore.markReady(subscription.feedUrl); + // The read succeeded, but reads come from the archive: a feed can serve its + // last-known items for days after it stopped crawling. Trust the crawler's + // verdict when there is one, and only claim "ready" when there isn't. + if (feed.health) { + feedStatusStore.applyHealthSnapshot({ [subscription.feedUrl]: feed.health }, [ + subscription.feedUrl, + ]); + } else { + feedStatusStore.markReady(subscription.feedUrl); + } // Update subscription title/siteUrl from feed metadata if (feed.title && shouldUpdateTitle(subscription.title, subscription.feedUrl, feed.title)) { diff --git a/frontend/src/lib/services/timelineSync.test.ts b/frontend/src/lib/services/timelineSync.test.ts index 3d27c5db..7a4f8993 100644 --- a/frontend/src/lib/services/timelineSync.test.ts +++ b/frontend/src/lib/services/timelineSync.test.ts @@ -2,8 +2,10 @@ import { describe, it, expect } from 'vitest'; import { buildSubscriptionIndex, groupTimelineItems, + isCircuitOpen, isRssSubscription, pruneAttemptedBackfills, + reconcileFeedHealth, selectBackfillTargets, shouldFallBackToBatch, shouldUpdateTitle, @@ -181,6 +183,66 @@ describe('pruneAttemptedBackfills', () => { }); }); +describe('reconcileFeedHealth', () => { + const broken = { errorCount: 3, error: 'Failed to fetch (HTTP 404)' }; + + it('flags every subscribed feed the crawler reports broken', () => { + const decisions = reconcileFeedHealth( + { 'https://a.example/f': broken }, + ['https://a.example/f'], + () => false + ); + expect(decisions).toEqual([{ feedUrl: 'https://a.example/f', kind: 'error', health: broken }]); + }); + + it('clears a feed that has dropped out of the report', () => { + const decisions = reconcileFeedHealth({}, ['https://a.example/f'], () => true); + expect(decisions).toEqual([{ feedUrl: 'https://a.example/f', kind: 'recovered' }]); + }); + + it('leaves a healthy feed with no error alone, so pending never becomes a fake success', () => { + expect(reconcileFeedHealth({}, ['https://a.example/f'], () => false)).toEqual([]); + }); + + it('ignores broken feeds the caller does not subscribe to', () => { + const decisions = reconcileFeedHealth( + { 'https://other.example/f': broken }, + ['https://a.example/f'], + () => false + ); + expect(decisions).toEqual([]); + }); + + it('handles a mixed report in one pass', () => { + const decisions = reconcileFeedHealth( + { 'https://a.example/f': broken }, + ['https://a.example/f', 'https://b.example/f', 'https://c.example/f'], + (url) => url === 'https://b.example/f' + ); + expect(decisions).toEqual([ + { feedUrl: 'https://a.example/f', kind: 'error', health: broken }, + { feedUrl: 'https://b.example/f', kind: 'recovered' }, + ]); + }); +}); + +describe('isCircuitOpen', () => { + const now = 1_770_000_000_000; + + it('treats nextRetryAt as milliseconds, not seconds', () => { + // The crawler sends `Date.now() + backoff`. Rescaling this by 1000 (the old + // behaviour) put the retry ~50,000 years out and retired the feed for good. + expect(isCircuitOpen(now + 600_000, now)).toBe(true); + expect(isCircuitOpen(now - 600_000, now)).toBe(false); + expect(isCircuitOpen(Math.floor(now / 1000), now)).toBe(false); + }); + + it('is closed when there is no retry time at all', () => { + expect(isCircuitOpen(undefined, now)).toBe(false); + expect(isCircuitOpen(0, now)).toBe(false); + }); +}); + describe('shouldUpdateTitle', () => { it('replaces a URL or hostname placeholder with a real title', () => { expect(shouldUpdateTitle(FEED_A, FEED_A, 'Real Title')).toBe(true); diff --git a/frontend/src/lib/services/timelineSync.ts b/frontend/src/lib/services/timelineSync.ts index c49d2b4a..11b1789d 100644 --- a/frontend/src/lib/services/timelineSync.ts +++ b/frontend/src/lib/services/timelineSync.ts @@ -156,6 +156,61 @@ export function pruneAttemptedBackfills( return [...attempted].filter((url) => live.has(url)); } +/** One broken feed as the timeline reports it. Timestamps are unix ms. */ +export interface TimelineFeedHealth { + errorCount: number; + error?: string; + lastErrorAt?: number; + nextRetryAt?: number; + lastFetchedAt?: number; +} + +export type FeedHealthDecision = + | { feedUrl: string; kind: 'error'; health: TimelineFeedHealth } + | { feedUrl: string; kind: 'recovered' }; + +/** + * Reconcile the client's per-feed status against the crawler's health report. + * + * The batch path learned a feed was broken from the response that failed to + * fetch it. Timeline reads are served from the archive and never touch the + * crawler, so "this feed returned nothing" carries no information — a dead feed + * and a quiet feed look identical. The server therefore sends the set it + * considers broken, and everything else the caller subscribes to is healthy by + * omission. That omission is what clears an error, including one left over from + * the batch path. + * + * A feed with no status yet (never fetched, or still 'pending') is left alone + * when it isn't in the report: absence means "not broken", not "confirmed + * fetched", and inventing a success would show a fetch that never happened. + */ +export function reconcileFeedHealth( + unhealthy: Record, + subscribedFeedUrls: Iterable, + hasError: (feedUrl: string) => boolean +): FeedHealthDecision[] { + const decisions: FeedHealthDecision[] = []; + for (const feedUrl of subscribedFeedUrls) { + const health = unhealthy[feedUrl]; + if (health) decisions.push({ feedUrl, kind: 'error', health }); + else if (hasError(feedUrl)) decisions.push({ feedUrl, kind: 'recovered' }); + } + return decisions; +} + +/** + * Is this feed in its retry cooldown? + * + * `nextRetryAt` is unix MILLISECONDS all the way from the crawler, which + * computes it as `Date.now() + backoff`. It used to be compared against (and + * rescaled by) seconds, which put every retry roughly fifty thousand years out: + * the feed then failed `canFetch` forever, so one transient error retired it for + * the life of the tab and the error popover offered a nonsense countdown. + */ +export function isCircuitOpen(nextRetryAt: number | undefined, now: number): boolean { + return !!nextRetryAt && nextRetryAt > now; +} + /** * Whether a subscription's title should be updated from feed metadata. * Returns true when the current title is a fallback (URL, hostname, etc.) diff --git a/frontend/src/lib/stores/feedStatus.svelte.ts b/frontend/src/lib/stores/feedStatus.svelte.ts index e25aaacb..cc7cc4f6 100644 --- a/frontend/src/lib/stores/feedStatus.svelte.ts +++ b/frontend/src/lib/stores/feedStatus.svelte.ts @@ -8,6 +8,12 @@ * - nextRetryAt?: number (Unix timestamp) */ +import { + isCircuitOpen, + reconcileFeedHealth, + type TimelineFeedHealth, +} from '$lib/services/timelineSync'; + export type FeedStatusType = 'ready' | 'pending' | 'error' | 'circuit-open'; export type ErrorType = 'transient' | 'permanent'; @@ -97,6 +103,12 @@ export interface V2FeedResult { hasMore?: boolean; } +/** + * One broken feed as the timeline reports it (`feedHealth` on the response). + * Timestamps are unix ms. + */ +export type FeedHealthSnapshot = TimelineFeedHealth; + function createFeedStatusStore() { let statuses = $state>(new Map()); @@ -149,25 +161,86 @@ function createFeedStatusStore() { lastCheckedAt: now, }); } else { - // Error response - const errorType = classifyError(result.error); - const isCircuitOpen = result.nextRetryAt && result.nextRetryAt > now / 1000; - - statuses.set(feedUrl, { - status: isCircuitOpen ? 'circuit-open' : 'error', - errorCount: result.errorCount || 1, - errorMessage: result.error, - errorType, - nextRetryAt: result.nextRetryAt ? result.nextRetryAt * 1000 : undefined, // Convert to ms - lastFetchedAt: result.lastFetchedAt, - lastCheckedAt: now, - }); + // Error response. `nextRetryAt` is already unix MILLISECONDS — the proxy + // computes it as `Date.now() + backoff` and passes it through untouched. + // It used to be re-scaled by 1000 here, which put every retry ~50,000 years + // out: `canFetch` then refused the feed forever and the popover offered an + // absurd countdown, so a feed that hit one transient error was never + // retried again for the life of the tab. + statuses.set( + feedUrl, + buildErrorStatus(feedUrl, now, { + errorCount: result.errorCount || 1, + error: result.error, + nextRetryAt: result.nextRetryAt, + lastFetchedAt: result.lastFetchedAt, + }) + ); } // Trigger reactivity statuses = new Map(statuses); } + /** + * Shared shape for "this feed is broken", from either the legacy batch + * response or the timeline's health payload. + */ + function buildErrorStatus(feedUrl: string, now: number, health: FeedHealthSnapshot): FeedStatus { + return { + status: isCircuitOpen(health.nextRetryAt, now) ? 'circuit-open' : 'error', + errorCount: health.errorCount || 1, + errorMessage: health.error, + errorType: classifyError(health.error), + nextRetryAt: health.nextRetryAt, + lastFetchedAt: health.lastFetchedAt ?? statuses.get(feedUrl)?.lastFetchedAt, + lastCheckedAt: now, + }; + } + + /** + * Apply the timeline's per-feed health for a whole subscription set. + * + * The timeline path has no per-request feed status to report — reads are served + * from the archive and never touch the crawler — so the server sends the set of + * feeds it currently considers broken and this reconciles against it. Only the + * broken ones are listed, so a subscribed feed that is ABSENT is healthy: that + * is what clears an error once a feed starts working again, including one + * inherited from the legacy batch path. + * + * Feeds the crawler hasn't reached yet are simply not in `subscribedFeedUrls`'s + * intersection with any known status, so they keep whatever state they had + * (usually 'pending') rather than being asserted healthy. + */ + function applyHealthSnapshot( + unhealthy: Record, + subscribedFeedUrls: Iterable + ): void { + const now = Date.now(); + const decisions = reconcileFeedHealth(unhealthy, subscribedFeedUrls, (feedUrl) => { + const status = statuses.get(feedUrl)?.status; + return status === 'error' || status === 'circuit-open'; + }); + if (decisions.length === 0) return; + + for (const decision of decisions) { + if (decision.kind === 'error') { + statuses.set(decision.feedUrl, buildErrorStatus(decision.feedUrl, now, decision.health)); + } else { + // Recovered. Keep the last known fetch time rather than stamping one: + // the report says the feed is no longer broken, not that we just read it. + statuses.set(decision.feedUrl, { + status: 'ready', + errorCount: 0, + lastFetchedAt: statuses.get(decision.feedUrl)?.lastFetchedAt, + lastCheckedAt: now, + }); + } + } + + statuses = new Map(statuses); + } + /** * Mark a feed as pending (initial state for new subscriptions) */ @@ -402,6 +475,7 @@ function createFeedStatusStore() { return permanentErrorFeeds; }, updateFromV2Result, + applyHealthSnapshot, markPending, markReady, markError, diff --git a/frontend/src/lib/types/index.ts b/frontend/src/lib/types/index.ts index d029bda1..b8acfb89 100644 --- a/frontend/src/lib/types/index.ts +++ b/frontend/src/lib/types/index.ts @@ -857,6 +857,17 @@ export interface ParsedFeed { imageUrl?: string; items: FeedItem[]; fetchedAt: number; + // Set only when the crawler currently considers this feed broken. A single-feed + // read is served from the archive, so it can succeed (with stale items) for a + // feed that has been failing to crawl for days — this is what says so. + // Timestamps are unix ms. + health?: { + errorCount: number; + error?: string; + lastErrorAt?: number; + nextRetryAt?: number; + lastFetchedAt?: number; + }; } export interface FeedItem { From e5b7653aca8f38fe09b87536fb9eb0e559825834 Mon Sep 17 00:00:00 2001 From: Tim Disney Date: Mon, 17 Aug 2026 15:31:48 -0700 Subject: [PATCH 7/7] Gate the timeline rollout on sync_state, not just the crawler heartbeat A fresh crawler heartbeat only says a crawler is attached. The proxy pulls the crawl set immediately at boot, so the first stamp lands seconds into a backfill that takes hours -- which meant `ingestActive` switched every reader onto the timeline at the moment the archive was emptiest, and each of them would then drag the entire backfill through the incremental drain: the fan-out-on-read scan at its worst case, for the duration, surfacing back-catalogue items as unread along the way. There was also no way back short of a Worker rollback or stopping the crawler for 30 minutes. `sync_state.timeline_enabled` (migration 0071) separates the two. `ingestActive` is now the AND of a fresh heartbeat and this flag, so ingest and admission can be sequenced: enable INGEST_URL, let the archive fill, watch ingest.pending trend to ~0, then open the gate with one UPDATE. Setting it back to '0' returns every client to the legacy batch path at its next poll, with no deploy. The wire contract is unchanged -- `ingestActive: false` has always meant "stay on /batch" -- so no frontend change. A gated request short-circuits rather than building a page the client is about to discard, which is what keeps the gated window (most of the rollout) at one sync_state read per poll. Only an explicit '0' gates; an absent row is open, so a hand-built schema or a future environment is never silently held back. The migration writes '0' for a database that already has users, so prod and staging start shut while local dev, e2e and CI start open. dev-local.sh and the e2e seed set it anyway, since a local database you have logged into before would otherwise quietly serve the path we are retiring. Co-Authored-By: Claude Opus 5 (1M context) --- backend/CLAUDE.md | 14 ++-- .../migrations/0071_timeline_rollout_gate.sql | 30 +++++++++ backend/src/routes/ingest.ts | 11 ++++ backend/src/routes/timeline.ts | 60 +++++++++++++---- backend/test/feed-timeline.spec.ts | 64 +++++++++++++++++++ docs/RUNBOOK.md | 36 +++++++++++ docs/plans/D1_FEED_TIMELINE.md | 42 ++++++++++-- e2e/seed.ts | 5 ++ scripts/dev-local.sh | 9 +++ 9 files changed, 247 insertions(+), 24 deletions(-) create mode 100644 backend/migrations/0071_timeline_rollout_gate.sql diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 4ba0eb56..3819d64a 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -14,10 +14,13 @@ crawl (`GET /api/internal/crawl-set`), and reports which feeds are failing to cr subscriptions and read state. Because reads never touch the crawler, a broken feed just goes quiet; the health report is the only thing that tells a reader its feed is dead rather than idle, and its payload is the COMPLETE unhealthy set (recovery = absence from the next report). All three internal -endpoints stamp `sync_state.crawler_heartbeat_at`, and -the timeline reports `ingestActive` from it: an environment whose proxy has no `INGEST_URL` tells -clients to stay on the legacy batch path instead of reading an archive nothing fills. See -`docs/plans/D1_FEED_TIMELINE.md`. The proxy is still on the path for `/api/extract`, feed +endpoints stamp `sync_state.crawler_heartbeat_at`, and the timeline reports `ingestActive` as the +AND of that heartbeat and `sync_state.timeline_enabled` — the operator's rollout gate. A crawler +heartbeat only says a crawler is attached; it arrives seconds into a backfill that takes hours, so +the gate is what actually admits readers to the archive (and, set back to `'0'`, is the fast +rollback that returns every client to the legacy batch path with no deploy). Either half false and +clients stay on `/batch`; the request short-circuits rather than building a page they will discard. +See `docs/plans/D1_FEED_TIMELINE.md`. The proxy is still on the path for `/api/extract`, feed discovery, standard.site documents, and social context — plus the one crawl per new subscription (`warmFeedIntoArchive`) and the subscription-gated pull-through in `/api/v2/feeds/fetch`. @@ -167,7 +170,8 @@ Key tables: reads now live in `item_labels_cache` as `item_type='document'`/`label='read'`) - `user_settings` - User preferences - `rate_limits` - Per-user rate limiting -- `sync_state` - Jetstream cursor, the archive generation token, and the crawler heartbeat +- `sync_state` - Jetstream cursor, the archive generation token, the crawler heartbeat, and + `timeline_enabled` (the rollout gate: only an explicit `'0'` holds clients on the batch path) - `system_status` - Cron-written health board (cron liveness, poller lag, proxy stats) - `metrics_snapshots` - Hourly trend points behind the admin's sparklines (90-day retention) diff --git a/backend/migrations/0071_timeline_rollout_gate.sql b/backend/migrations/0071_timeline_rollout_gate.sql new file mode 100644 index 00000000..8f19dc38 --- /dev/null +++ b/backend/migrations/0071_timeline_rollout_gate.sql @@ -0,0 +1,30 @@ +-- Rollout gate for the D1-served timeline. +-- +-- `ingestActive` alone flips every client onto the timeline the instant the +-- crawler first checks in — which is the moment the backfill STARTS, not the +-- moment it finishes. The proxy stamps a heartbeat on its very first crawl-set +-- pull (seconds after the release), so without this gate every reader cold-starts +-- against a nearly-empty archive and then drags the entire ~200-items-per-feed +-- backfill through its incremental drain: the expensive global scan, at its worst +-- case, for as long as the backfill takes. +-- +-- This decouples the two. Enable INGEST_URL, let the archive fill, watch the +-- proxy's `ingest.pending` trend to ~0, THEN flip this to '1'. It is also the +-- only fast way back: setting it to '0' returns every client to the legacy batch +-- path on their next poll, with no Worker deploy and no waiting out the 30-minute +-- heartbeat freshness window. +-- +-- npx wrangler d1 execute skyreader --remote \ +-- --command "UPDATE sync_state SET value='1', updated_at=unixepoch() WHERE key='timeline_enabled'" +-- +-- Only an explicit '0' gates: an absent row means enabled, so a hand-built schema +-- or a future environment can't be silently held on the legacy path. +-- +-- The value depends on whether this deployment already has readers. An existing +-- one (prod, staging) has users and a backfill ahead of it, so it starts gated and +-- an operator opens it deliberately. A fresh one (local dev, e2e, CI) has no users +-- and no backlog, so it starts open and nothing has to know this key exists. +INSERT OR IGNORE INTO sync_state (key, value, updated_at) +SELECT 'timeline_enabled', + CASE WHEN EXISTS (SELECT 1 FROM users) THEN '0' ELSE '1' END, + unixepoch(); diff --git a/backend/src/routes/ingest.ts b/backend/src/routes/ingest.ts index 20644d1e..28ccc783 100644 --- a/backend/src/routes/ingest.ts +++ b/backend/src/routes/ingest.ts @@ -31,6 +31,17 @@ export const MAX_ITEM_CONTENT_BYTES = 8 * 1024; export const CRAWLER_HEARTBEAT_KEY = 'crawler_heartbeat_at'; export const CRAWLER_HEARTBEAT_FRESH_SECONDS = 30 * 60; +// The rollout gate (migration 0071). A crawler heartbeat says a crawler is +// ATTACHED; it says nothing about whether the archive it fills is complete, and +// the first heartbeat lands seconds into a backfill that takes hours. This key is +// what actually moves readers onto the timeline, so the two can be sequenced: +// enable ingest, let the archive fill, then open the gate. Setting it back to '0' +// is also the fast rollback — every client is on the batch path at its next poll. +// +// Only an explicit '0' gates. An absent row means enabled, so a hand-built schema +// (the unit-test harness) or a new environment is never silently held back. +export const TIMELINE_ENABLED_KEY = 'timeline_enabled'; + // Revision token for the set of feeds the crawler currently considers broken. // The timeline sends the per-feed health payload only when the client's echoed // revision differs from this, so a steady-state poll costs no extra query. diff --git a/backend/src/routes/timeline.ts b/backend/src/routes/timeline.ts index 63cd2a05..cbeefa04 100644 --- a/backend/src/routes/timeline.ts +++ b/backend/src/routes/timeline.ts @@ -3,6 +3,7 @@ import { CRAWLER_HEARTBEAT_KEY, CRAWLER_HEARTBEAT_FRESH_SECONDS, FEED_HEALTH_REV_KEY, + TIMELINE_ENABLED_KEY, rssSubscriptionPredicate, } from './ingest'; @@ -128,37 +129,47 @@ export interface ArchiveState { // True when this environment's crawler has checked in recently. False means // nothing is filling this D1 (no INGEST_URL on the paired proxy, or the proxy // is down), so the client must not treat an empty/partial archive as the truth. - ingestActive: boolean; + crawlerFresh: boolean; + // The operator's rollout gate (`timeline_enabled`, migration 0071). A fresh + // heartbeat only proves a crawler is attached — it arrives seconds into a + // backfill that takes hours — so this is the switch that actually moves readers + // onto the timeline, and the switch that moves them back. + timelineEnabled: boolean; // Revision of the unhealthy-feed set. A client that echoes this back unchanged // already holds current health and is sent no health payload. healthRev: string; } /** - * Generation token, crawler liveness and the feed-health revision in one - * `sync_state` read (the timeline needs all three on every request, and they are - * three rows of the same small table). + * Generation token, crawler liveness, the rollout gate and the feed-health + * revision in one `sync_state` read (the timeline needs all four on every + * request, and they are four rows of the same small table). */ export async function readArchiveState(env: Env): Promise { const rows = await env.DB.prepare( - `SELECT key, value FROM sync_state WHERE key IN ('items_generation', ?, ?)` + `SELECT key, value FROM sync_state WHERE key IN ('items_generation', ?, ?, ?)` ) - .bind(CRAWLER_HEARTBEAT_KEY, FEED_HEALTH_REV_KEY) + .bind(CRAWLER_HEARTBEAT_KEY, FEED_HEALTH_REV_KEY, TIMELINE_ENABLED_KEY) .all<{ key: string; value: string }>(); let generation = ''; let heartbeat = 0; let healthRev = ''; + // Absent means enabled: only an explicit '0' holds clients on the batch path, + // so an environment that never learned about this key behaves as it did before. + let timelineEnabled = true; for (const row of rows.results) { if (row.key === 'items_generation') generation = row.value; else if (row.key === CRAWLER_HEARTBEAT_KEY) heartbeat = parseInt(row.value, 10) || 0; else if (row.key === FEED_HEALTH_REV_KEY) healthRev = row.value; + else if (row.key === TIMELINE_ENABLED_KEY) timelineEnabled = row.value !== '0'; } const age = Math.floor(Date.now() / 1000) - heartbeat; return { generation, - ingestActive: heartbeat > 0 && age <= CRAWLER_HEARTBEAT_FRESH_SECONDS, + crawlerFresh: heartbeat > 0 && age <= CRAWLER_HEARTBEAT_FRESH_SECONDS, + timelineEnabled, healthRev, }; } @@ -318,16 +329,41 @@ export async function handleTimeline( const coldOffset = Number.isInteger(parsedColdOffset) && parsedColdOffset > 0 ? parsedColdOffset : 0; - const { generation, ingestActive, healthRev } = await readArchiveState(env); + const { generation, crawlerFresh, timelineEnabled, healthRev } = await readArchiveState(env); + // Two independent conditions, one wire field. The client's contract is + // unchanged — `ingestActive: false` has always meant "stay on the batch path" — + // and it does not need to know WHY, only the operator does. + const ingestActive = timelineEnabled && crawlerFresh; + + // Server time (unix seconds) at annotation. The client seeds its forward + // read-delta cursor from this, exactly as /batch does today, so the delta + // starts from bootstrap with no client/server clock skew. + const readCursor = Math.floor(Date.now() / 1000); + + // A client told `ingestActive: false` discards the page and refetches through + // /batch, so building one is pure waste — and during a gated rollout that waste + // is every reader, every poll, for as long as the gate stays shut. Answer with + // the state and nothing else. `coldStart: true` keeps the older + // empty-cold-start heuristic pointing the same way as the flag, so a client + // reading either signal reaches the same conclusion. + if (!ingestActive) { + return json({ + items: [], + cursor: 0, + generation, + ingestActive, + hasMore: false, + readCursor, + coldStart: true, + healthRev, + }); + } + // Send health when the client's copy is stale. A cold start always gets it: // it is the one page that delivers already-archived items for a feed that may // have broken since, and its blanket "these feeds delivered, so they're fine" // pass would otherwise clear a live error. const healthStale = healthRevParam !== healthRev; - // Server time (unix seconds) at annotation. The client seeds its forward - // read-delta cursor from this, exactly as /batch does today, so the delta - // starts from bootstrap with no client/server clock skew. - const readCursor = Math.floor(Date.now() / 1000); const incremental = sinceSeq !== undefined && diff --git a/backend/test/feed-timeline.spec.ts b/backend/test/feed-timeline.spec.ts index 211e346d..236fd4d7 100644 --- a/backend/test/feed-timeline.spec.ts +++ b/backend/test/feed-timeline.spec.ts @@ -9,6 +9,7 @@ import { ingestProxyFeed, CRAWLER_HEARTBEAT_KEY, FEED_HEALTH_REV_KEY, + TIMELINE_ENABLED_KEY, } from '../src/routes/ingest'; import { handleTimeline, readFeedSlice } from '../src/routes/timeline'; import type { Env, FeedItem, Session } from '../src/types'; @@ -135,6 +136,19 @@ async function clearCrawlerHeartbeat() { await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(CRAWLER_HEARTBEAT_KEY).run(); } +async function setTimelineGate(value: string) { + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at` + ) + .bind(TIMELINE_ENABLED_KEY, value) + .run(); +} + +async function clearTimelineGate() { + await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(TIMELINE_ENABLED_KEY).run(); +} + async function reportHealth(feeds: unknown[], secret: string | null = SECRET) { const headers: Record = { 'Content-Type': 'application/json' }; if (secret !== null) headers['X-Proxy-Secret'] = secret; @@ -185,6 +199,9 @@ describe('feed timeline (D1 ingest + serve)', () => { await env.DB.prepare('DELETE FROM item_labels_cache').run(); await clearCrawlerHeartbeat(); await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(FEED_HEALTH_REV_KEY).run(); + // Absent is the open position, so this restores the default the rest of the + // suite runs under. + await clearTimelineGate(); }); describe('ingest auth', () => { @@ -621,6 +638,53 @@ describe('feed timeline (D1 ingest + serve)', () => { }); }); + describe('rollout gate (timeline_enabled)', () => { + it('holds clients on the batch path while the gate is shut, however live the crawler', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('g1'), contentHash: 'g1' }]); + expect((await timeline()).ingestActive).toBe(true); + + await setTimelineGate('0'); + const gated = await timeline(); + expect(gated.ingestActive).toBe(false); + // The whole point of the short-circuit: a page the client is about to throw + // away is never built, so the gated window costs one sync_state read. + expect(gated.items).toEqual([]); + // Both fallback signals agree, so a client reading either one stays put. + expect(gated.coldStart).toBe(true); + expect(gated.hasMore).toBe(false); + }); + + it('reopens without a deploy, serving the archive that filled while it was shut', async () => { + await addSubscription(TEST_DID, FEED_A); + await setTimelineGate('0'); + // Ingest keeps running while the gate is shut — that is the sequencing the + // gate exists to allow: fill the archive first, admit readers second. + await ingest(FEED_A, [{ item: item('g2'), contentHash: 'g2' }]); + expect((await timeline()).items).toEqual([]); + + await setTimelineGate('1'); + const open = await timeline(); + expect(open.ingestActive).toBe(true); + expect(open.items.map((i) => i.guid)).toEqual(['g2']); + }); + + it('is open when the row is absent, so an environment that never set it is unaffected', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('g3'), contentHash: 'g3' }]); + await clearTimelineGate(); + expect((await timeline()).ingestActive).toBe(true); + }); + + it('still requires a live crawler when open', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('g4'), contentHash: 'g4' }]); + await setTimelineGate('1'); + await clearCrawlerHeartbeat(); + expect((await timeline()).ingestActive).toBe(false); + }); + }); + describe('feed health', () => { it('requires the shared secret', async () => { expect((await reportHealth([brokenFeed(FEED_A)], null)).status).toBe(401); diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index d0f23f54..bbf7863a 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -424,6 +424,7 @@ walk when the reader looks stale but every tile above is green. | --------------------------- | ----------------------------------------------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | Crawler talking to us | `sync_state.crawler_heartbeat_at` (D1) | stamped within ~5 min | `npx wrangler d1 execute skyreader --remote --command "SELECT * FROM sync_state WHERE key = 'crawler_heartbeat_at'"` | | Clients reading the archive | `GET /api/v2/timeline` → `ingestActive` | `true` in an ingesting environment | Any authenticated timeline response | +| Rollout gate | `sync_state.timeline_enabled` (D1) | `'1'` once rolled out | `npx wrangler d1 execute skyreader --remote --command "SELECT * FROM sync_state WHERE key = 'timeline_enabled'"` | | Push backlog (outbox) | proxy `GET /stats` → `ingest.pending` | near zero in steady state | `curl -H "X-Proxy-Secret: $SECRET" https://skyreader-feed-proxy.fly.dev/stats` | | Push failing | Sentry `source: ingest-push`, proxy logs | silent | `fly logs -a skyreader-feed-proxy` → `Ingest push failed` | | Feeds gone quiet | Admin → Feeds, "Subscribed Feeds Not Ingesting" | small and stable | The dashboard's Feeds metrics | @@ -453,6 +454,41 @@ Fix, once you know which one it is: a wedged warm loop or push loop is `fly machine restart ` (never `fly scale count` — see §4's proxy-warmer entry); a heartbeat that never arrives is configuration, not a restart. +### The timeline rollout gate + +`ingestActive` is the AND of two things: a fresh crawler heartbeat and +`sync_state.timeline_enabled`. They are separate because the heartbeat lands +_seconds_ into the first backfill of an environment and that backfill takes hours +— without the gate, every reader switches to the timeline at the moment the +archive is emptiest and then drags the whole backfill through the incremental +drain (the expensive global scan, at its worst case, for the duration). + +Only an explicit `'0'` gates. Migration 0071 sets it for a database that already +has users, so prod and staging start shut and a fresh environment (local dev, +e2e, CI) starts open. + +```bash +# Open it — after ingest.pending has trended to ~0. +npx wrangler d1 execute skyreader --remote --command \ + "UPDATE sync_state SET value='1', updated_at=unixepoch() WHERE key='timeline_enabled'" + +# Shut it — every client is back on the legacy batch path at its next poll. +npx wrangler d1 execute skyreader --remote --command \ + "UPDATE sync_state SET value='0', updated_at=unixepoch() WHERE key='timeline_enabled'" +``` + +Shutting it is the **fast rollback for the read path**, and the first thing to +reach for if the timeline misbehaves after a rollout: no Worker deploy, no waiting +out the 30-minute heartbeat freshness window, and the crawler keeps filling the +archive the whole time. It is not a fix for a bad Worker deploy generally — only +for "readers should not be on the timeline right now". + +One caveat when shutting it: clients hold a committed `timelineCursor` and stop +advancing their per-subscription `feedCursors` while on the timeline, so the +batch path re-drains from wherever those cursors were left. The proxy's K=200 +window bounds that, and the merge dedupes by GUID, so the cost is one heavier +sync, not duplicates. + --- ## 5. Post-deploy smoke checks diff --git a/docs/plans/D1_FEED_TIMELINE.md b/docs/plans/D1_FEED_TIMELINE.md index 87a4dd7a..7d6217ec 100644 --- a/docs/plans/D1_FEED_TIMELINE.md +++ b/docs/plans/D1_FEED_TIMELINE.md @@ -53,9 +53,11 @@ read state (`getReadKeys`). Now: incremental drain (cursor derived from returned rows, `hasMore` via `limit+1`) and a **paged** per-feed newest-30 cold start (feeds walked in a stable order, `COLD_START_MAX_ITEMS` per page, continuation via `nextColdOffset`). Read state is an `EXISTS` probe in the same query; - `getReadKeys` is never called on the feed path. Every response carries `ingestActive`, derived - from the heartbeat: false means this deployment has no crawler filling D1, and clients stay on - the legacy batch path. A cursor above the archive head cold-starts (rewound-archive guard). + `getReadKeys` is never called on the feed path. Every response carries `ingestActive` — the AND of + a fresh crawler heartbeat (this deployment has a crawler filling D1) and the `timeline_enabled` + rollout gate (an operator has admitted readers to it; see Phase 4). False on either half means + clients stay on the legacy batch path, and the request short-circuits rather than building a page + they will discard. A cursor above the archive head cold-starts (rewound-archive guard). - `routes/ingest.ts` also serves `POST /api/internal/feed-health`: the crawler's periodic report of every feed it currently considers broken, which the timeline hands to readers. On the batch path a failing feed came back with `status: 'error'` inline; reads no longer touch the proxy, and a @@ -74,7 +76,7 @@ read state (`getReadKeys`). Now: arbitrary feeds. Subscribe time already crawls + ingests the feed (`warmFeedIntoArchive`, which replaced the old warm-and-discard), so the pull-through is normally not needed at all. The response carries the feed's `health` when it has any, and a successful pull-through clears it - (`clearFeedHealth`) — that path *is* the user's "retry this feed" action, so it must show a result + (`clearFeedHealth`) — that path _is_ the user's "retry this feed" action, so it must show a result now rather than after the crawler's next report. **Proxy** (`feed-proxy/`) @@ -181,10 +183,32 @@ client commits a cursor. That signal is server-side on purpose — subscribe-tim pull-through both write to the archive, so "the archive is empty for this user" would stop being true long before the crawler existed. +### Phase 4 — open the gate + The heartbeat means a crawler is attached; it does **not** mean the initial archive backfill is -complete. After enabling prod, watch `/stats` and wait for `ingest.pending` to trend to ~0 before -announcing the rollout. New or cleared clients remain correct while it drains, but their first cold -start can be sparse and will fill in over subsequent syncs. +complete, and the first stamp lands _seconds_ after the release (the proxy pulls the crawl set +immediately at boot). On its own it would therefore switch every reader onto the timeline at the +moment the archive is emptiest, and each of them would then drag the entire backfill through the +incremental drain — the fan-out-on-read scan at its worst case, for the hours the drain takes, +surfacing back-catalogue items as unread along the way. + +`sync_state.timeline_enabled` (migration `0071`) separates the two. `ingestActive` is the AND of a +fresh heartbeat and this flag, so: + +1. Enable `INGEST_URL`; the crawler fills the archive while every client stays on the batch path. +2. Watch `/stats` until `ingest.pending` trends to ~0. +3. Open the gate — one `UPDATE sync_state`, no deploy (commands in + [`RUNBOOK.md` §4d](../RUNBOOK.md)). Clients switch on their next poll. + +Only an explicit `'0'` gates; an absent row means enabled, so a hand-built schema or a future +environment is never silently held back. The migration writes `'0'` for a database that already has +users, so prod and staging start shut while local dev, e2e and CI start open. + +Setting it back to `'0'` is the **fast rollback for the read path** — every client returns to the +batch path at its next poll, with no Worker deploy and no waiting out the 30-minute heartbeat +freshness window, and the crawler keeps ingesting throughout. A gated timeline request +short-circuits: it answers with the state and an empty page rather than building one the client is +about to discard, so the gated window costs one `sync_state` read per poll. ### Phase 5 — cleanup (a later release, once no legacy traffic remains) @@ -200,6 +224,10 @@ step deleting `feeds`/`feed_items` rows whose feed has had **zero active subscri and silently skips rows. A cold start is the one exception, and only because it reads the head BEFORE its per-feed slices: anything ingested while it pages lands above that head and arrives on the next poll. +- **The heartbeat and the gate answer different questions.** "Is a crawler attached?" is not "is + the archive ready for readers?" — the first stamp arrives seconds into a backfill that takes + hours. Anything that collapses the two back into one signal reintroduces a rollout where every + client switches at the emptiest moment and then drains the entire backfill. - **Any D1 restore bumps `items_generation`** (one `UPDATE sync_state`): Time Travel rewinds seqs while the token would otherwise stay the same. The timeline also self-heals a cursor that sits above the head by cold-starting that client, so a forgotten bump degrades to one extra cold start diff --git a/e2e/seed.ts b/e2e/seed.ts index 7056f54a..13d72c5f 100644 --- a/e2e/seed.ts +++ b/e2e/seed.ts @@ -163,6 +163,11 @@ export async function seedFeedItems( // reports `ingestActive: false` and the client (correctly) stays on the legacy // batch path, which is not what these tests are exercising. `INSERT INTO sync_state (key, value, updated_at) VALUES ('crawler_heartbeat_at', ${sqlString(String(nowSeconds))}, ${nowSeconds}) ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`, + // The other half of the same gate: a heartbeat says a crawler is attached, + // this says readers may use what it wrote. Migration 0071 leaves it open on a + // fresh database, but a test DB that predates the migration with users in it + // starts closed — so set it rather than depend on which case this one is. + `INSERT INTO sync_state (key, value, updated_at) VALUES ('timeline_enabled', '1', ${nowSeconds}) ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`, ]; items.forEach((item, index) => { diff --git a/scripts/dev-local.sh b/scripts/dev-local.sh index dff0e774..67e45c4f 100755 --- a/scripts/dev-local.sh +++ b/scripts/dev-local.sh @@ -60,6 +60,15 @@ if ! echo "y" | npx wrangler d1 migrations apply skyreader --local; then fi echo -e "${GREEN}Migrations applied.${NC}\n" +# Open the timeline rollout gate locally. Migration 0071 gates any database that +# already has users, which is the right default for prod and staging but wrong +# here: a local DB you've logged into before would silently serve the legacy +# batch path, so you'd develop against the path we're retiring without noticing. +npx wrangler d1 execute skyreader --local --command \ + "INSERT INTO sync_state (key, value, updated_at) VALUES ('timeline_enabled', '1', unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = '1', updated_at = unixepoch()" >/dev/null 2>&1 \ + || echo -e "${YELLOW}Could not open the timeline gate; the reader will use the legacy batch path.${NC}" + # Start feed proxy (crawler + ingest pusher pointed at the local Worker) echo -e "${YELLOW}[1/4] Starting feed proxy...${NC}" cd "$FEED_PROXY_DIR"