Serve the feed timeline from D1; Fly proxy becomes an ingest-push crawler - #348
Merged
Conversation
…wler A refresh was `1 + ceil((N-8)/50)` sequential batch calls, each hopping Worker -> Fly and then paying ~18 chunked D1 queries to annotate read state. Now the proxy pushes new/edited items into a central D1 archive and a refresh is one `GET /api/v2/timeline` — a single query that joins subscriptions and read state, so `getReadKeys` leaves the feed path entirely and reads never touch Fly. Backend: migration 0061 drops the dormant pre-Fly trio and creates `feeds` + `feed_items` (monotonic seq, unique (feed_url, guid), NOT NULL content_hash) plus the items_generation token; `routes/ingest.ts` adds the secret-authenticated, fail-closed `POST /api/internal/ingest` and `GET /api/internal/crawl-set` with an idempotent upsert, an 8 KB stored content cap and the 5,000-item per-feed sanity cap (the only pruning — D1 is an archive); `routes/timeline.ts` serves incremental drains and per-feed cold starts; `/api/v2/feeds/fetch` is re-backed with D1 plus a pull-through for a feed nobody has crawled yet. Proxy: the durable log is the outbox. `push_state` marks what reached D1, dirty rows drain in seq order every 15 s with capped backoff, the crawl set is pulled back every 5 min to keep feeds warm now that reads no longer stamp them, and push_state cascades on every feed_items delete. Everything is gated on INGEST_URL, so an environment without it crawls exactly as before. Frontend: one drain loop against a single global cursor in Dexie metadata, with the legacy batch path kept as a fallback for an old backend, a rollback, or an environment whose crawler isn't pushing yet. Also: a staging Fly proxy (fly.staging.toml + CI job + staging FEED_PROXY_URL) so the two environments stop sharing one machine, and admin feed health re-pointed at the archive with storage/churn alerts. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
…lout gate, and paged cold starts Ten findings from the review of the D1 timeline change, most consequential first. The subscribe-time pull-through ingested a proxy feed FORWARD. A proxy feed is newest-first and seq is assigned in insert order, so the newest item got the lowest seq — and since a re-push of an unchanged item is a no-op, that inversion never healed: every later per-feed cold start served the feed's oldest entries, and the sanity-cap trim would have deleted its newest. `ingestProxyFeed` now walks the array backwards, as the proxy's own `writeFeedItems` does. The rollout gate was inferred from an empty archive, which one pull-through write falsified: a client could commit a cursor against a D1 nothing was crawling and stop refreshing 40 feeds in silence. Both internal endpoints now stamp `sync_state.crawler_heartbeat_at` (the crawl-set pull runs every 5 minutes regardless of feed activity), the timeline reports `ingestActive` from it, and the client stays on the batch path until the crawler is demonstrably alive. Cold starts were bounded only by 500 feeds x 30 items — up to 15,000 rows buffered in one Worker response, for everyone at once after a generation bump. They now page: feeds in a stable order, an item budget per page, continuation via `cold_offset`/`nextColdOffset`. The client keeps the FIRST page's cursor (the head read before the slices, so concurrent ingest can't be skipped) and commits it only once the last page merges. Also: - A cursor above the archive head now cold-starts that client, the D1 twin of the proxy's snapshot-restore guard — a Time Travel restore no longer stalls every client forever pending a manual `UPDATE sync_state`. - `contentTruncated` is no longer inert: it rides into the Article row and ArticleCard extracts the original when such an article is expanded, so a full-text feed doesn't quietly degrade to a two-sentence RSS summary. - Subscriptions that arrive from another device sit below the global cursor, so each is backfilled once through the per-feed endpoint (<= 10 per sync, tracked in Dexie). - New-subscription backfill no longer forces a crawl per feed and is paced (3 at a time, 1 s apart); subscribe time crawls straight into the archive (`warmFeedIntoArchive` replaces the old warm-and-discard), and the endpoint gets its own light rate limit. A 250-feed OPML import stops 429ing halfway. - The pull-through requires the caller to subscribe to the feed, so the shared never-pruned archive isn't an open write surface. - Staging keeps pointing at the prod proxy until Phase 3 provisions its own Fly app, and the staging deploy job skips itself with a notice until then, instead of taking staging's extract/discovery down on merge. - The incremental drain's scan cost is documented where it lives. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
Co-Authored-By: codexbot.disnetdev.com (did:plc:hbonvqr5ysrscg5wdyb5klie) <codexbot.disnetdev.com@noreply.radial>
Co-Authored-By: codexbot.disnetdev.com (did:plc:hbonvqr5ysrscg5wdyb5klie) <codexbot.disnetdev.com@noreply.radial>
Co-Authored-By: codexbot.disnetdev.com (did:plc:hbonvqr5ysrscg5wdyb5klie) <codexbot.disnetdev.com@noreply.radial>
The merge of main into the D1 timeline branch left main's hourly metrics writer querying `feed_metadata`, which 0061_feed_timeline drops. Backend CI was red on it and, once the migration reached prod, `metrics_snapshots` would have stopped gaining rows entirely — every admin trend sparkline frozen and an hourly Sentry issue, with the cron still reporting healthy. - ops-metrics: count `feeds` instead of the dropped `feed_metadata`, and source `feeds_with_errors` from the proxy's `feedsInError` (already in the `proxy_stats` row the cron writes every 5 minutes). Migration 0069 drops the column's NOT NULL so an unreachable proxy records null rather than drawing a healthy zero through its own outage. - admin: `feeds_with_errors` is nullable, and the trend is labelled "Proxy Feeds in Error" to match the tile it now shares a source with. - backend/CLAUDE.md: restore the Observability section, the health and telemetry route rows, the `system_status`/`metrics_snapshots` tables and the secrets block, all dropped when the conflict took the branch side. - ARCHITECTURE.md: same class of staleness — it still listed `feed_metadata`/`feed_cache` as live tables and described the proxy as the read path. - Renumber 0061_feed_timeline -> 0068 so it no longer ties with main's 0061_linkblog_publication (wrangler orders by the numeric prefix, so a tie falls back to directory order on a fresh apply). - RUNBOOK: a §4d on ingest health (crawler heartbeat, outbox depth, what `last_ingest_at` does and doesn't mean), the feed-freshness SLO renamed to crawl freshness with the second hop stated as unmeasured, and the log query example moved off the legacy batch route. Backend: 539 tests pass, tsc and prettier clean. Admin: check and tests pass. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
Timeline reads are served from the D1 archive and never touch the crawler,
so a feed that fails to crawl simply goes quiet — indistinguishable from one
that hasn't published. feedFetcher only ever called markReady for feeds that
delivered items, so after the cutover the sidebar badge and Manage Sources
popover went blank and stale errors from the batch era were never cleared.
The crawler now reports the feeds it is having trouble with every 5 minutes
(POST /api/internal/feed-health, right after the crawl-set pull so a feed
registered this cycle is already in the set). Two distinct faults ride the
same report:
- error_count: the fetch fails. Readers see this, via feedHealth on the
timeline response and health on GET /api/v2/feeds/fetch.
- crawl_stale: the feed is in the crawl set but unfetched for 2h — starved
by a saturated warm loop rather than failing. Operator signal only, and
kept out of feed_health_rev so capacity churn can't make every client
re-download the reader payload.
The payload is the COMPLETE trouble set, not a delta: recovery is inferred
from a feed's absence, so nothing has to announce that it started working.
The recovery sweep is a set difference against the currently flagged feeds
rather than a NOT IN list of every healthy feed, which at ~1,300 feeds is the
bound-parameter wall /batch already hit once. A feed-health rev token gates
the reader payload so a steady-state poll stays at one query; a cold start
always gets it, since it replays archived items from feeds that may have
broken since and its markReady pass would otherwise clear a live error.
Admin: health is now the crawler's verdict instead of an inference from
last_ingest_at, which only moves when a fetch yields a NEW item and so
flagged every low-frequency feed as broken. "Subscribed Feeds Not Ingesting"
(permanently warning, meaningless) becomes "Subscribed Feeds Erroring" and
"Subscribed Feeds Not Being Crawled". The Feeds page filters on
All/Erroring/Not Crawled/OK and shows the crawler's actual message, failure
count, retry time and last good fetch; last_ingest_at stays, relabelled
"Last Item", as the publishing cadence it actually measures. The proxy-wide
"Proxy Feeds in Error" tile and its trend series are untouched — different
population, and the series is documented as matching its tile.
Also fixes a pre-existing unit bug: updateFromV2Result treated nextRetryAt as
seconds and rescaled it, but the crawler sends Date.now() + backoff in ms.
That put every retry ~50,000 years out, so canFetch retired a feed
permanently after one transient error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A fresh crawler heartbeat only says a crawler is attached. The proxy pulls the crawl set immediately at boot, so the first stamp lands seconds into a backfill that takes hours -- which meant `ingestActive` switched every reader onto the timeline at the moment the archive was emptiest, and each of them would then drag the entire backfill through the incremental drain: the fan-out-on-read scan at its worst case, for the duration, surfacing back-catalogue items as unread along the way. There was also no way back short of a Worker rollback or stopping the crawler for 30 minutes. `sync_state.timeline_enabled` (migration 0071) separates the two. `ingestActive` is now the AND of a fresh heartbeat and this flag, so ingest and admission can be sequenced: enable INGEST_URL, let the archive fill, watch ingest.pending trend to ~0, then open the gate with one UPDATE. Setting it back to '0' returns every client to the legacy batch path at its next poll, with no deploy. The wire contract is unchanged -- `ingestActive: false` has always meant "stay on /batch" -- so no frontend change. A gated request short-circuits rather than building a page the client is about to discard, which is what keeps the gated window (most of the rollout) at one sync_state read per poll. Only an explicit '0' gates; an absent row is open, so a hand-built schema or a future environment is never silently held back. The migration writes '0' for a database that already has users, so prod and staging start shut while local dev, e2e and CI start open. dev-local.sh and the e2e seed set it anyway, since a local database you have logged into before would otherwise quietly serve the path we are retiring. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the D1-served feed timeline and Fly ingest-push crawler architecture, rebased onto current observability and standard.site teardown work.
Delivered
GET /api/v2/timelinewith read state joined in D1; Fly leaves the read path and pushes crawl deltas through authenticated internal endpoints.Review fixes (latest commit)
writeMetricsSnapshotno longer queries the droppedfeed_metadata. It countsfeeds, and takesfeeds_with_errorsfrom the proxy'sfeedsInErrorin theproxy_statsrow the cron already writes. Migration0069drops that column'sNOT NULLso a stale/unreachable proxy records null instead of a healthy-looking zero; the admin type and trend label follow.backend/CLAUDE.md(the section, the health/telemetry route rows,system_status/metrics_snapshots, and the secrets block) on top of the branch's feed-architecture rewrite, plus the same class of staleness inARCHITECTURE.md.0061_feed_timeline.sql→0068_feed_timeline.sqlso it no longer ties with main's0061_linkblog_publication.sql.last_ingest_atactually means), the feed-freshness SLO renamed to crawl freshness with the unmeasured second hop stated plainly, and the log-query example moved off the legacy batch route.Checks
tsc --noEmitandprettier --check: pass.npm run checkandnpm test: pass (23 tests).Radial artifact