Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 47 additions & 188 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 6 additions & 4 deletions packages/discovery-index/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,14 @@ ENV PORT=8080
EXPOSE 8080
# Provide at runtime (NOT baked into the image): DISCOVERY_INDEX_SHARED_SECRET (bearer auth for callers),
# DISCOVERY_INDEX_GITHUB_TOKEN (this service's own GitHub token, isolated from any other component's),
# SENTRY_DSN/SENTRY_ENVIRONMENT/SENTRY_RELEASE (error tracking, #4934 -- optional, no-op when SENTRY_DSN is
# unset), and for source-map upload specifically: SENTRY_AUTH_TOKEN, SENTRY_ORG, SENTRY_PROJECT (also
# optional -- upload-sourcemaps.ts skips itself with a log line, never fails the boot, when any is unset).
# POSTHOG_API_KEY/POSTHOG_HOST/POSTHOG_ENVIRONMENT/POSTHOG_RELEASE (error tracking, #8289 -- optional, no-op
# when POSTHOG_API_KEY is unset), and for source-map upload specifically: POSTHOG_CLI_API_KEY,
# POSTHOG_CLI_PROJECT_ID, POSTHOG_CLI_HOST (also optional -- upload-sourcemaps.ts skips itself with a log
# line, never fails the boot, when POSTHOG_CLI_API_KEY/PROJECT_ID are unset). All PostHog vars are pending
# #7875 (a real, loopover-owned PostHog project), so this is currently inert in every real deployment.
#
# upload-sourcemaps.js runs FIRST, at container startup (not baked into the image at build time, since
# SENTRY_AUTH_TOKEN is a runtime-only secret) -- deletes the .map files it just uploaded before the server
# POSTHOG_CLI_API_KEY is a runtime-only secret) -- deletes the .map files it just uploaded before the server
# starts, so they're never served or left on disk in the running container. Mirrors
# review-enrichment/Dockerfile's identical CMD shape.
CMD ["sh", "-c", "node dist/upload-sourcemaps.js && find dist -type f -name '*.map' -delete && node dist/server.js"]
2 changes: 1 addition & 1 deletion packages/discovery-index/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,4 @@ All opted-in miners authenticate with **one shared `DISCOVERY_INDEX_SHARED_SECRE

`/health` (liveness) and `/ready` (readiness — reports whether `DISCOVERY_INDEX_GITHUB_TOKEN` is configured) are the two routes an external uptime check should poll; both are exempt from the rate limit. `/metrics` exposes Prometheus-format counters/histograms for request outcomes and latency. This service is **not** wired into the self-host fleet's own Grafana/Alloy stack — it isn't a self-hosted instance, it's the one hosted plane, deployed and observed separately (Cloudflare's own Worker observability, per `wrangler.jsonc`) rather than folded into infrastructure that assumes a self-hoster's own box.

**Error tracking (#4934):** route/process-level errors report into the shared `metagraphed` Sentry project (`jsonbored` org), the same project other infra/operational pieces already flow into — check it first when triaging a spike in `/metrics`' error counters, before digging into Cloudflare's own Worker logs. See `src/sentry.ts` for what gets redacted before an event ever leaves the process, and `README.md`'s "Error tracking & releases" section for the source-map/release-upload path that makes those Sentry stack traces point at real TypeScript instead of a minified bundle.
**Error tracking (#8289):** route/process-level errors report into PostHog — check it first when triaging a spike in `/metrics`' error counters, before digging into Cloudflare's own Worker logs. See `src/posthog.ts` for what gets redacted before an event ever leaves the process, and `README.md`'s "Error tracking & releases" section for the source-map/release-upload path that makes those stack traces point at real TypeScript instead of a minified bundle. Previously reported into the shared `metagraphed` Sentry project (`jsonbored` org) other infra/operational pieces flow into, until #8289 replaced it outright.
13 changes: 5 additions & 8 deletions packages/discovery-index/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,14 @@ Soft-claim design note: the shipped client payload never carries caller identity
| `DISCOVERY_INDEX_CACHE_TTL_MS` | TTL for cached query results, per unique `(repos, orgs, searchTerms)` scope. Default `300000` (5 minutes). |
| `DISCOVERY_INDEX_SOFT_CLAIM_TTL_MS` | TTL for a soft claim before it's reclaimable. Default `1800000` (30 minutes). |
| `PORT` | HTTP port. Default `8080`. |
| `SENTRY_DSN` | Error tracking (#4934), reporting into the shared `metagraphed` Sentry project. Unset ⇒ Sentry stays inert (no-op) everywhere, matching this repo's other services. Non-sensitive (write-only ingest URL) — set as a plain `wrangler.jsonc` var, not a secret. |
| `SENTRY_ENVIRONMENT` | Sentry environment tag. Default `production`. |
| `SENTRY_RELEASE` | This deploy's release identifier (`loopover-discovery-index@<sha>`). Set per-deploy via `wrangler deploy --var`, not a static config value — see `wrangler.jsonc`'s header comment. Falls back to deriving one from `SENTRY_COMMIT_SHA` if unset. |
| `SENTRY_AUTH_TOKEN` | Full Sentry API token, used only by `upload-sourcemaps.ts` at container startup to upload this build's source maps and manage the release. Genuinely sensitive — always a `wrangler secret`. Unset ⇒ source-map upload skips itself with a log line; never fails the boot. |
| `SENTRY_ORG` / `SENTRY_PROJECT` | Sentry org/project slugs for source-map upload. Same non-sensitive/plain-var treatment as `SENTRY_DSN`. |
| `POSTHOG_API_KEY` / `POSTHOG_HOST` / `POSTHOG_ENVIRONMENT` / `POSTHOG_RELEASE` | PostHog error tracking (#8289). Unset ⇒ PostHog stays inert (no-op) everywhere, matching this repo's other services. `POSTHOG_API_KEY` is a write-only project token, safe as a plain `wrangler.jsonc` var once set; `POSTHOG_RELEASE` is set per-deploy, not static — see `wrangler.jsonc`'s header comment. **Pending #7875** (a real, loopover-owned PostHog project) — currently unset everywhere, so this stays a complete no-op. |
| `POSTHOG_CLI_API_KEY` / `POSTHOG_CLI_PROJECT_ID` / `POSTHOG_CLI_HOST` | PostHog **personal** API key (error-tracking write + organization read scopes, distinct from `POSTHOG_API_KEY`'s project token) + numeric project id, used only by `upload-sourcemaps.ts` at container startup to upload this build's source maps. `POSTHOG_CLI_API_KEY` is genuinely sensitive — always a `wrangler secret`. Unset ⇒ source-map upload skips itself with a log line; never fails the boot. **Pending #7875**, same as above. |

## Error tracking & releases (#4934)
## Error tracking & releases (#4934, #8289)

Runtime errors (route handler failures, unhandled rejections/exceptions) report into the shared **`metagraphed`** Sentry project (`jsonbored` org) that this repo's other infra/operational pieces already flow into — not a separate project. See `src/sentry.ts` for the full redaction rules (secret-named fields and GitHub-token-shaped values are always filtered before anything leaves the process) and `OPERATIONS.md`'s Monitoring section for how this fits alongside `/metrics`.
Runtime errors (route handler failures, unhandled rejections/exceptions) report into PostHog. See `src/posthog.ts` for the full redaction rules (secret-named fields and GitHub-token-shaped values are always filtered before anything leaves the process) and `OPERATIONS.md`'s Monitoring section for how this fits alongside `/metrics`. This surface previously reported into the shared `metagraphed` Sentry project (`jsonbored` org) that this repo's other infra/operational pieces flow into -- a genuine cross-project entanglement -- until #8289 replaced it outright.

Source maps and release/deploy tracking are handled by `src/upload-sourcemaps.ts`, which runs as the **first step of the container's runtime `CMD`** (not at Docker build time — `SENTRY_AUTH_TOKEN` is a runtime-only secret, never baked into an image layer): it creates the Sentry release, associates commits, uploads and validates source maps, then records a deploy and finalizes the release, before deleting the `.map` files it just uploaded and starting `server.ts`. Any step failing is non-fatal to the boot unless `DISCOVERY_INDEX_SENTRY_UPLOAD_STRICT=true` is set. Mirrors `review-enrichment/src/upload-sourcemaps.ts`'s identical design.
Source maps and release tracking are handled by `src/upload-sourcemaps.ts`, which runs as the **first step of the container's runtime `CMD`** (not at Docker build time — `POSTHOG_CLI_API_KEY` is a runtime-only secret, never baked into an image layer): it injects PostHog chunk ids, uploads the exact post-injection source maps, then verifies at least one symbol set exists for the release via `scripts/validate-posthog-release.mjs` (queries PostHog's `error_tracking/symbol_sets` API — PostHog's release model has no equivalent of Sentry's commits/deploys/finalize lifecycle, so this is a narrower check by design), before deleting the `.map` files it just uploaded and starting `server.ts`. Any step failing is non-fatal to the boot unless `DISCOVERY_INDEX_POSTHOG_UPLOAD_STRICT=true` is set. `posthog-cli sourcemap inject`/`upload` run with an explicit `--release-version` (not the CLI's own git-metadata auto-detection — the Docker build stage never copies `.git` into the image).

## Deployment

Expand Down
6 changes: 3 additions & 3 deletions packages/discovery-index/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@
"@cloudflare/containers": "^0.3.7",
"@hono/node-server": "^2.0.11",
"@loopover/engine": "^3.4.0",
"@sentry/cli": "^3.6.0",
"@sentry/node": "^10.63.0",
"hono": "^4.12.27"
"@posthog/cli": "0.9.1",
"hono": "^4.12.27",
"posthog-node": "^5.44.0"
},
"devDependencies": {
"@cloudflare/workers-types": "^5.20260721.1",
Expand Down
25 changes: 25 additions & 0 deletions packages/discovery-index/scripts/validate-posthog-release.d.mts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
export type PostHogReleaseValidationConfig = {
apiKey: string | undefined;
projectId: string | undefined;
release: string | undefined;
baseUrl: string;
};

export type PostHogReleaseValidationResult = {
release: string | undefined;
symbolSetCount: number;
};

export declare class PostHogReleaseValidationError extends Error {
failures: string[];
constructor(message: string, failures?: string[]);
}

export declare function loadPostHogReleaseValidationConfig(
env?: Record<string, string | undefined>,
): PostHogReleaseValidationConfig;

export declare function validatePostHogRelease(
env?: Record<string, string | undefined>,
fetchImpl?: typeof fetch,
): Promise<PostHogReleaseValidationResult>;
Loading
Loading