feat(alerts): fan out notifications to every configured channel

[jordan-simonovski]

Alert notifications now go to every configured channel concurrently, each with its own deadline, span and metrics. One slow or dead target can no longer delay the others or the alert evaluation loop.

What changed

Template rendering used to send inline: a Handlebars helper awaited each webhook as it rendered, so sends were serial and a hung endpoint blocked that alert indefinitely. Rendering now collects notification jobs, and dispatchNotifications runs them concurrently after the render.

Each send is wrapped in an alerts.notify CLIENT span, a per-target deadline, and per-target metrics, and never throws — the caller gets one result per target and records failures individually. Alert execution errors now name the webhook that failed, so a multi-channel alert says which target broke.

Each alert evaluation also gets a processAlert span carrying team context.

Key decisions

The deadline stops waiting; it does not cancel. Delivery stays at-least-once, so an abandoned send is allowed to finish. Its eventual rejection is swallowed so it cannot surface as an unhandled rejection and kill the task process.

A timed-out HTTP attempt is not retried. withRetry treats only 3xx and 4xx as terminal, and an abort surfaces as DOMException code 23 — so without this, a receiver that was merely slow would get three duplicate POSTs where it previously got one delivery. The timeout is surfaced as a 408 so the existing retry policy stops on it.

Per-attempt timeout defaults to 30s, not 10s. The bound exists to release a black-holed socket, not to police slow receivers; 30s leaves headroom inside the 60s deadline while not failing endpoints that succeed today.

renderAlertTemplate returns the rendered body alongside the results. Returning only the results would have made the rendering and template-injection assertions untestable, since those tests configure no webhooks and so produce no transport calls to inspect.

Impact

Behaviour changes for existing single-channel alerts, worth attention on merge:

• Generic webhook attempts are now bounded by ALERT_NOTIFICATION_FETCH_TIMEOUT_MS (default 30s), and Slack sends by the same value. Neither had a per-attempt bound before.
• A missing webhook no longer aborts the whole event; other channels still fire and the failure is recorded against that target.
• Execution error messages now name the failing webhook.

New env vars: ALERT_NOTIFICATION_DEADLINE_MS (default 60s) and ALERT_NOTIFICATION_FETCH_TIMEOUT_MS (default 30s). Both fall back to the default when unset or malformed.

Implementation detail

MAX_NOTIFICATIONS_PER_EVENT (20) caps jobs per fire/resolve event, covering configured channels and @webhook- message mentions together. A channel dropped by the cap records an execution error rather than only a log line and a metric, so a partially-notified alert does not look healthy.

New metrics: hyperdx.alerts.notifications (attrs channel_type, service, outcome) and hyperdx.alerts.notification.duration_ms. The existing hyperdx.alerts.webhook_deliveries transport metrics are unchanged.

Tests cover failure isolation, deadline timeout for both generic and Slack targets, the abort-not-retried path, the malformed-env fallback, the per-event cap, and a pre-multi-channel document that only has channel.

Verification: 273 checkAlerts integration tests, plus the notification unit suite.

[changeset-bot]

🦋 Changeset detected

Latest commit: cf3d49d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/api	Minor
@hyperdx/app	Minor
@hyperdx/otel-collector	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview	Aug 10, 2026 1:49am
hyperdx-storybook	Ready	Preview	Aug 10, 2026 1:49am

[greptile-apps]

Greptile Summary

Alert rendering now collects notification jobs and dispatches each configured target concurrently with isolated deadlines, telemetry, and per-target errors. The latest changes also deduplicate resolved webhook IDs before cap enforcement and split the multi-channel integration suite to comply with the repository’s file-size rule.

• Fans alert notifications out across all configured channels.
• Adds per-target deadlines, HTTP attempt timeouts, tracing, and outcome metrics.
• Preserves legacy single-channel alerts and reports individual delivery failures.
• Fixes duplicate webhook delivery and false cap errors through canonical-ID deduplication.
• Splits the multi-channel integration test into a focused specification and shared harness.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/api/src/tasks/checkAlerts/template.ts	Collects notification jobs and deduplicates resolved webhook IDs before applying the per-event cap, fixing both prior duplicate-related findings.
packages/api/src/tasks/checkAlerts/notifications.ts	Adds concurrent per-target dispatch, bounded HTTP attempts, deadlines, tracing, metrics, and isolated result reporting.
packages/api/src/tasks/checkAlerts/index.ts	Processes per-target notification results, records attributable execution errors, and adds team-scoped alert-processing spans.
packages/api/src/tasks/checkAlerts/tests/multiChannelAlerts.int.test.ts	Provides focused integration scenarios in a 215-line file, with shared setup extracted into a separate 148-line harness.
packages/api/src/tasks/checkAlerts/tests/renderAlertTemplate.int.test.ts	Covers canonical target deduplication, duplicate handling at the cap boundary, and reporting of genuinely dropped targets.

Sequence Diagram

sequenceDiagram
  participant Eval as processAlert
  participant Render as renderAlertTemplate
  participant Queue as Notification jobs
  participant Targets as Configured webhooks
  Eval->>Render: Render alert and resolve channels
  Render->>Render: Deduplicate canonical webhook IDs
  Render->>Queue: Collect jobs up to event cap
  Queue->>Targets: Dispatch concurrently
  Targets-->>Queue: Per-target success, error, or timeout
  Queue-->>Eval: Notification results
  Eval->>Eval: Record individual execution errors

Loading

_{Reviews (3): Last reviewed commit: "fix(alerts): dedupe notification targets..." | Re-trigger Greptile}

[github-actions]

Deep Review

✅ No critical issues found. No P0/P1 issues surfaced. The refactor is sound: all callers of the changed signatures (renderAlertTemplate → { body, results } + required teamId, getDefaultExternalAction → getDefaultExternalActions, unexported notifyChannel) are updated in-diff; the SSRF redirect guard and validateWebhookUrl are preserved; AbortSignal.any/timeout are supported on the repo's Node >=22.16.0; the abort→408→terminal mapping correctly stops withRetry from re-sending; failure isolation via Promise.all cannot let one dead target reject the batch; and legacy single-channel documents still resolve via getAlertChannels. New metrics/spans comply with the documented low-cardinality rules.

🟡 P2 -- recommended

• packages/api/src/tasks/checkAlerts/template.ts:472 -- The per-event cap check jobs.length + preFailedResults.length >= MAX_NOTIFICATIONS_PER_EVENT counts non-target pre-failures (unsupported @mentions, not-found webhooks) toward the cap, and because default channels are appended after the user template these are processed first, so a message with enough bad mentions/deleted-webhook references can push real configured channels over the cap and record a spurious NotificationCapExceededError against a channel that was never notified.
  • Fix: Gate the cap on queued job count (and genuine capped-out real targets) only, so pre-dispatch failures do not consume the notification budget.
  • _{correctness, testing}
• packages/api/src/tasks/checkAlerts/__tests__/multiChannelAlerts.int.test.ts:57 -- Grouped alerts (groupBy) combined with multiple channels have no coverage; every multi-channel end-to-end test uses non-grouped alerts, so the eventId group branch interacting with the new per-render dedupe and fan-out is unverified.
  • Fix: Add a grouped alert with 2+ channels and 2+ groups asserting each group fans out to every channel once with distinct eventIds.
  • _testing

🔵 P3 nitpicks (7)

• packages/api/src/utils/slack.ts:8 -- getTimeoutMs re-implements notifications.ts positiveMsEnv/getWebhookFetchTimeoutMs for the same ALERT_NOTIFICATION_FETCH_TIMEOUT_MS var and 30_000 default; the two copies are documented as needing to match but nothing enforces it, so a future change to one silently diverges Slack vs generic-webhook attempt timeouts.
  • Fix: Export one shared timeout parser and consume it from both modules.
  • _{maintainability, api-contract, kieran-typescript}
• packages/api/src/tasks/checkAlerts/template.ts:484 -- In the cap-exceeded branch getPopulatedChannel has already succeeded, but recordPreFailure stores the raw id/name-prefix as both webhookId and webhookName, so the resulting error names the webhook by its raw identifier instead of its real name, unlike every dispatched-target branch.
  • Fix: Pass the resolved channel.channel.name/_id when recording a cap pre-failure, and stop overloading webhookId === webhookName.
  • _{kieran-typescript, maintainability}
• packages/api/src/tasks/checkAlerts/template.ts:433 -- A plain Slack mention such as @here/@channel in an alert message is rewritten into the notify helper, fails safeParse, and is recorded as a WEBHOOK_ERROR execution error, so a healthy alert whose message contains an ordinary mention shows a spurious error on every fire (net improvement over the prior render-aborting behavior, but still surfaced as an error).
  • Fix: Treat unrecognized @word tokens whose channel is not a known channel type as inert text rather than recording an execution error.
  • _correctness
• packages/api/src/tasks/checkAlerts/notifications.ts:339 -- ALERT_NOTIFICATION_DEADLINE_MS and ALERT_NOTIFICATION_FETCH_TIMEOUT_MS are read directly from process.env in notifications.ts and slack.ts rather than centralized in config.ts where other tunables live, and are documented only in the changeset.
  • Fix: Surface both vars (and their defaults) in config.ts and any env example, and import them from there.
  • _{api-contract, project-standards}
• packages/api/src/tasks/checkAlerts/index.ts:243 -- makeNotificationAlertError repeats the .slice(0, 10000) magic-number truncation across every branch and rebuilds the webhook "…" suffix by hand in each arm, duplicating the redirect branch already in makeWebhookAlertError.
  • Fix: Extract a named truncation constant/helper applied once at the return and build the branches as a predicate→message table.
  • _{maintainability}
• packages/api/src/tasks/checkAlerts/notifications.ts:179 -- WebhookAttemptTimeoutError (per-HTTP-attempt, status 408) and NotificationTimeoutError (per-notification deadline) have near-identical names for two distinct layers, forcing the reader to inspect construction sites to tell them apart.
  • Fix: Rename one to reflect its layer (e.g. NotificationDeadlineExceededError) or add a contrasting doc comment.
  • _{maintainability}
• packages/api/src/tasks/checkAlerts/__tests__/renderAlertTemplate.int.test.ts:574 -- Cap/mention tests assert via String(error).toContain(...), coupling to exact message wording and stringification rather than the error type or outcome, so a message reword silently changes the contract without a precise failure.
  • Fix: Assert toBeInstanceOf(NotificationCapExceededError) / UnsupportedMentionError and the outcome instead of stringifying.
  • _testing

---

Reviewers (9): correctness, security, testing, maintainability, project-standards, api-contract, kieran-typescript, agent-native, learnings-researcher. (reliability and adversarial were also dispatched but had not returned at synthesis time; their focus areas — deadline/retry interaction and cap-accounting abuse — are covered by the correctness and testing findings above.)

Testing gaps:

• No test exercises cap starvation where pre-failed targets (bad @mentions/missing webhooks) consume the cap and drop a healthy channel (the behavior behind the P2 above).
• The Slack per-attempt timeout wiring (IncomingWebhook({ timeout }) in slack.ts) is mocked out in every test, so it is never actually exercised; Slack sends receive no AbortSignal, so a hung Slack POST is bounded only by that untested per-attempt timeout and can land after the dispatcher deadline (at-least-once by design).
• makeNotificationAlertError's UnsupportedMention and NotificationCapExceeded branches are not covered end-to-end through processAlert into stored executionErrors.
• Time-based unit tests in notifications.test.ts use real timers with tight windows (50ms deadline, 80ms late rejection, 120ms drain), which risks nondeterminism on loaded CI runners; consider fake timers.

[github-actions]

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

• Background tasks or delivery pipeline substantially modified — 603 lines (bar: 30):
  • packages/api/src/tasks/checkAlerts/errors.ts
  • packages/api/src/tasks/checkAlerts/index.ts
  • packages/api/src/tasks/checkAlerts/notifications.ts
  • packages/api/src/tasks/checkAlerts/template.ts

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

• Production files changed: 5
• Production lines changed: 613 (+ 818 in test files, excluded from tier calculation)
• Critical-path lines changed: 603
• Branch: jordansimonovski/alerts-multi-channel-dispatch
• Author: jordan-simonovski

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[github-actions]

E2E Test Results

✅ All tests passed • 277 passed • 1 skipped • 876s

Status	Count
✅ Passed	277
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →