fix(webhooks): requeue deliveries dropped by retryable setup infrastructure failures

[icecrasher321]

Summary

• Webhook deliveries were permanently dropped when setup hit a transient infrastructure failure (e.g. a database connection timeout in the worker): the task runs with maxAttempts: 1, ingress has already ACKed the provider, and the idempotency key was memoized as failed for the 7-day dedupe window — so even provider redeliveries of the same event were rejected
• Requeue the same delivery (same execution id, bounded jittered backoff) when setup fails on retryable infrastructure before any block has run — mirrors the schedule executor's retryable_setup_failure handling, including the workflowCoreStarted boundary so errors after the executor starts are never reclassified as retryable (no double-run risk); retries are capped by WEBHOOK_INFRA_RETRY_MAX_ATTEMPTS (default 5, 0 disables) and an exhausted budget still faults the run loudly
• New RetryableSetupError contract in lib/core/errors/retryable-infrastructure: it certifies the operation performed no effect, so executeWithIdempotency releases the claim instead of memoizing the failure — the requeued run (or a later provider redelivery) can claim and execute
• Classify postgres.js client connection codes (CONNECT_TIMEOUT, CONNECTION_CLOSED, CONNECTION_ENDED, CONNECTION_DESTROYED) as retryable infrastructure — they previously fell outside the Node syscall / undici / Postgres server-code sets, which also weakened the schedule path's existing infra retry
• Preprocessing gains suppressRetryableFailureLogs so an attempt that will be retried does not leave a terminal failed execution-log row (the retry reuses the execution id, which is unique per log row); the final attempt still records it
• Trigger.dev applies the requeue delay server-side; the database backend executes jobs only through an in-process runner, so the runner sleeps out the backoff itself

Type of Change

• Bug fix

Testing

• New coverage across the touched suites: requeue on retryable preprocessing failures and setup reads, retry-budget exhaustion, non-retryable passthrough, post-core errors never reclassified, enqueue-failure fallback to loud failure, idempotency claim release vs memoization, classifier codes, log-row suppression against the real preprocessing module
• tsc --noEmit, bun run lint, bun run check:audits (33 audits), and bun run check:api-validation pass

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview	Aug 24, 2026 6:26pm

[cursor]

PR Summary

Medium Risk
Touches webhook admission, idempotency, and execution logging on failure paths; incorrect retry boundaries could double-run workflows or leave silent drops, though the workflowCoreStarted guard and tests target that.

Overview
Fixes webhook deliveries being permanently dropped when setup hits transient infra (DB timeouts, etc.): ingress already ACKed the provider, the worker runs with maxAttempts: 1, and idempotency could memoize a failed outcome for days.

Webhook execution now re-enqueues the same delivery (same executionId / request / idempotency inputs) with jittered backoff when setup fails on retryable infrastructure before executeWorkflowCore runs. A workflowCoreStarted guard ensures post-core errors are never reclassified (no double-run). Retries are capped by new env knobs (WEBHOOK_INFRA_RETRY_*, default 5 attempts; 0 disables). Successful requeue returns { requeued: true }; exhausted budget or failed enqueue still faults loudly, with a best-effort terminal log row if requeue enqueue fails.

Introduces RetryableSetupError and extends retryable infra classification (postgres.js CONNECT_TIMEOUT, CONNECTION_CLOSED, etc.). IdempotencyService releases the claim on that error instead of caching failure. preprocessExecution adds suppressRetryableFailureLogs so retry-bound attempts don’t write terminal failed execution rows. interruptibleSleep is shared in @sim/utils/helpers for DB-job runner backoff and hosted-key waits.

^{Reviewed by Cursor Bugbot for commit 95e763e. Configure here.}

[greptile-apps]

Greptile Summary

The PR requeues webhook deliveries after effect-free, retryable setup failures while preserving execution identity and releasing the idempotency claim.

• Adds bounded, jittered infrastructure retries with backend-aware delay handling.
• Suppresses retry-bound terminal log rows and restores one when replacement enqueueing fails.
• Extends infrastructure classification for postgres.js connection errors.
• Extracts an abort-aware shared sleep helper.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/sim/background/webhook-execution.ts	Adds bounded setup-failure requeueing, effect-boundary classification, cancellation-aware database backoff, and enqueue-failure logging restoration.
apps/sim/lib/core/idempotency/service.ts	Releases owned idempotency claims for typed effect-free setup failures rather than memoizing them.
apps/sim/lib/execution/preprocessing.ts	Marks retryable infrastructure failures and conditionally defers terminal error logging for attempts that will be retried.
apps/sim/lib/core/errors/retryable-infrastructure.ts	Adds postgres.js connection codes and the typed RetryableSetupError contract.
packages/utils/src/helpers.ts	Adds a shared interruptible sleep that cleans up its abort listener and timer.

Sequence Diagram

sequenceDiagram
  participant Provider
  participant WebhookJob
  participant Setup
  participant Idempotency
  participant Queue
  Provider->>WebhookJob: Delivery already acknowledged
  WebhookJob->>Idempotency: Claim delivery
  WebhookJob->>Setup: Preprocess and load state
  alt Retryable failure before workflow core
    Setup-->>WebhookJob: RetryableSetupError
    WebhookJob->>Idempotency: Release claim
    WebhookJob->>Queue: Enqueue same execution with backoff
    Queue->>WebhookJob: Retry attempt
  else Workflow core starts
    WebhookJob->>WebhookJob: Execute workflow once
    WebhookJob->>Idempotency: Memoize outcome
  end

Loading

_{Reviews (2): Last reviewed commit: "fix(webhooks): restore terminal log on f..." | Re-trigger Greptile}

[icecrasher321]

@greptile

[icecrasher321]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 95e763e. Configure here.}