fix(knowledge): stop retrying an embedding key with no credit left

[waleedlatif1]

Problem

A connector sync ran the full hour and hit MAX_DURATION_EXCEEDED. The cause is not rate limiting.

OpenAI answers an exhausted balance with 429, the same status as a rate limit:

{ "error": {
    "message": "You have no credits remaining. Add credits to continue using the API at ...",
    "type": "insufficient_quota",
    "code": "credit_balance_exhausted" } }

isTransientEmbeddingError classified every 429 as transient, so these were retried like a rate limit. But a rate limit reopens and a spent account does not — no amount of waiting can make the next attempt succeed.

That alone would be wasted work. What makes it persistent is the interaction with the stuck-document sweep: a failed document is re-queued on every sync, so each one burns its full retry budget again, forever. An account with no credit stops being a one-off failure and becomes permanent load, which is what consumed the hour.

Measured in production before this change:

Documents failed on insufficient_quota	2,286 across 2 knowledge bases
Continuously retrying since	2026-07-27 (three weeks)
Embedding failures from a genuine rate limit	0

Every terminal embedding 429 in production is an exhausted balance. Not one is a rate limit.

Change

The rejection body is what distinguishes them, so it is read when the error is constructed and recorded on the error. Retries against a spent key stop immediately.

Retrying and failing over are decided separately, and that distinction is the point:

• Retry asks whether another attempt against the same provider could work. An exhausted balance rules that out.
• Failover asks whether a different provider could work. An exhausted balance says nothing about the next key in the chain.

So the error stops the retries while remaining eligible for failover, rather than being reclassified as fatal.

Why this matters for what just shipped

The retry budget was widened in the previous PR (5 retries, 30s ceiling). That is correct for a real rate limit, but against a spent account it would have made this worse — 6 attempts per document instead of 4. The regression test reproduces exactly that: it fails at 6 attempts without this change and passes at 1.

Verification

• 632 tests pass; 29/29 audits; type-check clean
• Three tests, each verified to fail without the change: the quota case stops at one attempt; a rate_limit_exceeded body with the same status still retries and recovers; an exhausted-balance error stays eligible for failover

Note

The underlying account still has no credit — this stops us from burning an hour per sync discovering that, and surfaces the provider's own actionable message ("add credits at ...") instead of retrying past it. The sweep re-queueing failed documents with no cooldown remains open and is the next thing worth fixing.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Aug 19, 2026 10:23pm

[cursor]

PR Summary

Medium Risk
Changes embedding retry and knowledge sync failure handling; behavior is narrower and well-tested, but misclassification could still skip needed retries or failover in edge cases.

Overview
OpenAI uses 429 for both rate limits and exhausted credit, but only rate limits recover. The client treated every 429 as transient and kept retrying; with failed docs re-queued on each sync, spent accounts became endless load (observed over weeks on thousands of documents).

The error response body is parsed when building EmbeddingAPIError, and quotaExhausted is set for insufficient_quota / credit_balance_exhausted. isWorthRetrying replaces the inline retry condition so quota failures do not retry on the same key, while isTransientEmbeddingError still allows failover to another provider.

Tests lock in: one attempt on exhausted balance, retry + success on rate_limit_exceeded, and quota errors remaining failover-eligible.

^{Reviewed by Cursor Bugbot for commit d6d63c5. Configure here.}

[greptile-apps]

Greptile Summary

This PR distinguishes exhausted embedding-provider credit from recoverable rate limiting so requests stop retrying the same depleted key while remaining eligible for provider failover.

• Records exhausted-quota classification on EmbeddingAPIError.
• Introduces a retry-specific predicate separate from transient-error failover classification.
• Adds regression coverage for exhausted credit, genuine rate limiting, and failover eligibility.

Confidence Score: 5/5

The PR appears safe to merge with no actionable changed-code failures identified.

The new classifier is attached to the original embedding error, the retry utility evaluates that intact error, and failover continues to classify quota-related 429 responses as transient.

Important Files Changed

Filename	Overview
apps/sim/lib/embeddings/client.ts	Adds body-based exhausted-quota detection and separates same-provider retry eligibility from cross-provider failover eligibility without an identified defect.
apps/sim/lib/embeddings/client.test.ts	Adds focused regression tests proving exhausted quota stops after one request while genuine rate limiting retries and quota errors remain failover-eligible.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Embedding request] --> B{Provider response OK?}
  B -->|Yes| C[Return embeddings]
  B -->|No| D[Read error body]
  D --> E{Exhausted quota?}
  E -->|Yes| F[Stop same-provider retries]
  E -->|No| G{Transient and retryable?}
  G -->|Yes| A
  G -->|No| H[Surface error]
  F --> I{Fallback provider configured?}
  I -->|Yes| J[Try next provider]
  I -->|No| H

Loading

_{Reviews (1): Last reviewed commit: "fix(knowledge): stop retrying an embeddi..." | Re-trigger Greptile}