feat(proxy): opt-in same-target 429 wait-and-retry before key failover (#487) - #865
Conversation
lidge-jun#487) Codex never retries HTTP 429 (openai/codex#30471 keeps retry_429=false and the misleading 'exceeded retry limit' error), and single-key pools have no failover, so provider-level retryOn429 waits (Retry-After or fixed interval) and replays the identical pre-stream request on the same key before any key rotation. - types.ts: RateLimitRetryPolicy + OcxProviderConfig.retryOn429 - config.ts: lenient zod validation (strip unknown; typo degrades, never rejects config) - key-failover.ts: rateLimitRetryPolicyFor + rateLimitRetryDelayMs (Retry-After capped at maxIntervalMs) - responses/core.ts: recovery-loop replay before multi-key failover; abort-aware sleep; covers Responses, chat completions, and routed Claude messages - usage/log.ts: AttemptRecoveryKind 'rate-limit-429' - docs-site configuration reference + structure/04 transport note + devlog plan unit - tests: policy unit tests + e2e (single-key replay, passthrough without knob, exhaustion, retry-before-failover ordering) Verified: typecheck, privacy scan, 43/43 retry-related tests; full suite failures are pre-existing on pristine dev in this environment (WS/auth/connection-refused) plus missing-gui-deps artifacts that pass once installed.
📝 WalkthroughWalkthroughAdds opt-in, same-key HTTP 429 retries for API-key providers. Retries use bounded delays, ChangesRate-limit retry policy and implementation
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant ResponsesCore
participant ProviderTransport
participant UpstreamProvider
Client->>ResponsesCore: Submit request
ResponsesCore->>ProviderTransport: Send with current API key
ProviderTransport->>UpstreamProvider: Forward request
UpstreamProvider-->>ResponsesCore: HTTP 429
ResponsesCore->>ResponsesCore: Release body and wait
ResponsesCore->>ProviderTransport: Replay identical request
ProviderTransport->>UpstreamProvider: Forward on same key
UpstreamProvider-->>ResponsesCore: Response or final 429
ResponsesCore->>Client: Return response or error
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
- usage/log: whitelist the rate-limit-429 recovery kind so persisted usage rows and post-restart /api/logs keep the reason - key-failover: gate retryOn429 to key-auth providers (no same-token OAuth replays, no silent no-op on forward passthrough); accept Retry-After 0 as immediate - config: cap maxIntervalMs at the effective 10-minute cooldown ceiling - core: hoist the retry budget outside the recovery loop so 413/401 replays cannot re-arm it; cancel the unread 429 body before aborting on client cancel - gui: add rate-limit-429 (and pre-existing anthropic-oauth-429) to the Logs union - tests: OAuth/forward gating, HTTP-date Retry-After, Retry-After 0, persisted recovery-kind, and a deterministic direct-handler abort test (real-socket disconnect propagation is async in Bun, so the e2e version was timing-flaky) - docs: key-auth-only note, maxIntervalMs cap, abort-propagation nuance, locale rows
… xhigh) - core: the Responses passthrough wire (openai-responses key-auth gateways, e.g. the built-in DeepSeek preset) now replays 429 on the same key pre-relay, before the forward-pool logic; Anthropic terminal-guard continuations replay before key/account failover - images/loop + web-search/loop: same-target replay before on429 key rotation via a new retryOn429Policy dep (abort-aware, heartbeat seams preserved) - key-failover: the fixed fallback is now capped at maxIntervalMs (a single wait never exceeds the cap); authMode local runtimes are gated out alongside oauth/forward, so the knob matches the documented API-key scope exactly - tests: key-auth openai-responses passthrough e2e, terminal-guard continuation replay, image/web-search same-key replay (rotation stays zero), fallback cap, local-mode gating - docs: coverage and cap wording in devlog 010, structure/04, and all five configuration locale rows
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@devlog/_plan/260802_429_same_target_retry/010_design.md`:
- Around line 57-67: Update the latency and concurrency sections of the retry
design to distinguish retry-wait time from total request latency, noting that
connection, response, and configured timeout durations also contribute. Revise
the request-volume bound to account for multi-key failover, including the
possibility of exhausting the retry budget on one key before attempting another,
and document the combined bound alongside the existing failover behavior.
- Around line 7-9: Update same-key retry handling around the
continuation-request rebuild in core response processing so every retry replays
a cached, body-safe representation of the identical upstream request, including
serialized body and authentication headers, rather than rebuilding it each time.
Apply this to passthrough Responses and Anthropic terminal continuations; only
rebuild after the target or adapter changes, defining deterministic equivalence
where adapters must rebuild. Extend the rate-limit retry and terminal-guard
tests to assert body and authentication/header equality, not just send counts.
- Around line 32-34: Update rate-limit retry authentication handling to fail
closed: normalize omitted key-provider auth modes to "key", preserve "local" in
providerConfigSeed (including the mapping in derive), and make
rateLimitRetryPolicyFor allow retries only when authMode === "key". Add
regression coverage for omitted, local, and unknown authentication modes while
preserving existing key-provider behavior.
In `@docs-site/src/content/docs/reference/configuration.md`:
- Line 353: Update the retryOn429 documentation across the English, Japanese,
Korean, Russian, and Chinese provider and adapter reference pages. State that it
applies only to authMode: "key" and excludes oauth, forward, and local
providers; document same-key replay, raw openai-responses passthrough,
translated openai-chat/Anthropic requests, and that custom runTurn transports
are excluded. Keep the existing defaults and behavior description consistent
across all pages.
In `@src/images/loop.ts`:
- Around line 467-490: The 429 retry waits must not consume the cumulative
header deadline in either loop. In src/images/loop.ts lines 467-490, update the
flow around the headerDeadline and rateLimitRetryPolicy retry loop so each
deliberate wait is excluded, while preserving the configured retry outcome and
preventing the 504 header-timeout path from firing due to that wait; apply the
identical change in src/web-search/loop.ts lines 361-384 around its
headerDeadline and retry loop.
In `@src/providers/key-failover.ts`:
- Around line 76-96: Ensure local providers cannot enter the key-failover retry
path when authMode is undefined: preserve authMode: "local" through the provider
derivation/routing flow or pass the registry auth kind into
rateLimitRetryPolicyFor, while retaining undefined as the default for custom
API-key providers. Add a regression test covering a local provider configured
with retryOn429: {}.
In `@src/server/responses/core.ts`:
- Around line 1631-1671: Update the retry closure in the passthrough 429 loop to
pass `"rate-limit-429"` to `noteAttemptSend` whenever `fetchWithTransientRetry`
provides no recovery kind, while preserving any recovery kind it does provide.
Keep the existing `rateLimitPolicy` retry flow unchanged.
- Around line 2629-2665: Hoist the rateLimitPolicy and rateLimitRetries
declarations out of the terminal continuation while loop, placing them beside
imageTierBias so the retry budget persists across key rotation, account
rotation, and image-tier 413 continues. Resolve rateLimitPolicy once from the
initial route.provider and preserve the existing retry loop behavior while
preventing the budget from resetting per iteration.
In `@tests/rate-limit-retry.test.ts`:
- Around line 99-159: The retry tests lack coverage proving the retry budget is
scoped per request rather than reset during failover. Add focused regression
coverage near the existing retry-loop tests using a two-key provider pool,
retryOn429 with attempts set to 1, and upstream responses that always return
429; assert the total upstream send count remains within the request-wide bound,
covering both the main recovery loop and terminal-continuation path.
In `@tests/usage-log.test.ts`:
- Around line 39-62: Replace the as never cast in the
persists-the-rate-limit-429-recovery-kind-on-attempts test with a
PersistedUsageEntry-typed object. Keep the recoveryKinds literal type-checked
against the recovery-kind union, and cast only individual fields that genuinely
require it while preserving the existing round-trip assertion.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: a0cd7539-4937-4e9d-bf2a-e78d059e114a
📒 Files selected for processing (22)
devlog/_plan/260802_429_same_target_retry/000_research.mddevlog/_plan/260802_429_same_target_retry/010_design.mddocs-site/src/content/docs/ja/reference/configuration.mddocs-site/src/content/docs/ko/reference/configuration.mddocs-site/src/content/docs/reference/configuration.mddocs-site/src/content/docs/ru/reference/configuration.mddocs-site/src/content/docs/zh-cn/reference/configuration.mdgui/src/pages/Logs.tsxsrc/config.tssrc/images/loop.tssrc/providers/key-failover.tssrc/server/responses/core.tssrc/types.tssrc/usage/log.tssrc/web-search/loop.tsstructure/04_transports-and-sidecars.mdtests/images/loop.test.tstests/rate-limit-retry.test.tstests/server-rate-limit-retry-e2e.test.tstests/terminal-guard-server.test.tstests/usage-log.test.tstests/web-search.test.ts
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a06a416024
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… deadlines, recovery labels) - key-failover: fail closed - only authMode key (or the documented omitted default) may replay; unknown modes rejected; an already-expired HTTP-date Retry-After retries immediately (like Retry-After: 0) - derive: providerConfigSeed preserves the registry auth kind (incl. local) so the gate survives the seed round-trip and routing - core: release the unread 429 body BEFORE every backoff (main loop, passthrough, continuations); passthrough and continuation replay sends record the rate-limit-429 recovery kind; continuation budget hoisted outside the failover loop so rotation/413 cannot re-arm it; continuation waits sleep on the upstream signal so an SSE body-cancel aborts them too - images/web-search loops: release the 429 body before the wait, restart the response-header deadline after each deliberate wait (backoffs never consume the connect budget or surface as 504), and record rate-limit-429 on replay sends - config: load-time degradation for invalid optional retryOn429 fields (warn + drop the field) instead of tripping the whole schema and hiding all providers behind a default config; the management write boundary still rejects - tests: fail-closed auth modes (incl. unknown), expired HTTP-date, per-request budget across 2-key pools (e2e + terminal continuation), byte-identical replay bodies/auth (passthrough + continuation), recovery telemetry in both loops, config load degradation, registry auth-kind preservation, typed usage-log entry - docs: retry-wait vs total-latency and attempts+poolKeys volume bounds, identical-replay equivalence, deadline/backoff behavior in devlog 010 and structure/04; retryOn429 boundary notes in the provider guide and adapter reference (English + ja/ko/ru/zh-cn)
|
All 17 inline review comments are addressed in 58c36c9 (plus docstrings in 0feede1 for the coverage check). Highlights: fail-closed auth (registry now preserves authMode "local"), expired Retry-After dates retry immediately, unread 429 bodies are released before every backoff, rate-limit-429 is recorded on every retry surface, bridge loops restart their header deadline after each wait, the continuation budget is per-request, invalid optional retryOn429 fields degrade at load instead of discarding the config, and replay identity is asserted byte-for-byte in tests. Replies were posted on each thread; the 7 open threads are resolved. The PR remains a draft. |
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/server/responses/core.ts (1)
2647-2685: 🩺 Stability & Availability | 🔴 Critical | 🏗️ Heavy liftEmit upstream heartbeats during 429 backoff
The bridge checks upstream activity every 2,000 ms. It increments
stallTickswhen no adapter event arrives and aborts atceil(stallTimeoutSec * 1000 / 2000)ticks. Wire heartbeats do not reset this counter.The retry waits can be silent for up to 600,000 ms. This exceeds the default 300-second core budget and the default 330-second web-search budget.
Apply the fix to these live generators:
src/server/responses/core.ts:2647-2685src/web-search/loop.ts:363-393src/images/loop.ts:469-498Split each
sleepWithAbortcall into chunks shorter than 2,000 ms, such as 1,000 ms. Yield{ type: "heartbeat" }after every chunk, including the final chunk. Add regression coverage for a backoff longer thanstallTimeoutSec. The pre-stream retry loops insrc/server/responses/core.tsdo not require this change.Update the web-search timeout documentation and tests to state that retry backoff remains live through these upstream heartbeats.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/server/responses/core.ts` around lines 2647 - 2685, Update the live retry-backoff loops in src/server/responses/core.ts:2647-2685, src/web-search/loop.ts:363-393, and src/images/loop.ts:469-498 to split sleepWithAbort waits into sub-2,000 ms chunks, yielding a heartbeat after every chunk including the final one, while preserving abort handling; leave pre-stream retry loops unchanged. Add regression coverage for backoffs exceeding stallTimeoutSec, and update web-search timeout documentation and tests to describe continued liveness through upstream heartbeats.src/images/loop.ts (1)
469-476: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick winKeep the retry budget scoped consistently.
src/images/loop.tsinitializesrateLimitRetriesinsideprepareIterationEvents, while onerunWithImageBridgerequest can execute multiple image-loop iterations. This can re-arm the same-key budget and exceed the documented per-request send bound.
src/images/loop.ts#L469-L476: move the policy and counter to request scope, or explicitly define the budget per iteration.structure/04_transports-and-sidecars.md#L235-L235: update theattempts + poolKeysbound if per-iteration scope is intentional.devlog/_plan/260802_429_same_target_retry/010_design.md#L95-L110: add a multi-iteration image-bridge regression test.#!/usr/bin/env bash set -euo pipefail rg -n -C 8 'prepareIterationEvents|rateLimitRetries|rateLimitRetryPolicy|HARD_CAP|retryOn429Policy' \ src/images/loop.ts tests/images/loop.test.tsAs per path instructions, add a focused regression in the flat Bun tests for this
src/**behavior change.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/images/loop.ts` around lines 469 - 476, Keep the 429 retry policy and counter request-scoped in prepareIterationEvents/runWithImageBridge so multiple image-loop iterations cannot reset the same-key retry budget; preserve the documented per-request send bound. Update structure/04_transports-and-sidecars.md:235 to reflect the chosen request-scoped bound, and extend devlog/_plan/260802_429_same_target_retry/010_design.md:95-110 with a multi-iteration image-bridge regression scenario. Add a focused regression to the flat Bun tests covering this behavior.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@devlog/_plan/260802_429_same_target_retry/010_design.md`:
- Around line 88-89: Align the design and associated tests with the current
implementation: update the expired HTTP-date and numeric Retry-After: 0 cases to
expect the configured fallback interval when the parser returns undefined,
unless the parser and rateLimitRetryDelayMs are intentionally changed to
propagate a zero delay. Keep the documented behavior, parser expectations, and
tests consistent.
In `@docs-site/src/content/docs/ja/reference/adapters.md`:
- Around line 41-43: Update the retry descriptions in
docs-site/src/content/docs/ja/reference/adapters.md:41-43,
docs-site/src/content/docs/ko/reference/adapters.md:48-50,
docs-site/src/content/docs/ru/reference/adapters.md:51-53, and
docs-site/src/content/docs/zh-cn/reference/adapters.md:46-48 to state that
same-key 429 replay occurs before other handling or failover, while preserving
the existing translated behavior and custom runTurn exception.
In `@docs-site/src/content/docs/zh-cn/guides/providers.md`:
- Around line 37-40: Update the Chinese provider guide passage describing
retryOn429 to explicitly state that it is opt-in and disabled by default unless
configured, while preserving the existing API-key-only restriction and OAuth,
forward, and local exclusions. Keep the wording consistent with the
corresponding English provider guide.
In `@src/config.ts`:
- Around line 1135-1149: Update the retryOn429 sanitization around the fields
definition and loop to iterate over all keys in policy, warning and ignoring any
key not in the recognized field set. Preserve the existing validators and
warning behavior for known fields with invalid values, and continue rebuilding
p.retryOn429 from only accepted entries.
In `@src/images/loop.ts`:
- Around line 479-486: Await the unread 429 response body cancellation in the
retry flow around sleepWithAbort, while preserving handling for already-closed
bodies and cancellation failures. Update
devlog/_plan/260802_429_same_target_retry/010_design.md lines 53-55 and
structure/04_transports-and-sidecars.md lines 236-237 only as needed to keep
their “release before backoff” claims accurate after the implementation change.
- Around line 482-498: The retry flow around sleepWithAbort in
src/images/loop.ts lines 482-498 must clear headerDeadline before sleeping,
check signal.aborted immediately afterward and throw the existing 499 LoopError
before telemetry or replay, then create the replacement deadline before
onRateLimitRetrySend and fetchOnce. Update
structure/04_transports-and-sidecars.md lines 238-242 to preserve and document
the 499-before-replay guarantee; no other behavior needs changing.
---
Outside diff comments:
In `@src/images/loop.ts`:
- Around line 469-476: Keep the 429 retry policy and counter request-scoped in
prepareIterationEvents/runWithImageBridge so multiple image-loop iterations
cannot reset the same-key retry budget; preserve the documented per-request send
bound. Update structure/04_transports-and-sidecars.md:235 to reflect the chosen
request-scoped bound, and extend
devlog/_plan/260802_429_same_target_retry/010_design.md:95-110 with a
multi-iteration image-bridge regression scenario. Add a focused regression to
the flat Bun tests covering this behavior.
In `@src/server/responses/core.ts`:
- Around line 2647-2685: Update the live retry-backoff loops in
src/server/responses/core.ts:2647-2685, src/web-search/loop.ts:363-393, and
src/images/loop.ts:469-498 to split sleepWithAbort waits into sub-2,000 ms
chunks, yielding a heartbeat after every chunk including the final one, while
preserving abort handling; leave pre-stream retry loops unchanged. Add
regression coverage for backoffs exceeding stallTimeoutSec, and update
web-search timeout documentation and tests to describe continued liveness
through upstream heartbeats.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 9881d8f6-9808-4382-a953-a3a067cfc38b
📒 Files selected for processing (26)
devlog/_plan/260802_429_same_target_retry/010_design.mddocs-site/src/content/docs/guides/providers.mddocs-site/src/content/docs/ja/guides/providers.mddocs-site/src/content/docs/ja/reference/adapters.mddocs-site/src/content/docs/ko/guides/providers.mddocs-site/src/content/docs/ko/reference/adapters.mddocs-site/src/content/docs/reference/adapters.mddocs-site/src/content/docs/ru/guides/providers.mddocs-site/src/content/docs/ru/reference/adapters.mddocs-site/src/content/docs/zh-cn/guides/providers.mddocs-site/src/content/docs/zh-cn/reference/adapters.mdsrc/config.tssrc/images/loop.tssrc/providers/derive.tssrc/providers/key-failover.tssrc/server/responses/core.tssrc/web-search/loop.tsstructure/04_transports-and-sidecars.mdtests/config-user-edits.test.tstests/images/loop.test.tstests/provider-registry-parity.test.tstests/rate-limit-retry.test.tstests/server-rate-limit-retry-e2e.test.tstests/terminal-guard-server.test.tstests/usage-log.test.tstests/web-search.test.ts
…ne race, config warnings, locale docs) - images/web-search loops: AWAIT the unread 429 body cancellation before the backoff; clear the old header deadline BEFORE the sleep; re-check client cancellation after the wait so 499 wins over stale-deadline edges; start the fresh deadline before telemetry and replay - config: sanitizeRetryOn429ForLoad warns about misnamed keys (e.g. attempt) instead of silently dropping them - docs: locale adapter pages state same-key replay runs before other handling/failover; provider guide (English + ja/ko/ru/zh-cn) states retryOn429 is opt-in (absent = off); devlog and structure/04 note the awaited cancellation and the 499-before-replay guarantee - test: config-user-edits covers the misnamed-key drop
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
devlog/_plan/260802_429_same_target_retry/010_design.md (1)
47-53: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDocument the
runTurnexception for bridge retries.
src/images/loop.tsenters theadapter.runTurnbranch at Lines 355-433 and returns before the HTTP 429 retry loop at Lines 477-513. A key-auth customrunTurnadapter therefore does not receive this HTTP retry policy. State that bridge retries apply to HTTP adapters only and exclude customrunTurntransports.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@devlog/_plan/260802_429_same_target_retry/010_design.md` around lines 47 - 53, Update the retry-policy documentation near the bridge retry references to state that image/video bridge retries apply only to HTTP adapters. Explicitly exclude custom adapters using the adapter.runTurn path in src/images/loop.ts, which returns before the HTTP 429 retry loop; do not imply that runTurn transports receive the same wait-and-replay behavior.src/server/responses/core.ts (1)
2668-2707: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winAwait terminal-continuation body cancellation before the backoff.
At Line 2679,
response.body.cancel()is detached withvoid.sleepWithAbort()then starts at Line 2681 while cancellation can still be pending. Under repeated 429 responses, unread bodies can remain active through the retry wait and replay.Await cancellation before sleeping.
Proposed fix
- try { void response.body?.cancel().catch(() => {}); } catch { /* already closed */ } + try { await response.body?.cancel().catch(() => {}); } catch { /* already closed */ }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/server/responses/core.ts` around lines 2668 - 2707, In the terminal-continuation retry loop, update the response body cleanup before sleepWithAbort to await response.body cancellation rather than detaching the promise. Preserve the existing safe handling for absent or already-closed bodies, and ensure the backoff starts only after cancellation has settled.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@devlog/_plan/260802_429_same_target_retry/010_design.md`:
- Around line 47-53: Update the retry-policy documentation near the bridge retry
references to state that image/video bridge retries apply only to HTTP adapters.
Explicitly exclude custom adapters using the adapter.runTurn path in
src/images/loop.ts, which returns before the HTTP 429 retry loop; do not imply
that runTurn transports receive the same wait-and-replay behavior.
In `@src/server/responses/core.ts`:
- Around line 2668-2707: In the terminal-continuation retry loop, update the
response body cleanup before sleepWithAbort to await response.body cancellation
rather than detaching the promise. Preserve the existing safe handling for
absent or already-closed bodies, and ensure the backoff starts only after
cancellation has settled.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 72915daa-eace-482c-83cd-3c1d098cc169
📒 Files selected for processing (24)
devlog/_plan/260802_429_same_target_retry/010_design.mddocs-site/src/content/docs/guides/providers.mddocs-site/src/content/docs/ja/guides/providers.mddocs-site/src/content/docs/ja/reference/adapters.mddocs-site/src/content/docs/ko/guides/providers.mddocs-site/src/content/docs/ko/reference/adapters.mddocs-site/src/content/docs/ru/guides/providers.mddocs-site/src/content/docs/ru/reference/adapters.mddocs-site/src/content/docs/zh-cn/guides/providers.mddocs-site/src/content/docs/zh-cn/reference/adapters.mdsrc/config.tssrc/images/loop.tssrc/providers/derive.tssrc/providers/key-failover.tssrc/server/responses/core.tssrc/types.tssrc/web-search/loop.tsstructure/04_transports-and-sidecars.mdtests/config-user-edits.test.tstests/images/loop.test.tstests/server-rate-limit-retry-e2e.test.tstests/terminal-guard-server.test.tstests/usage-log.test.tstests/web-search.test.ts
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1af83c39cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… send telemetry, invalid master switch, secret-safe warnings)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4e1978821
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ncel, post-sleep abort re-checks, pinned xAI req-id)
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
src/server/responses/core.ts (1)
2617-2666: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReduce the duplicated recovery-label expression in
fetchContinuation.Lines 2638 and 2647 both derive the same label from
replay. The second one additionally merges the transient-retry kind. The logic is correct, but the label rule now lives in two places, so a future change to the label (for example a distinct kind for continuation replays) can easily update only one branch.♻️ Optional consolidation
if (continuationEstimate !== undefined) logCtx.usageLogInputTokens = continuationEstimate; + const replayKind: AttemptRecoveryKind | undefined = replay ? "rate-limit-429" : undefined; try { if (activeAdapter.fetchResponse) { - noteAttemptSend(logCtx.activeAttempt, continuationEstimate, replay ? "rate-limit-429" : undefined); + noteAttemptSend(logCtx.activeAttempt, continuationEstimate, replayKind); return await activeAdapter.fetchResponse(continuationRequest, { @@ return await fetchWithResetRetry( recovery => { - noteAttemptSend(logCtx.activeAttempt, continuationEstimate, recovery ?? (replay ? "rate-limit-429" : undefined)); + noteAttemptSend(logCtx.activeAttempt, continuationEstimate, recovery ?? replayKind);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/server/responses/core.ts` around lines 2617 - 2666, Consolidate the replay-derived recovery label in fetchContinuation by computing it once before the fetch branches, then reuse that value in both noteAttemptSend calls while preserving the fetchWithResetRetry recovery override behavior.src/config.ts (1)
1144-1158: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueInclude the accepted range in the warning for out-of-range numbers.
The validators at Lines 1146-1148 reject both wrong types and out-of-range numbers, but the warning at Line 1157 reports only
typeof value. A user who writes"attempts": 999seesproviders.x.retryOn429.attempts (number) is invalid, which gives no hint about the accepted bound. Numbers and booleans cannot carry secret material, so the range can be stated safely while the value itself stays unlogged.♻️ Proposed diagnostic improvement
- const fields: Array<[string, (value: unknown) => boolean]> = [ - ["enabled", value => typeof value === "boolean"], - ["attempts", value => typeof value === "number" && Number.isInteger(value) && value >= 1 && value <= 20], - ["intervalMs", value => typeof value === "number" && Number.isInteger(value) && value >= 100 && value <= 600_000], - ["maxIntervalMs", value => typeof value === "number" && Number.isInteger(value) && value >= 100 && value <= 600_000], - ["respectRetryAfter", value => typeof value === "boolean"], + const fields: Array<[string, (value: unknown) => boolean, string]> = [ + ["enabled", value => typeof value === "boolean", "expected boolean"], + ["attempts", value => typeof value === "number" && Number.isInteger(value) && value >= 1 && value <= 20, "expected integer 1-20"], + ["intervalMs", value => typeof value === "number" && Number.isInteger(value) && value >= 100 && value <= 600_000, "expected integer 100-600000"], + ["maxIntervalMs", value => typeof value === "number" && Number.isInteger(value) && value >= 100 && value <= 600_000, "expected integer 100-600000"], + ["respectRetryAfter", value => typeof value === "boolean", "expected boolean"], ]; const cleaned: Record<string, unknown> = {}; - for (const [key, isValid] of fields) { + for (const [key, isValid, expectation] of fields) { const value = policyRecord[key]; if (value === undefined) continue; if (isValid(value)) cleaned[key] = value; // Log only the received type, never the value (provider config can hold secrets). - else console.warn(`⚠️ config.json providers.${name}.retryOn429.${key} (${typeof value}) is invalid — ignoring the field`); + else console.warn(`⚠️ config.json providers.${name}.retryOn429.${key} (${typeof value}) is invalid — ${expectation}; ignoring the field`); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/config.ts` around lines 1144 - 1158, Update the validation warning in the retry policy field loop around the `fields` validators to include the accepted ranges for numeric keys such as `attempts`, `intervalMs`, and `maxIntervalMs`, while retaining type-only reporting and never logging the received value. Keep boolean warnings unchanged and preserve the existing validation and field-cleaning behavior.src/images/loop.ts (1)
483-515: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winBound pre-header 429 retries before creating SSE
For non-
runTurnimage requests and all web-search requests,prepareIterationDrainedconsumes the retry heartbeat beforebridgeToResponsesSSEis created. The client receives no response headers while same-target 429 retries wait. The default policy can wait 3 minutes; valid configuration can wait up to 200 minutes (20 × 600_000ms). Bound the aggregate eager-phase wait to the request header budget, or move these retries into the liveproduce()phase. Apply the fix to both loops.runTurnimage requests already skip the eager drain.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/images/loop.ts` around lines 483 - 515, Bound same-target 429 retry waiting during the eager preparation phase so it cannot exceed the request header-timeout budget before SSE creation, or defer the retries into the live produce phase. Apply the corresponding fix to the retry loop in src/images/loop.ts lines 483-515 and src/web-search/loop.ts lines 380-412; preserve the existing runTurn image behavior, which already skips eager draining.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/providers/xai-transport.ts`:
- Around line 135-138: Rename the second parameter of withGeneratedRequestId to
reflect that it receives the already-resolved request ID, whether configured or
generated. Update all references within the helper while preserving the single
fallback in the request setup where configuredRequestId ?? randomUUID() is
computed, ensuring retries continue using the same ID.
In `@tests/config-user-edits.test.ts`:
- Around line 183-185: In the test assertion for the retryOn429 diagnostic,
replace the loose "string" substring check with an assertion anchored to the
retryOn429 field and its parenthesized received type. Keep the secret-redaction
assertion unchanged and ensure the expectation specifically verifies the
type-only warning from loadConfig().
---
Outside diff comments:
In `@src/config.ts`:
- Around line 1144-1158: Update the validation warning in the retry policy field
loop around the `fields` validators to include the accepted ranges for numeric
keys such as `attempts`, `intervalMs`, and `maxIntervalMs`, while retaining
type-only reporting and never logging the received value. Keep boolean warnings
unchanged and preserve the existing validation and field-cleaning behavior.
In `@src/images/loop.ts`:
- Around line 483-515: Bound same-target 429 retry waiting during the eager
preparation phase so it cannot exceed the request header-timeout budget before
SSE creation, or defer the retries into the live produce phase. Apply the
corresponding fix to the retry loop in src/images/loop.ts lines 483-515 and
src/web-search/loop.ts lines 380-412; preserve the existing runTurn image
behavior, which already skips eager draining.
In `@src/server/responses/core.ts`:
- Around line 2617-2666: Consolidate the replay-derived recovery label in
fetchContinuation by computing it once before the fetch branches, then reuse
that value in both noteAttemptSend calls while preserving the
fetchWithResetRetry recovery override behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 3f71039b-4261-460f-853e-cf5dc91ff470
📒 Files selected for processing (11)
src/config.tssrc/images/loop.tssrc/providers/xai-transport.tssrc/server/responses/core.tssrc/web-search/loop.tstests/config-user-edits.test.tstests/images/loop.test.tstests/rate-limit-retry.test.tstests/terminal-guard-server.test.tstests/web-search.test.tstests/xai-transport.test.ts
The shared-request cache refactor inserted the cachedRequest/cachedAdapter declarations between the fetchOnce JSDoc and the function itself, orphaning the doc block; reattach an adjacent JSDoc to both fetchOnce declarations and add one to the bounded-body onAbort hook so the declarations stay documented.
|
@coderabbitai resume Docstring root-cause found: the shared-request cache refactor had orphaned the JSDoc blocks — |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
src/server/responses/core.ts (2)
2881-2886: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winRedact terminal-continuation errors before yielding them.
The catch blocks at Lines 2881-2886 and 2929-2934 copy raw
error.messagetext intoAdapterEventmessages. The Responses bridge can serialize these messages to the client. A request-building or transport error may contain credential-shaped text or upstream request details.The surrounding error paths already use
redactSecretStringat Lines 2475-2476, 2781-2783, and 3034. Apply the same sanitizer to both continuation messages.Proposed fix
- yield { type: "error", message: `Provider continuation failed: ${error instanceof Error ? error.message : String(error)}` }; + yield { type: "error", message: `Provider continuation failed: ${redactSecretString(error instanceof Error ? error.message : String(error))}` };Apply the same change in both catch blocks.
As per path instructions,
src/**code must never log or serialize tokens or OAuth material into responses.Also applies to: 2929-2934
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/server/responses/core.ts` around lines 2881 - 2886, Apply redactSecretString to the error detail used in both terminal-continuation catch blocks, including the paths yielding “Provider continuation failed” messages near the continuation handlers. Preserve the existing client-aborted response and fallback String(error) behavior, but sanitize the resulting error text before placing it in either AdapterEvent message.Source: Path instructions
2845-2846: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winPreserve recovery kinds across terminal-continuation failover.
fetchContinuationonly maps the boolean replay state torate-limit-429at Lines 2845-2846. After key rotation, Anthropic account rotation, or image-tier retry, the loop callsfetchContinuation()with the default state. The next send therefore has nokey-429,anthropic-oauth-429, orimage-413recovery classification.The main recovery loop records these classifications. The continuation path must use the same contract so persisted attempt data and GUI recovery details remain accurate.
Pass an
AttemptRecoveryKindthroughfetchContinuation, or store a pending recovery kind before eachcontinue.Based on the PR objectives, recovery classifications must be persisted and rendered in the GUI.
Also applies to: 2949-2954, 2976-2984, 2997-2999
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/server/responses/core.ts` around lines 2845 - 2846, Preserve the active AttemptRecoveryKind when continuation retries follow key rotation, Anthropic account rotation, or image-tier recovery. Update fetchContinuation and its callers to accept and forward the pending recovery classification instead of deriving only rate-limit-429 from the boolean replay state, ensuring subsequent sends persist and render key-429, anthropic-oauth-429, and image-413 consistently with the main recovery loop.docs-site/src/content/docs/ja/reference/configuration/providers.md (1)
36-36: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winPrefix the localized routing links.
English uses the root path, while translated pages use locale prefixes. Update:
docs-site/src/content/docs/ja/reference/configuration/providers.md:36to/ja/reference/configuration/routing/docs-site/src/content/docs/ko/reference/configuration/providers.md:36to/ko/reference/configuration/routing/The current links open the English routing page instead of the localized page.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs-site/src/content/docs/ja/reference/configuration/providers.md` at line 36, Update the routing link in docs-site/src/content/docs/ja/reference/configuration/providers.md:36 to use the /ja/reference/configuration/routing/ locale-prefixed path, and make the corresponding change in docs-site/src/content/docs/ko/reference/configuration/providers.md:36 to use /ko/reference/configuration/routing/.Source: Path instructions
docs-site/src/content/docs/ru/reference/configuration/providers.md (1)
37-37: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winKeep translated routing links inside their locale trees.
Both changed links use the English absolute route from translated configuration pages. Use the locale-prefixed route at each site.
docs-site/src/content/docs/ru/reference/configuration/providers.md#L37-L37: use/ru/reference/configuration/routing/.docs-site/src/content/docs/zh-cn/reference/configuration/providers.md#L35-L35: use/zh-cn/reference/configuration/routing/.As per path instructions, translated
docs-site/**pages must preserve locale navigation.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs-site/src/content/docs/ru/reference/configuration/providers.md` at line 37, Update the routing link in docs-site/src/content/docs/ru/reference/configuration/providers.md:37-37 to use /ru/reference/configuration/routing/. Update the corresponding link in docs-site/src/content/docs/zh-cn/reference/configuration/providers.md:35-35 to use /zh-cn/reference/configuration/routing/, preserving locale-specific navigation in both translated pages.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs-site/src/content/docs/ja/reference/configuration/providers.md`:
- Line 36: Update the routing link in
docs-site/src/content/docs/ja/reference/configuration/providers.md:36 to use the
/ja/reference/configuration/routing/ locale-prefixed path, and make the
corresponding change in
docs-site/src/content/docs/ko/reference/configuration/providers.md:36 to use
/ko/reference/configuration/routing/.
In `@docs-site/src/content/docs/ru/reference/configuration/providers.md`:
- Line 37: Update the routing link in
docs-site/src/content/docs/ru/reference/configuration/providers.md:37-37 to use
/ru/reference/configuration/routing/. Update the corresponding link in
docs-site/src/content/docs/zh-cn/reference/configuration/providers.md:35-35 to
use /zh-cn/reference/configuration/routing/, preserving locale-specific
navigation in both translated pages.
In `@src/server/responses/core.ts`:
- Around line 2881-2886: Apply redactSecretString to the error detail used in
both terminal-continuation catch blocks, including the paths yielding “Provider
continuation failed” messages near the continuation handlers. Preserve the
existing client-aborted response and fallback String(error) behavior, but
sanitize the resulting error text before placing it in either AdapterEvent
message.
- Around line 2845-2846: Preserve the active AttemptRecoveryKind when
continuation retries follow key rotation, Anthropic account rotation, or
image-tier recovery. Update fetchContinuation and its callers to accept and
forward the pending recovery classification instead of deriving only
rate-limit-429 from the boolean replay state, ensuring subsequent sends persist
and render key-429, anthropic-oauth-429, and image-413 consistently with the
main recovery loop.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 9176eec6-2272-4a3f-8827-7e21f9957d95
📒 Files selected for processing (18)
docs-site/src/content/docs/ja/guides/providers.mddocs-site/src/content/docs/ja/reference/configuration/providers.mddocs-site/src/content/docs/ko/guides/providers.mddocs-site/src/content/docs/ko/reference/configuration/providers.mddocs-site/src/content/docs/reference/configuration/providers.mddocs-site/src/content/docs/ru/guides/providers.mddocs-site/src/content/docs/ru/reference/configuration/providers.mddocs-site/src/content/docs/zh-cn/guides/providers.mddocs-site/src/content/docs/zh-cn/reference/configuration/providers.mdgui/src/i18n/de.tsgui/src/i18n/en.tsgui/src/i18n/ja.tsgui/src/i18n/ko.tsgui/src/i18n/ru.tsgui/src/i18n/zh.tssrc/server/responses/core.tsstructure/04_transports-and-sidecars.mdtests/web-search.test.ts
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '=== Current head and merge commit ==='
git rev-parse HEAD
git show --no-patch --format='commit=%H%nparents=%P%nsubject=%s' c3b0ec8b
printf '%s\n' '=== Changed paths in the merge resolution ==='
git diff-tree --no-commit-id --name-status -m -r c3b0ec8b -- src/server/responses/core.ts
printf '%s\n' '=== Merge-resolution diff for core.ts ==='
git show --cc --format= --unified=35 c3b0ec8b -- src/server/responses/core.ts
printf '%s\n' '=== Focused passthrough 429 retry context ==='
rg -n -C 18 \
'passthrough|rateLimitRetry|rateLimitRetries|rateLimitRetryDelayMs|status\s*===\s*429|response\.status\s*===\s*429' \
src/server/responses/core.ts
printf '%s\n' '=== Fixed-account forward-pool retry context ==='
rg -n -C 18 \
'fixedAccount|forward.*pool|pool.*forward|authCtx' \
src/server/responses/core.ts
printf '%s\n' '=== Relevant function structure ==='
ast-grep outline src/server/responses/core.ts --items all --type function --match 'handleResponses|fetch' --view expandedLength of output: 127992
The passthrough same-target 429 retry loop remains in
The forward-pool alternate-account branch now checks
The inspected merge resolution preserves both required behaviors. Normal review is resumed. ✅ Action performedReviews resumed. |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (5)
docs-site/src/content/docs/ru/reference/configuration/providers.md (1)
98-98: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDo not describe retry-budget exhaustion as disabling key failover.
retryOn429.attemptslimits additional same-key replays for one request. The documented contract performs those replays before the normal key-failover path. The current Russian text says that an exhausted budget returns 429 without key failover.State that the budget stops further same-key replays. Then document that normal failover or final-error handling applies according to the available targets.
As per path instructions, user-facing documentation must match actual CLI and API behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs-site/src/content/docs/ru/reference/configuration/providers.md` at line 98, Update the retryOn429 documentation row to clarify that exhausting attempts only stops additional same-key replays. State that normal key failover or final-error handling then proceeds according to the available targets, removing the claim that exhaustion directly returns 429 without key failover.Source: Path instructions
src/images/loop.ts (2)
444-494: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winBridge transient-retry telemetry is incomplete.
Both bridge loops report telemetry before
fetchWithResetRetry, so additional helper-driven sends are not recorded with their recovery kind.
src/images/loop.ts#L444-L494: moveonAttemptSendinto thefetchWithResetRetrycallback and useretryRecovery ?? recovery.src/web-search/loop.ts#L343-L397: apply the same callback-based telemetry and preserve the initial recovery kind.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/images/loop.ts` around lines 444 - 494, Update the fetch flows in src/images/loop.ts lines 444-494 and src/web-search/loop.ts lines 343-397: move onAttemptSend into each fetchWithResetRetry callback so every helper-driven send is recorded, passing retryRecovery ?? recovery to preserve the initial recovery kind when no retry recovery is available. Apply the same telemetry change in both sites.
502-536: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winBridge key-failover telemetry omits
key-429.The rotated adapter sends are not classified as key failovers, unlike the core recovery loop.
src/images/loop.ts#L502-L536: callfetchOnce(adapter, "key-429")afterdeps.on429returns the rotated adapter.src/web-search/loop.ts#L405-L439: make the samefetchOnce(adapter, "key-429")change.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/images/loop.ts` around lines 502 - 536, Update the rotated-adapter retry paths in src/images/loop.ts lines 502-536 and src/web-search/loop.ts lines 405-439 so the fetchOnce call made after deps.on429 returns the rotated adapter uses the "key-429" classification, matching core recovery telemetry; no other retry behavior should change.structure/04_transports-and-sidecars.md (1)
256-280: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winExclude
runTurntransports from the retry scope.src/images/loop.ts:364-426handlesrunTurnadapters through an event queue and receives no HTTP status, so the policy passed atsrc/server/responses/core.ts:2236cannot replay a 429. Limit lines 256-279 to HTTP-capable API-key adapters and explicitly exclude customrunTurntransports.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@structure/04_transports-and-sidecars.md` around lines 256 - 280, Update the retryOn429 documentation to scope the policy only to HTTP-capable API-key adapters, explicitly excluding custom runTurn transports handled through the event queue in the image loop. Remove any wording that implies runTurn adapters can receive HTTP 429 statuses or replay requests via the policy passed from the Responses core.docs-site/src/content/docs/ko/reference/configuration/providers.md (1)
36-36: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winKeep translated routing links inside their locale trees.
docs-site/src/content/docs/ko/reference/configuration/providers.md#L36-L36: use/ko/reference/configuration/routing/.docs-site/src/content/docs/zh-cn/reference/configuration/providers.md#L35-L35: use/zh-cn/reference/configuration/routing/.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs-site/src/content/docs/ko/reference/configuration/providers.md` at line 36, Update the routing links in docs-site/src/content/docs/ko/reference/configuration/providers.md:36-36 and docs-site/src/content/docs/zh-cn/reference/configuration/providers.md:35-35 to remain within their locale trees, using the /ko/reference/configuration/routing/ and /zh-cn/reference/configuration/routing/ paths respectively.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs-site/src/content/docs/ko/reference/configuration/providers.md`:
- Line 36: Update the routing links in
docs-site/src/content/docs/ko/reference/configuration/providers.md:36-36 and
docs-site/src/content/docs/zh-cn/reference/configuration/providers.md:35-35 to
remain within their locale trees, using the /ko/reference/configuration/routing/
and /zh-cn/reference/configuration/routing/ paths respectively.
In `@docs-site/src/content/docs/ru/reference/configuration/providers.md`:
- Line 98: Update the retryOn429 documentation row to clarify that exhausting
attempts only stops additional same-key replays. State that normal key failover
or final-error handling then proceeds according to the available targets,
removing the claim that exhaustion directly returns 429 without key failover.
In `@src/images/loop.ts`:
- Around line 444-494: Update the fetch flows in src/images/loop.ts lines
444-494 and src/web-search/loop.ts lines 343-397: move onAttemptSend into each
fetchWithResetRetry callback so every helper-driven send is recorded, passing
retryRecovery ?? recovery to preserve the initial recovery kind when no retry
recovery is available. Apply the same telemetry change in both sites.
- Around line 502-536: Update the rotated-adapter retry paths in
src/images/loop.ts lines 502-536 and src/web-search/loop.ts lines 405-439 so the
fetchOnce call made after deps.on429 returns the rotated adapter uses the
"key-429" classification, matching core recovery telemetry; no other retry
behavior should change.
In `@structure/04_transports-and-sidecars.md`:
- Around line 256-280: Update the retryOn429 documentation to scope the policy
only to HTTP-capable API-key adapters, explicitly excluding custom runTurn
transports handled through the event queue in the image loop. Remove any wording
that implies runTurn adapters can receive HTTP 429 statuses or replay requests
via the policy passed from the Responses core.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 0eb360b8-4f4d-4694-8457-2ec054d0b575
📒 Files selected for processing (21)
docs-site/src/content/docs/ja/guides/providers.mddocs-site/src/content/docs/ja/reference/configuration/providers.mddocs-site/src/content/docs/ko/guides/providers.mddocs-site/src/content/docs/ko/reference/configuration/providers.mddocs-site/src/content/docs/reference/configuration/providers.mddocs-site/src/content/docs/ru/guides/providers.mddocs-site/src/content/docs/ru/reference/configuration/providers.mddocs-site/src/content/docs/zh-cn/guides/providers.mddocs-site/src/content/docs/zh-cn/reference/configuration/providers.mdgui/src/i18n/de.tsgui/src/i18n/en.tsgui/src/i18n/ja.tsgui/src/i18n/ko.tsgui/src/i18n/ru.tsgui/src/i18n/zh.tssrc/images/loop.tssrc/lib/upstream-retry.tssrc/server/responses/core.tssrc/web-search/loop.tsstructure/04_transports-and-sidecars.mdtests/web-search.test.ts
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
repo="lidge-jun/opencodex"
pr="865"
commit="4650baef"
printf '%s\n' '=== Commit and PR head ==='
git show --no-patch --format='commit=%H%nsubject=%s' "$commit"
gh pr view "$pr" --repo "$repo" \
--json headRefOid,baseRefOid,headRefName \
--jq '{headRefName,headRefOid,baseRefOid}'
head="$(gh pr view "$pr" --repo "$repo" --json headRefOid --jq '.headRefOid')"
printf '%s\n' '=== Changed files in 4650baef ==='
git diff-tree --no-commit-id --name-status -r "$commit"
printf '%s\n' '=== JSDoc and declaration contexts ==='
rg -n -C 10 \
'fetchOnce|cachedRequest|cachedAdapter|onAbort|releaseResponseBodyBestEffort' \
src/images/loop.ts src/web-search/loop.ts src/lib/upstream-retry.ts
printf '%s\n' '=== Exact production patch ==='
git show --format= --unified=20 "$commit" -- \
src/images/loop.ts src/web-search/loop.ts src/lib/upstream-retry.ts
printf '%s\n' '=== Docstring Coverage check runs on the PR head ==='
gh api --paginate "repos/${repo}/commits/${head}/check-runs?per_page=100" \
--jq '.check_runs[]
| select((.name + " " + (.workflow_name // "") + " " + (.output.title // ""))
| test("docstring|coverage"; "i"))
| {
name, status, conclusion, details_url,
title: .output.title, summary: .output.summary, text: .output.text
}' || true
printf '%s\n' '=== Legacy coverage statuses on the PR head ==='
gh api "repos/${repo}/commits/${head}/status" \
--jq '.statuses[]
| select((.context + " " + (.description // "")) | test("docstring|coverage"; "i"))
| {context,state,description,target_url}' || trueLength of output: 35311
The JSDoc blocks are now adjacent to the declarations that the cache variables previously separated:
This fixes the documented attachment failure mode. I could not verify a refreshed Docstring Coverage result. GitHub currently returns no matching check run and no legacy status for Docstring Coverage on I will resume normal review. If the refreshed check still reports below 80%, provide its run URL or output and I will identify the remaining declarations from the actual evaluator data. ✅ Action performedReviews resumed. |
|
Please put your Pull-Request on Ready for Review, once you are finished. |
CodeRabbit round on 4650bae: - images/web-search loops: move onAttemptSend into the fetchWithResetRetry callback (retryRecovery ?? recovery) so helper-driven connection-reset sends are recorded; classify rotated-adapter fetches as key-429 like core recovery. - providers docs (en/ja/ko/ru/zh-cn): state explicitly that exhausting attempts only stops same-key replays, with normal failover or final-error handling next. - structure/04: scope retryOn429 to HTTP-capable API-key adapters and exclude custom runTurn transports. - ko/zh-cn providers ref: keep routing links inside each locale tree.
|
@coderabbitai resume Addressed all five findings from the 4650bae round in 6fb4fe2:
Verified: typecheck, privacy scan, 99 bridge/retry tests pass. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs-site/src/content/docs/reference/configuration/providers.md`:
- Line 95: Correct the duplicate sentence terminator in the retryOn429
documentation row by changing the ending after “surfaces as-is” to a single
period. Keep the rest of the retry description unchanged and ensure equivalent
localized documentation remains synchronized if applicable.
In `@docs-site/src/content/docs/ru/reference/configuration/providers.md`:
- Line 98: In the retryOn429 documentation entry, replace the semicolon
immediately before the sentence beginning “Исчерпание attempts” with a period,
preserving all surrounding Russian text and punctuation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: c9e7857b-57d8-41ee-80e2-5994209cbb06
📒 Files selected for processing (8)
docs-site/src/content/docs/ja/reference/configuration/providers.mddocs-site/src/content/docs/ko/reference/configuration/providers.mddocs-site/src/content/docs/reference/configuration/providers.mddocs-site/src/content/docs/ru/reference/configuration/providers.mddocs-site/src/content/docs/zh-cn/reference/configuration/providers.mdsrc/images/loop.tssrc/web-search/loop.tsstructure/04_transports-and-sidecars.md
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
commit="6fb4fe26"
printf '%s\n' '=== Commit and current head ==='
git rev-parse HEAD
git show --no-patch --format='commit=%H%nparents=%P%nsubject=%s' "$commit"
printf '%s\n' '=== Changed paths ==='
git diff-tree --no-commit-id --name-status -r "$commit"
printf '%s\n' '=== Production file outlines ==='
ast-grep outline src/images/loop.ts --items all --type function,interface --match 'fetchOnce|fetchWithResetRetry|ImageBridgeDeps' --view expanded || true
ast-grep outline src/web-search/loop.ts --items all --type function,interface --match 'fetchOnce|fetchWithResetRetry|WebSearchLoopDeps' --view expanded || true
printf '%s\n' '=== Exact patch ==='
git show --format= --unified=35 "$commit" -- \
src/images/loop.ts \
src/web-search/loop.ts \
structure/04_transports-and-sidecars.md \
docs-site/src/content/docs/reference/configuration/providers.md \
docs-site/src/content/docs/ja/reference/configuration/providers.md \
docs-site/src/content/docs/ko/reference/configuration/providers.md \
docs-site/src/content/docs/ru/reference/configuration/providers.md \
docs-site/src/content/docs/zh-cn/reference/configuration/providers.md \
docs-site/src/content/docs/ko/guides/providers.md \
docs-site/src/content/docs/zh-cn/guides/providers.md \
tests
printf '%s\n' '=== Image telemetry and failover context ==='
rg -n -C 14 \
'fetchWithResetRetry|onAttemptSend|retryRecovery|recovery|fetchOnce\(|deps\.on429|key-429' \
src/images/loop.ts
printf '%s\n' '=== Web-search telemetry and failover context ==='
rg -n -C 14 \
'fetchWithResetRetry|onAttemptSend|retryRecovery|recovery|fetchOnce\(|deps\.on429|key-429' \
src/web-search/loop.ts
printf '%s\n' '=== Documentation scope, exhaustion behavior, and locale routing links ==='
rg -n -C 5 \
'retryOn429|attempts|exhaust|failover|final.error|runTurn|HTTP.capable|/ko/reference/configuration/routing/|/zh-cn/reference/configuration/routing/' \
structure/04_transports-and-sidecars.md \
docs-site/src/content/docs/reference/configuration/providers.md \
docs-site/src/content/docs/ja/reference/configuration/providers.md \
docs-site/src/content/docs/ko/reference/configuration/providers.md \
docs-site/src/content/docs/ru/reference/configuration/providers.md \
docs-site/src/content/docs/zh-cn/reference/configuration/providers.md \
docs-site/src/content/docs/ko/guides/providers.md \
docs-site/src/content/docs/zh-cn/guides/providers.md \
2>/dev/null || true
printf '%s\n' '=== Relevant tests changed by this commit ==='
git show --format= --name-only "$commit" -- tests | sed '/^$/d'Length of output: 174826
The five reported findings are addressed.
One minor documentation defect remains: The reported test results were not independently runnable in this sandbox. Source inspection confirms the implementation changes. ✅ Action performedReviews resumed. |
|
@coderabbitai resume 38278f2 fixes the punctuation defect you flagged: the en and ru retryOn429 rows now end the exhaustion clause cleanly (no double period, sentence boundary after the budget clause). |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
commit="38278f20"
printf '%s\n' '=== Commit metadata and changed paths ==='
git show --no-patch --format='commit=%H%nsubject=%s' "$commit"
git diff-tree --no-commit-id --name-status -r "$commit"
printf '%s\n' '=== Exact documentation patch ==='
git show --format= --unified=16 "$commit" -- \
docs-site/src/content/docs/reference/configuration/providers.md \
docs-site/src/content/docs/ru/reference/configuration/providers.md
printf '%s\n' '=== Final retryOn429 row contexts ==='
rg -n -C 5 'retryOn429|as-is\.\.|attempts|failover|final' \
docs-site/src/content/docs/reference/configuration/providers.md \
docs-site/src/content/docs/ru/reference/configuration/providers.mdLength of output: 41241
The punctuation finding is addressed. Normal review is resumed. ✅ Action performedReviews resumed. |
|
@lidge-jun could you take a look? The PR is ready for review: all 49 review threads are resolved, the conflicts with the latest dev (incl. #671 exact account routing) are merged, CI gates are green, and the PR is mergeable. Recent rounds addressed the bridge send-telemetry findings, the retryOn429 documentation contract (all locales), the locale link paths, and the JSDoc attachment defects. The only open pre-merge item is the CodeRabbit Docstring Coverage warning (last evaluated at 64.10% before the final doc fixes; the bot has not re-run the full check due to its review rate limit) — it is a warning, not a merge block. Happy to adjust anything you flag in your review. |
|
@lidge-jun one more ask: the full CI suite has not run on this fork PR — only the |
|
[GD] Addressed feedback feedbacks:
commit: eb9890e |
[GD] Verdict: approve-commentTLDR
Full verdictSemantic propagation
Linked: none ( UsefulnessUseful and strategically aligned with the owner review. Codex does not retry HTTP 429 client-side, so single-key providers currently fail over or surface 429 immediately. This PR adds an opt-in provider policy that waits ( Bugs / correctness
Security
Spec / standards
Reviews
Base / CI
Simplification (for the PR owner)No edits/pushes applied (foreign PR). High-confidence optional candidates only:
If neither is worth the risk, leave as-is. Nothing else was high-confidence enough to recommend. Gatenone Bottom lineApprove-comment on |
[GD] Merge readyPR: Reviews
Tip freshness
Checks
Residualoptional owner follow-ups only: continuation redaction consistency, continuation recovery-kind parity, optional shared 429 wait helper Ready to merge. |
Redact terminal-continuation provider errors, tag failover continuation sends with the matching recovery kind, and consolidate same-target 429 body-release + wait into prepareSameTarget429Wait for core, image, and web-search paths.
|
[GD] Addressed feedback feedbacks:
commit: 70c80b4 |
|
@coderabbitai resume |
✅ Action performedReviews resumed. |
[GD] Verdict: approve-commentTLDR
Full re-review after residual + simplifyUsefulnessThe PR still solves a real operational gap: Codex never retries 429s client-side, and single-key pools have no failover. Opt-in Semantic propagation
No unmapped producers/consumers found for the residual delta. Bug
SecurityRequired surfaces reviewed (secrets/config, authn, authz, logging/privacy, outbound, etc.). Residual change is privacy-positive (more redaction) and does not widen OAuth/forward/local replay. Public disclosure: no exploit detail needed. Spec + StandardsMatches the PR contract and maintainer architecture review items already fixed earlier (watchdog heartbeats, immutable same-target cache, deadline restart). Residual simplify reduces duplication without behavior change. Feedback
Checks
Bottom lineApprove and merge. |
|
Thanks @harryzhou2000 — merging this. Why it helps: Codex never retries HTTP 429 client-side and single-key pools have no failover, so providers that rate-limit would fail hard. This lands opt-in Ship it. |
3 participants
Summary
Adds an opt-in, provider-level
retryOn429policy: on HTTP 429 the proxy waits (upstreamRetry-Afteror a fixed interval, capped) and replays the identical pre-stream request on the same key before any multi-key failover.Why
retry_429=falsein codex-rs endpoint configs, surfaced as the misleading "exceeded retry limit" error (HTTP 429 is surfaced as "exceeded retry limit" without retrying; expose retry_429 or clarify error openai/codex#30471, still open; the suggested fix only rewords the message).Retry-Afterso clients that honor it (Claude Code) recover; Codex does not act on it.Behavior
providers.<name>.retryOn429 = { enabled?, attempts?, intervalMs?, maxIntervalMs?, respectRetryAfter? }(defaults: enabled=true, attempts=3, intervalMs=5000, maxIntervalMs=60000, respectRetryAfter=true). Default off when absent → zero behavior change.authMode: "key", or the documented omitted default for custom API-key providers). Fail closed: OAuth/forward credentials are never replayed on the same token; local runtimes (Ollama etc.) have no remote key to preserve; unknown/custom auth modes are rejected rather than guessed at.providerConfigSeednow preserves the registry auth kind (including"local") so the gate survives the seed round-trip.Retry-After.attempts)./v1/responses,/v1/chat/completions, and routed/v1/messages(all enterhandleResponses), plus the other key-auth surfaces that bypass that loop: the Responses passthrough wire (openai-responseskey-auth gateways, e.g. the built-in DeepSeek preset), the image/video bridge and web-search sidecar loops (before theiron429key rotation), and Anthropic terminal-guard continuations (before key/account failover).attemptsbudget.Retry-Afteror the fixed fallback — is capped atmaxIntervalMs; the schema capsmaxIntervalMsat 600000 (the effective per-wait cooldown ceiling). An already-expired HTTP-dateRetry-Afterretries immediately (likeRetry-After: 0).rate-limit-429recovery kind on replay sends; the image/video and web-search bridge loops restart their response-header deadline after each deliberate wait, so backoffs never consume the connect budget or surface as a 504.retryOn429fields with a warning instead of tripping the whole schema (which would hide every provider/key behind a default config); the management write boundary still rejects invalid policies.Files
src/types.ts—RateLimitRetryPolicy+OcxProviderConfig.retryOn429src/config.ts— zod validation (outer provider schema stays passthrough; a typo insideretryOn429degrades, never rejects the whole config),maxIntervalMs≤ 600000src/providers/key-failover.ts— policy normalization gated to key-auth + delay computation (Retry-After seconds/HTTP-date/0, capped)src/providers/derive.ts—providerConfigSeedpreserves the registry auth kind (incl."local")src/server/responses/core.ts— recovery-loop replay before the multi-key failoverwhile; passthrough-wire replay before the forward-pool logic; terminal-guard continuation replay before key/account failover; abort path cancels the unread 429 body firstsrc/images/loop.ts,src/web-search/loop.ts— same-target replay before theiron429key rotation (newretryOn429Policydep)src/usage/log.ts—AttemptRecoveryKindmemberrate-limit-429and its persisted-usage whitelist entrygui/src/pages/Logs.tsx— recovery-kind union includesrate-limit-429(and pre-existinganthropic-oauth-429)docs-siteconfiguration reference (all five locales),structure/04transport note,devlog/_plan/260802_429_same_target_retry/Test plan
bun test tests/rate-limit-retry.test.ts tests/server-rate-limit-retry-e2e.test.ts tests/usage-log.test.ts tests/key-failover.test.ts tests/retry-after-429.test.ts— 79 pass (policy/delay units incl. fail-closed auth gating, expired HTTP-date,Retry-After: 0, fallback cap; persistedrate-limit-429recovery kind; deterministic direct-handler abort test; e2e: replay-to-success with byte-identical bodies, opt-in passthrough, exhaustion, retry-before-failover ordering, key-authopenai-responsespassthrough replay with identical body/auth, per-request budget across a 2-key pool)tests/terminal-guard-server.test.ts); image bridge + web-search loop same-key replay before rotation withrate-limit-429telemetry (tests/images/loop.test.ts,tests/web-search.test.ts); config load degradation (tests/config-user-edits.test.ts); registry auth-kind preservation (tests/provider-registry-parity.test.ts)bun run typecheckbun run privacy:scancd gui && bun run lintdevin this environment (WS/auth/connection-refused) plus GUI tests that pass oncegui/deps are installedCommits
667ad08f— feature implementation2efb887b— audit round-1 fixes (usage-log whitelist, key-auth gating, budget scope, abort body cancel, schema cap, GUI union, docs)a06a4160— audit round-3 xhigh fixes: coverage extended to passthrough wire / image+web-search bridges / terminal continuations, fixed-fallback cap, local-mode gating, docs + tests58c36c9e— review-bot round: fail-closed auth (registry preserves local), expired Retry-After → immediate, release 429 bodies before backoff,rate-limit-429recovery on every surface, bridge header-deadline restart, continuation budget hoisted + upstream-signal sleep, config load degradation, identical-replay + budget regression tests, provider/adapter docs (5 locales)568e565c— review-bot round 2: awaited body-cancel before backoff, stale-deadline cleared pre-sleep + 499 re-check post-sleep, misnamed retryOn429 keys warn at load, opt-in + ordering wording in provider/adapter guides (5 locales)3c19337e— docstring coverage pass on the diff (JSDoc for seeded provider config, responses core helpers, image/web-search iteration prep, terminal-guard continuation, and test helpers)1af83c39— attach JSDoc directly to the remaining diff-touched declarations (terminal-guard continuation, image/web-search fetchOnce, cooldown check, provider-config interface, persisted-usage helper)d4e19788— review round 3: retry budget hoisted per request across bridge iterations, single dispatch-time attempt telemetry (recovery kind passed through), invalidenabledmaster switch discards the whole policy, secret-safe config warnings (path + type only)58bf694f— local audit round (gpt-5.6-terra + DeepSeek-V4-Flash): one request-wide 429 budget shared between the main recovery loop and the terminal-guard continuation; awaited 429-body cancellation before every backoff (all surfaces); post-sleep client-abort re-check before dispatching every replay; xAIx-grok-req-idpinned per logical request so same-key replays are byte-identical; new regression tests (shared continuation budget, continuation abort-during-wait, bridge budget not re-armed after rotation, far-future HTTP-date cap, stable xAI request id)4fee87b5— review round 4: unrecognizedretryOn429field NAMES are redacted before logging (secret-shaped property names become[REDACTED], ordinary typos stay readable)5945711b— review round 4 cleanup: rename the xAI request-id param topinnedRequestId(fallback only at transport resolution); anchor the secret-warning test assertion to the exact field+type diagnostice65106f0— review round 4 follow-up: JSON-escape the redacted field name in load warnings so control-character property names (newline/ANSI) cannot forge log lines89535fb7— review round 5: redact and JSON-escape the PROVIDER name in allretryOn429load warnings (sanitizer runs pre-validation, so the name is untrusted; secret-shaped names log as[REDACTED], control characters escaped)eb9890e5— maintainer architecture review round: (1) deliberate 429 backoffs now yield adapter heartbeats every min(10s, stall/2) in the terminal continuation and image/web-search loops, so a wait that outlives the bridge stall budget can never tripupstream_stall_timeout; (2) one immutable cached outbound request per same-target sequence — the main recovery loop, terminal continuation, and both sidecar loops reuse the exact URL/serialized body/headers, rebuilding only after key/account/oauth/tier changes (builder runs once per target, asserted); (3) regression tests: wait > stall budget still succeeds (all three surfaces), wait >connectTimeoutMsrestarts the header deadline (no 504), full-header equality on e2e replays4e172054— review round 6: clampsleepWithHeartbeatsstep to ≥1ms (non-positive interval can no longer spin unobserved); pin the contract that a policy degraded to{}still resolves as enabled (object presence = opt-in) with sanitizer + resolved-policy assertions02601815— review round 6 follow-up: normalize NaN heartbeat intervals to the 1ms step (NaN no longer aborts the wait after one beat); regression test asserts the full duration elapsese8613e36— mergeupstream/dev(docs overhaul restructure). Conflicts were doc-only (5 localeconfiguration.mdindexes): dev's restructured pages are kept, and theretryOn429reference row moved into the new per-localeconfiguration/providers.mdpages afterresponsesItemIdRepair. 302 targeted tests green on the merged tree.Draft — opened for review, not for merge.
Summary by CodeRabbit
New Features
Retry-After, configurable limits, cancellation, heartbeats, and failover.retryOn429configuration for API-key authentication.Bug Fixes
Documentation