Don't spam config poller cache miss retries#2071
Open
ogtownsend wants to merge 2 commits into
Open
Conversation
ogtownsend
force-pushed
the
ogt/dont-spam-retry-source-chain-config-fetch-retries
branch
from
f5b6d07 to
0ca190d
Compare
ogtownsend
force-pushed
the
ogt/dont-spam-retry-source-chain-config-fetch-retries
branch
from
0ca190d to
4d05090
Compare
|
makramkd
previously approved these changes
May 18, 2026
ogtownsend
requested review from
AmrMohamedRezk,
asoliman92,
b-gopalswami,
carte7000,
chris-de-leon-cll and
matYang
winder
reviewed
May 18, 2026
Comment on lines
305
to
310
| } else if _, attempted := destChainCache.attemptedSourceChains[chain]; !attempted { | ||
| // Only treat as missing if we haven't attempted to fetch this chain before. | ||
| // Chains that were attempted but not returned are either unconfigured on-chain | ||
| // or had an RPC error. Re-fetching inline on every call creates a doom-loop | ||
| // of redundant RPC calls and error log spam. The background poller will retry. | ||
| missingChains = append(missingChains, chain) |
Contributor
There was a problem hiding this comment.
question: Would it be possible for a bad config to cause the config to disappear? In that case this cache would need to have some sort of expiry to make sure we detect it's now missing.
winder
previously approved these changes
May 18, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reduces repeated source-chain config fetches/logs when a source-chain config read fails and is omitted from the config poller cache.
Changes:
- Tracks attempted source-chain config fetches to avoid repeated inline cache-miss refreshes.
- Updates config-poller tests for failed, recovered, and newly discovered source chains.
- Downgrades per-source-chain config result failures from error to warning.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
pkg/reader/config_poller_v2.go |
Adds attempted-source-chain tracking and updates cache refresh behavior. |
pkg/reader/config_poller_v2_test.go |
Adds tests around failed source-chain inline retry suppression and background recovery. |
pkg/chainaccessor/config_processors.go |
Downgrades source-chain config result failures from error to warning. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
305
to
310
| } else if _, attempted := destChainCache.attemptedSourceChains[chain]; !attempted { | ||
| // Only treat as missing if we haven't attempted to fetch this chain before. | ||
| // Chains that were attempted but not returned are either unconfigured on-chain | ||
| // or had an RPC error. Re-fetching inline on every call creates a doom-loop | ||
| // of redundant RPC calls and error log spam. The background poller will retry. | ||
| missingChains = append(missingChains, chain) |
Comment on lines
+467
to
+468
| for _, chain := range sourceChainSelectors { | ||
| cache.attemptedSourceChains[chain] = struct{}{} |
| "GetAllConfigsLegacy", | ||
| mock.Anything, | ||
| destChain, | ||
| mock.MatchedBy(chainSelectorSliceMatcher(sourceChains))). |
Comment on lines
+81
to
83
| lggr.Warnw("Failed to get source chain config from result", | ||
| "chain", chain, | ||
| "error", err) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOPs reported excessive error log spam from
processSourceChainConfigResults()when a source chain'sGetSourceChainConfigRPC call fails persistently (e.g., due to an RPC misconfiguration like gas limit below intrinsic gas).Root cause:
GetAllConfigsLegacyfetches source chain configs and receives some failure,processSourceChainConfigResultslogs the error and skips that chain and it's never added to the config poller cache.configPollerV2.GetOfframpSourceChainConfigs(), any chain not in the cache is treated as a "cache miss," which triggers an inline call tobatchRefreshChainAndSourceConfigs.asyncObserver.sync()(ObserveOffRampNextSeqNumsandObserveLatestOnRampSeqNums) both hit this path on every tick, multiplying the log volume.Fix:
pkg/reader/config_poller_v2.go:attemptedSourceChainsmap tochainCacheto track which source chains have been included in at least one fetch attempt.GetOfframpSourceChainConfigs: A chain missing from the cache is only treated as a "cache miss" (triggering an inline fetch) if it has never been attempted. Chains that were previously attempted but failed are not re-fetched inlinerefreshAllKnownChains) will still retry them on its regular schedule.batchRefreshChainAndSourceConfigs: After each fetch, all requested source chain selectors are recorded inattemptedSourceChainsand sourceChainRefresh is always updated, regardless of whether individual chains succeeded.pkg/chainaccessor/config_processors.go:Errorlevel creates excessive noise