Skip to content

Don't spam config poller cache miss retries#2071

Draft
ogtownsend wants to merge 1 commit into
mainfrom
ogt/dont-spam-retry-source-chain-config-fetch-retries
Draft

Don't spam config poller cache miss retries#2071
ogtownsend wants to merge 1 commit into
mainfrom
ogt/dont-spam-retry-source-chain-config-fetch-retries

Conversation

@ogtownsend
Copy link
Copy Markdown
Contributor

NOPs reported excessive error log spam from processSourceChainConfigResults() when a source chain's GetSourceChainConfig RPC call fails persistently (e.g., due to an RPC misconfiguration like gas limit below intrinsic gas).

Root cause:

  • When GetAllConfigsLegacy fetches source chain configs and receives some failure, processSourceChainConfigResults logs the error and skips that chain and it's never added to the config poller cache.
  • In configPollerV2.GetOfframpSourceChainConfigs(), any chain not in the cache is treated as a "cache miss," which triggers an inline call to batchRefreshChainAndSourceConfigs.
  • Since the failing chain never gets cached, every subsequent call triggers another full batch fetch, producing the same error logs every time.
  • Two concurrent callers in the merkle root asyncObserver.sync() (ObserveOffRampNextSeqNums and ObserveLatestOnRampSeqNums) both hit this path on every tick, multiplying the log volume.

Fix:
pkg/reader/config_poller_v2.go:

  • Added attemptedSourceChains map to chainCache to track which source chains have been included in at least one fetch attempt.
  • GetOfframpSourceChainConfigs: A chain missing from the cache is only treated as a "cache miss" (triggering an inline fetch) if it has never been attempted. Chains that were previously attempted but failed are not re-fetched inline
  • The background poller (refreshAllKnownChains) will still retry them on its regular schedule.
  • batchRefreshChainAndSourceConfigs: After each fetch, all requested source chain selectors are recorded in attemptedSourceChains and sourceChainRefresh is always updated, regardless of whether individual chains succeeded.

pkg/chainaccessor/config_processors.go:

  • Downgraded the "Failed to get source chain config from result" log from Errorw to Warnw.
  • This is a per-chain partial failure that is retried by the background poller, Error level creates excessive noise

@ogtownsend ogtownsend force-pushed the ogt/dont-spam-retry-source-chain-config-fetch-retries branch from f5b6d07 to 0ca190d Compare May 15, 2026 19:41
@ogtownsend ogtownsend force-pushed the ogt/dont-spam-retry-source-chain-config-fetch-retries branch from 0ca190d to 4d05090 Compare May 15, 2026 20:21
@ogtownsend ogtownsend requested review from makramkd and winder May 15, 2026 20:23
@github-actions
Copy link
Copy Markdown

Metric ogt/dont-spam-retry-source-chain-config-fetch-retries main
Coverage 70.1% 69.7%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant