fix: Handling of site connection issues during outage by rippyboii · Pull Request #3470 · python-discord/bot

rippyboii · 2026-03-01T16:04:01Z

Summary

This PR improves bot startup reliability and moderator visibility when extensions/cogs fail to load.

Aggregate extension + cog load failures during startup and report them as a single alert in #mod-log.
Add retry + exponential backoff for cogs that depend on external sites/APIs (e.g., temporary 5xx/429/timeouts), with clear mod notifications on final failure.
Add unit tests to validate retry behavior, error classification, and startup reporting.

During setup_hook, extensions are loaded concurrently. When an extension/cog fails due to a transient outage (rate limits, 5xx, timeouts), failures can either:

stop startup unexpectedly, or
fail noisily/fragmentedly, making it hard to see what broke and why.

This change standardizes both resilience (retry when appropriate) and visibility (one clean startup report + targeted alerts).

Changes

1) Startup failure aggregation (single `#mod-log` alert)

Added utils/startup_reporting.py to format a standardized startup failure message.
Updated bot.py to:
- collect extension + cog load failures (import/setup/add_cog)
- wait for all load tasks to complete
- send one aggregated alert summarizing all failures
Reporting is defensive: it does not crash if the log channel is unavailable.
Startup continues for non-critical failures.

2) Retry + backoff for external/API-dependent cogs

Implemented retry logic with exponential backoff and explicit “retriable vs non-retriable” classification, plus moderator notifications when retries are exhausted.

Covered cogs include:

Filtering: 3 attempts with backoff (1s, 2s, 4s); retries on HTTP 429, HTTP 5xx, TimeoutError, OSError; final failure logs + alerts #mod-alerts.
Reminders: retry count is configured via URLs.connect_max_retries; warns are logged to Sentry; final failure posts to #mod-log.
PythonNews: retries on 408, 429, 5xx, TimeoutError, OSError; on max retries logs + alerts mod_alerts and re-raises to stop startup.
Superstarify cogs: added retry + notification and corresponding tests.

Tests / Verification

Added unit tests covering:
- retry-then-success
- max-retries then alert + failure behavior
- non-retriable errors
- retry classification logic
- aggregated startup failure reporting for faulty extensions/cogs

Suggested checks:

uv run task test
Run the bot and simulate a faulty extension/cog load to confirm a single aggregated #mod-log startup alert.

Moderator Alert in Discord:

Closes #2918

jb3 · 2026-03-01T19:45:17Z

bot/exts/filtering/filtering.py

+
+        return isinstance(error, (TimeoutError, OSError))
+
+    async def _alert_mods_filter_load_failure(self, error: Exception, attempts: int) -> None:


This method is never actually getting called.

Hi, Thank you for mentioning it.

Initially we alerted mods directly via the individual cogs call. However, later when we refactored to centralize the process during the startup period, we missed to refactor it.

This unused method is removed

jb3 · 2026-03-01T19:46:32Z

bot/exts/filtering/filtering.py

+            await mod_alerts_channel.send(
+                ":warning: Filtering failed to load filter lists during startup "
+                f"after {attempts} attempt(s). Error: `{error_details}`"
+            )


This should ping Moderators.

When there is any failure in startup process, including the cog load. Now it is all collected directly by StartupFailureReporter in startup_reporting.py, and it mentions the moderator when reporting the error

jb3 · 2026-03-01T19:48:05Z

bot/exts/info/python_news.py

+MAX_ATTEMPTS = constants.URLs.connect_max_retries
+INITIAL_BACKOFF_SECONDS = 1


These should be added to constants and instead of being redefined as constants at the top of the file usages should just use the imported class (e.g. do from bot.constants import URLs and then use URLs.connect_max_retries as an argument)

Thank you for your suggestion,

We have fixed this issue!

jb3 · 2026-03-01T19:48:16Z

bot/exts/filtering/filtering.py

+FILTER_LOAD_MAX_ATTEMPTS = constants.URLs.connect_max_retries
+INITIAL_BACKOFF_SECONDS = 1


Move both to constants and reference the constants.

Thank you for the suggestion!

We have fixed this issue!

jb3 · 2026-03-01T19:48:45Z

bot/exts/filtering/filtering.py

+    @staticmethod
+    def _retryable_filter_load_error(error: Exception) -> bool:
+        """Return whether loading filter lists failed due to some temporary error, thus retrying could help."""
+        if isinstance(error, ResponseCodeError):
+            return error.status in (408, 429) or error.status >= 500
+
+        return isinstance(error, (TimeoutError, OSError))


This should be made a generic utility and moved to a separate file for all backoff-compatible cogs to use.

Hi, Thank you for your suggestion.

We have now added new file in utility for this retry mechanism and is reflected across the code

jb3 · 2026-03-01T19:57:19Z

bot/utils/startup_reporting.py

+if TYPE_CHECKING:
+    from bot.bot import Bot
+
+@dataclass(frozen=True)


Why is this a dataclass?

Hi,

This issue has been fixed!

jb3 · 2026-03-01T19:58:41Z

bot/utils/startup_reporting.py

+        for k in keys:
+            e = failures[k]


Use properly named variables.

Hi, thank you for mentioning it,

we have fixed this one too

jb3 · 2026-03-01T20:00:56Z

bot/utils/startup_reporting.py

+
+        return textwrap.dedent(f"""
+        Failed items:
+        {chr(10).join(lines)}


Not the way to do this in so many ways, you can just do "\n".join(lines)

Can do something like:

await ctx.send( "Failed items:\n" + "\n".join(lines) )

Hi, we have fixed this issue!

jb3 · 2026-03-01T20:07:14Z

bot/bot.py

+_current_extension: contextvars.ContextVar[str | None] = contextvars.ContextVar(
+    "current_extension", default=None
+)


I'm not exactly sure why you need a context var to implement this, it seems like unnecessary overcomplication and just makes things harder to follow.

Hi, thank you for mentioning it

We have fixed this issue, bot.py no longer uses contextvars or _content_extensions at all.

jb3 · 2026-03-01T20:08:50Z

tests/bot/exts/info/test_python_news.py

+    def test_retryable_python_news_load_error(self):
+        """`_retryable_site_load_error` should classify temporary failures as retryable."""
+        test_cases = (
+            (ResponseCodeError(MagicMock(status=408)), True),
+            (ResponseCodeError(MagicMock(status=429)), True),
+            (ResponseCodeError(MagicMock(status=500)), True),
+            (ResponseCodeError(MagicMock(status=503)), True),
+            (ResponseCodeError(MagicMock(status=400)), False),
+            (ResponseCodeError(MagicMock(status=404)), False),
+            (TimeoutError("timeout"), True),
+            (OSError("os error"), True),
+            (AttributeError("attr"), False),
+            (ValueError("value"), False),
+        )


These tests should all be moved to a general utility (like the retriable error method in each of the cogs as mentioned).

Hi Thank you for mentioning this,

I have moved this one to it's respective test file to follow the structure, following from the utility in main.

rippyboii · 2026-03-02T23:44:59Z

Hi @jb3 , Thank you so much for bringing out the suggestions. We have followed your guidance and implemented the refactor to most of it. Could you please review it once again in your free time?

Thank you so much

rippyboii requested a review from mbaruh as a code owner March 1, 2026 16:04

jb3 requested changes Mar 1, 2026

View reviewed changes

rippyboii requested a review from jb3 March 2, 2026 23:43

rippyboii closed this Mar 24, 2026

rippyboii force-pushed the main branch from 988666a to b62ad1f Compare March 24, 2026 13:43

rippyboii mentioned this pull request Mar 24, 2026

fix: Handling of site connection issues during outage #3482

Open


		return isinstance(error, (TimeoutError, OSError))

		async def _alert_mods_filter_load_failure(self, error: Exception, attempts: int) -> None:

		MAX_ATTEMPTS = constants.URLs.connect_max_retries
		INITIAL_BACKOFF_SECONDS = 1

		FILTER_LOAD_MAX_ATTEMPTS = constants.URLs.connect_max_retries
		INITIAL_BACKOFF_SECONDS = 1

Uh oh!

Conversation

rippyboii commented Mar 1, 2026

Summary

Changes

1) Startup failure aggregation (single #mod-log alert)

2) Retry + backoff for external/API-dependent cogs

Tests / Verification

Moderator Alert in Discord:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rippyboii commented Mar 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

1) Startup failure aggregation (single `#mod-log` alert)