Roll back automatically when a deploy fails its health check - #23
Merged
Conversation
On 2026-08-16 the indexing run deployed an image that never served traffic. The health check correctly detected it and failed the run, but the broken release stayed live: the site was down from 03:41 until a human noticed at 08:12. Verifying after the fact is not enough if nothing undoes the damage. Capture the currently-serving release before deploying, and restore it if the new one does not return 200 within ~5 minutes. Then re-verify the rollback itself and say plainly in the job summary whether the site came back, so a failure that needs a human is distinguishable from one that self-healed. Also stop recording LAST_PHYSLIB_SHA unless the deploy verified. It was being set unconditionally, so a failed deploy marked that PhysLib revision as done and the next run would skip it -- stranding the indexed work silently. Select the rollback target by release description rather than the slug field: container releases carry no slug, so the previous check matched nothing and would have crashed at the moment it was needed.
There was a problem hiding this comment.
Pull request overview
This PR improves the reliability of the scheduled indexing deploy by automatically rolling back to the previously serving Heroku release if the newly released container never becomes healthy, and by only marking LAST_PHYSLIB_SHA after a verified healthy deploy.
Changes:
- Record the currently-serving Heroku release (image-bearing) prior to deploying, to use as a rollback target.
- Verify the post-release health check and trigger a rollback + post-rollback re-verification if the new release stays unhealthy.
- Move
LAST_PHYSLIB_SHAupdate to occur only after a successful, verified deploy.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+343
to
+345
| echo "::error title=Deploy unhealthy — rolling back::New release never returned HTTP 200. Restoring v${{ steps.prev_release.outputs.prev }}." | ||
| heroku rollback "v${{ steps.prev_release.outputs.prev }}" --app physlibsearch | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On 2026-08-16 the indexing run deployed an image that never served traffic. The health check did detect it and failed the run — but the broken release stayed live. The site was down from 03:41 until a human noticed at 08:12 (~8.5 hours).
Verifying after the fact is worthless if nothing undoes the damage.
What
LAST_PHYSLIB_SHAafter a verified deploy. It was set unconditionally, so a failed deploy marked that PhysLib revision as indexed and the next run would skip it, silently stranding the work.slugfield. Container releases carry no slug, so my first version matched nothing and would have crashed at exactly the moment it was needed. Caught by testing against the live app.Verified
v41 (Rollback to v38)Not fixed yet
The underlying reason v39's image crash-loops while v38 (same commit, index differs by 357 vectors) runs fine. Ruled out so far: memory (no R14/R15, 148 MB), file ownership (verified writable on a live dyno as UID 19437). This PR ensures that whatever the cause, it can't leave the site down.