fix(replays): stop bulk delete progress stalling and rewinding - #120772
Open
JoshuaKGoldberg wants to merge 4 commits into
Open
fix(replays): stop bulk delete progress stalling and rewinding#120772JoshuaKGoldberg wants to merge 4 commits into
JoshuaKGoldberg wants to merge 4 commits into
Conversation
Task delivery is at-least-once, so a job can have more than one live call chain: a redelivered activation re-enters at the offset it was given while the successor it already enqueued keeps going. Both chains wrote the whole row via save(), so the reported count rotated between their two positions and a stale chain could revert a terminal status. Progress and status are now written per-column, the offset only moves forward, and a chain that fell behind the checkpoint exits instead of re-deleting rows. A blown processing deadline raises a BaseException, so it bypassed the task's exception handler, and the broker discards the activation once retries run out. That left the job reporting in-progress forever with nothing to advance it. It now reaches a terminal failed status on the last attempt. Deleting a batch's recordings used a fresh thread pool per replay, serializing the batch behind whichever replay had the most segments. Locally a 100-replay batch took 5.4s where one shared pool takes 0.07s, which is what pushed batches past the 600s deadline. The batch now shares one pool. Refs #120413
github-actions
Bot
added
the
Scope: Backend
Member
Author
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
The guard exited early when an activation arrived behind the checkpoint, on the assumption that another chain was already ahead of it. But the checkpoint is written before the successor is enqueued, so an activation killed in that gap came back, saw itself behind, and returned without enqueueing anything or moving the status. That stranded the job at in-progress, which is the failure this PR is meant to remove. The guard was not load-bearing: the monotonic offset write already makes the counter immune to a lagging chain, and reproducing the two-chain interleaving without the guard yields the identical sequence. A duplicate chain now just redoes idempotent work instead of being suppressed. Reported by Cursor Bugbot on #120772.
JoshuaKGoldberg
changed the title
JoshuaKGoldberg
marked this pull request as ready for review
July 28, 2026 20:28
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 04e2dba. Configure here.
| # namespace="replays") continue to resolve and execute. | ||
| alias_namespace=replays_tasks, | ||
| retry=Retry(times=5), | ||
| retry=Retry(times=5, on=(ProcessingDeadlineExceeded,)), |
Contributor
There was a problem hiding this comment.
Retry policy drops exception recovery
Medium Severity
Narrowing retry to only ProcessingDeadlineExceeded drops retries for ordinary exceptions, but _advance_offset, status completion, and the next .delay() still sit outside the try/except that marks the job failed. A transient DB or enqueue error after a successful batch can leave the job stuck in-progress with nothing left to advance it.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 04e2dba. Configure here.
Member
Author
There was a problem hiding this comment.
Errm. I think this is already the case on master? Someone please tell me if I'm wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bulk replay delete jobs showed two counter symptoms in production:
Count Deletedfrozen at a value for hours while the job still reportedin-progress: fixed by pooling recording deletes once per batch rather than per replay, and by marking the job as'failed'on the final attempt.Closes #120413.