Raise the Days=0 expiration budget to 420 seconds - #2477
Open
delthas wants to merge 1 commit into
Open
Conversation
Top flake of the CTST census: 7 of 30 runs on one head, all on the Versioned example. Expecting an empty bucket is correct, but the Versioned path needs three sequential conductor batches to get there (expire current, expire noncurrent, then the EODM cleanup, which only runs once no noncurrent key shows up in the same listing). Each batch waits on LifecycleConductor._controlBacklog, whose consumer-progress check reads offsets published to ZooKeeper every 60s, so a batch can sit 60-120s on a stale metric. 180s left no room for that: passing runs already spent 85-91% of the budget, and in one traced failure the third batch started 0.34s after the deadline and emptied the bucket 2.2s later. 420s covers three gated batches with slack. The single budget applies to all three examples, so a large regression on the single-batch Non versioned path would no longer be caught here. Issue: ZENKO-5337
Contributor
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
delthas
marked this pull request as ready for review
August 5, 2026 10:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The top flake of the recent CTST flakiness census: 7 of 30 runs from a single head, always the
Versionedexample ofDays=0 expiration empties a "<versioningConfiguration>" bucket.The assertion is right — backbeat does auto-clean the expired delete marker — but the
Versionedpath legitimately needs three sequential conductor batches to get there: expire the current version, expire the noncurrent one, then the EODM cleanup, which only deletes the delete marker once no noncurrent key appears in the same listing.Suspendedneeds two batches,Non versionedone, and the measured durations track that ordering exactly.Each batch waits on
LifecycleConductor._controlBacklog, whose consumer-progress check reads offsets published to ZooKeeper only every 60 s, so a batch can stall 60–120 s on a stale metric.180 s left no room for that. Passing runs were already spending 85–91 % of the budget (163.9 s in one clean run), and in one traced failure the third batch started 0.34 s after the step deadline and emptied the bucket 2.2 s after the assertion had failed. 420 s covers three gated batches with slack, and the step timeout goes to 8 minutes to stay above it.
Issue: ZENKO-5337