Skip to content

Poll the cascade settle step and fix its PENDING diagnostics - #2480

Draft
delthas wants to merge 3 commits into
development/2.16from
improvement/ZENKO-5340/cascade-settle-poll-and-diagnostics
Draft

Poll the cascade settle step and fix its PENDING diagnostics#2480
delthas wants to merge 3 commits into
development/2.16from
improvement/ZENKO-5340/cascade-settle-poll-and-diagnostics

Conversation

@delthas

@delthas delthas commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Two defects in steps/crrCascade.ts found by the recent CTST flakiness census. They ship together because they are in the same file and both come from the same misreading of what PENDING means for a cascade destination.

The settle step did not poll (1 of 30 runs). It waited a fixed When I wait 15 seconds and then asserted once. In the failing run, CR churn stretched the cascade hops to 94 s and 168 s, and location C became REPLICA 21 s after the assertion had already failed.

This was never a capacity problem — a healthy cascade settles in about 20 s and the sibling "replicate to location" steps get 300 s, roughly 15x headroom. It was simply a one-shot assertion in a suite where every comparable step polls. The step now reads should be settled within {int} seconds, matching how every other step in this file is written, takes 120 s from the scenario (6x the healthy case), and polls until settled. The four blind 15 s sleeps go away with it, reclaiming a minute of pure sleep per run.

The failure message was wrong. On timeout the replicate step said object ... not found. In the run investigated, cloudserver had returned 200 to 184 HEAD requests on the object it called missing: the object was present and PENDING. That message sent an investigation down the wrong path for most of a day, which is why it is worth fixing even though it changes no pass/fail outcome. It now reports the last observed ReplicationStatus, and still says not found when the HEAD genuinely never succeeded.

While checking that, a related problem showed up in a sibling: both the replicate step and the tag step declared a cucumber timeout of exactly the budget the feature passes them ({ timeout: 300_000 } against within 300 seconds), so their own assert.fail always lost the race against the framework timeout and never printed. Both now get 330 s. This is slightly beyond the ticket's letter, but it is the same defect in the same file and leaving it would keep one of the two diagnostics unprintable.

Verified locally: yarn build:cucumber clean, yarn lint clean (38 pre-existing warnings, 0 errors), and a --dry-run over the cascade tags resolves the renamed step with no undefined steps across all 6 scenarios.

The product cause of dropped replication status entries — service-account key rotation on every overlay write, and a transient 403 turned into permanent loss — is tracked in the issues linked from ZENKO-5340 and is out of scope here.

Issue: ZENKO-5340

@bert-e

bert-e commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request TBA
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@scality scality deleted a comment from bert-e Aug 4, 2026
@bert-e

bert-e commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

delthas added 3 commits August 5, 2026 11:53
The step waited a fixed 15 seconds and then asserted once. In one census
run, CR churn stretched the cascade hops to 94s and 168s and location C
became REPLICA 21s after the assertion had already failed.

A healthy cascade settles in about 20s and the sibling steps in this file
get 300s, so this was never a capacity problem: it was a one-shot
assertion in a suite where every comparable step polls. Take the budget
from the scenario, like those siblings, and poll until settled. The four
blind 15-second sleeps go away with it.

Issue: ZENKO-5340
Both polling steps declared a cucumber timeout of exactly the budget the
feature passes them: { timeout: 300_000 } against "within 300 seconds".
Their own assert.fail therefore always lost the race against the
framework timeout, so on a real timeout the failure was reported as the
generic "function timed out" and the step's own diagnostic never printed.

Give them 330s so the assertion inside the step is the one that fires.

Issue: ZENKO-5340
…sing

On timeout the step failed with "object ... not found". In the run
investigated, cloudserver had returned 200 to 184 HEAD requests on the
object it called missing: it was present and PENDING. That message sent
an investigation down the wrong path for most of a day.

Track the last status the poll actually saw and report it, keeping the
"not found" wording only for the case where no HEAD ever succeeded.

Issue: ZENKO-5340
@delthas
delthas force-pushed the improvement/ZENKO-5340/cascade-settle-poll-and-diagnostics branch from ff7267d to b91090b Compare August 5, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants