Skip to content

Resynchronise count-items in the Quotas restore scenarios - #2478

Draft
delthas wants to merge 2 commits into
development/2.16from
improvement/ZENKO-5338/quotas-restore-count-items-resync
Draft

Resynchronise count-items in the Quotas restore scenarios#2478
delthas wants to merge 2 commits into
development/2.16from
improvement/ZENKO-5338/quotas-restore-count-items-resync

Conversation

@delthas

@delthas delthas commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Two quota restore scenarios fail when the hourly count-items cron snapshots at the wrong moment — 2 of 30 runs in the recent CTST census, surfacing as a 429 behind the unhelpful Expected values to be strictly equal.

Both scenarios upload the object and transition it to e2e-cold before setting any quota, and cloudserver skips utilization accounting entirely while no quota exists. So a cron snapshot taken in that window records the object as still hot, nothing corrects it, and the later restore reserves another 100 B on top: 200 > 101, rejected. Of 10 census reports analysed, the 2 whose scenario window crossed the top of the hour failed and the other 8 passed.

First commit — add the resync step this file already uses twice (:122, :158) after the transition, so the accounting describes a cold object before the quotas go in. No new step definition.

Second commit — pay for it. The resync was green on all three CI attempts but cost ~23 min of ctst wall time (135 min against 112 on a sibling branch), and almost none of that is work: count-items keeps running after counting until Prometheus scrapes its final metrics, and the operator gives its ServiceMonitor a 60 s interval (confirmed on a live cluster), so each run idles up to a minute. The test helper also serialises these runs behind a lock, so waiting workers idle too. Scraping that one target every 5 s caps the tail — spec.s3utils.metrics has a single consumer in the operator, so no other component changes cadence, and production keeps its 60 s.

Considered and rejected: disabling the cron in CI. It would make the flake impossible, but that interleaving is what production does, and it is what surfaced the real accounting gaps in the first place.

Quotas.feature is byte-identical on development/2.14, 2.15 and 2.16, so this applies unchanged if we ever want it on the older lines.

Issue: ZENKO-5338

Both restore scenarios upload the object and transition it to e2e-cold
before any quota is set, and cloudserver skips utilization accounting
entirely when no quota is set, so neither the PUT nor the hot-to-cold
transition is visible to scuba. When the hourly count-items cron
snapshots during that window it records the object as still hot, nothing
corrects it, and the later restore reserves another 100 B: 200 > 101 and
the request is rejected where the scenario expects success.

2 of 30 census runs failed this way, and they are exactly the 2 whose
scenario window crossed the top of the hour; the 8 other analysed runs
all passed.

Rebuild the metrics after the transition with the step this file already
uses twice, so the accounting describes a cold object before the quotas
go in.

Issue: ZENKO-5338
@bert-e

bert-e commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request TBA
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@scality scality deleted a comment from bert-e Aug 4, 2026
@bert-e

bert-e commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The count-items cronjob keeps running after the count completes, until
either Prometheus scrapes its final metrics or the dwell expires
(PROMETHEUS_POLLING_ATTEMPTS x PROMETHEUS_POLLING_PERIOD, 5 x 30s in
s3utils). The operator puts a 60s interval on its ServiceMonitor by
default, confirmed on a live cluster, so each run spends up to a minute
idle after a few seconds of actual work.

The suite triggers that job explicitly, once per metrics scenario setup
and now also in the two restore scenarios. Those additions cost about 23
minutes of ctst wall time, 135 min against 112 min on a sibling branch;
the excess over the sum of the job durations comes from the /tmp lock
that serialises them, which makes waiting workers idle too.

Scrape that one target every 5s in CI instead. spec.s3utils.metrics has
a single consumer in the operator, the count-items ServiceMonitor, so
nothing else changes cadence, and no behaviour under test is affected:
the job still publishes its metrics, it just stops waiting to be asked.

Issue: ZENKO-5338
@delthas
delthas force-pushed the improvement/ZENKO-5338/quotas-restore-count-items-resync branch from da0e42d to 6a9322e Compare August 5, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants