Skip to content

✨ S3UTILS-230 implement CRR multi-site V2 replication format#396

Open
DarkIsDude wants to merge 14 commits into
development/1from
feature/S3UTILS-230/multi-crr
Open

✨ S3UTILS-230 implement CRR multi-site V2 replication format#396
DarkIsDude wants to merge 14 commits into
development/1from
feature/S3UTILS-230/multi-crr

Conversation

@DarkIsDude

@DarkIsDude DarkIsDude commented May 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add V2 replication format support to `ReplicationStatusUpdater`, implementing the CRR to Multiple Sites design
  • V2 configs (rules with a `Filter` element) route through a new `_markObjectPendingV2` path that writes per-backend `destination`/`role` and removes legacy top-level `storageClass`/`storageType`/`destination`
  • V1 configs (no `Filter` on any rule) retain existing behavior unchanged
  • Arsenal static methods (`resolveBackends`, `resolveSourceRole`, `resolveDestinationRole`) are reused instead of maintaining parallel implementations; arsenal dependency pinned to `improvement/ARSN-571/crr-multi`

Changes

`CRR/ReplicationStatusUpdater.js`

  • `_isV1Format()` — V2 detection aligned with arsenal: a rule is V2 if it carries a `Filter` element
  • `_getMatchingRules()` — prefix-matches object key against enabled rules, deduplicates by site (highest priority wins); feeds the adapter for `resolveBackends`
  • `_computeTopLevelStatus()` — aggregates backend statuses: FAILED > PROCESSING > COMPLETED
  • `_removeV1Fields()` — strips legacy top-level `storageClass`/`storageType`/`destination` when migrating to V2
  • `_markObjectPendingV2()` — V2 metadata writer: adapts AWS SDK rule shape to arsenal's `ReplicationConfigurationMetadata`, calls `ReplicationConfiguration.resolveBackends` for prefix matching / priority dedup / per-backend destination+role stamping, then calls `resolveSourceRole` for the top-level role

`package.json` — arsenal pinned to `improvement/ARSN-571/crr-multi` for the new static methods

`tests/utils/crr.js` — V2 bucket replication config fixture

`tests/unit/CRR/ReplicationStatusUpdater.js` — 7 new V2 test cases (single match, dual-rule prefix overlap, no-match skip, SITE_NAME filter, all-up-to-date skip, PROCESSING aggregate status, forceUsingConfiguration)

Test plan

  • All 456 unit tests pass (`yarn test:unit`)
  • Lint clean (`yarn lint`)
  • V2 metadata written with per-backend `destination`/`role` and no legacy top-level fields
  • V1 configs produce identical output to before this change

Issue: S3UTILS-230

⚠️ The procedure at this page seems not up to date : https://scality.atlassian.net/wiki/spaces/TS/pages/2312536220/Ingestion+Replication+XDM#Step-one---Clear-the-replication-queue. The ENDPOINT should now be http://.... with the port :80. Should we ping CS or update the page directly ?

@DarkIsDude DarkIsDude self-assigned this May 28, 2026
@bert-e

This comment was marked as resolved.

@bert-e

This comment was marked as resolved.

Comment thread tests/unit/CRR/ReplicationStatusUpdater.js Outdated
@claude

This comment was marked as resolved.

@codecov

codecov Bot commented May 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.08911% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.31%. Comparing base (eb9592d) to head (b781b84).
⚠️ Report is 9 commits behind head on development/1.

Files with missing lines Patch % Lines
CRR/ReplicationStatusUpdater.js 91.08% 5 Missing and 4 partials ⚠️
Additional details and impacted files
@@                Coverage Diff                @@
##           development/1     #396      +/-   ##
=================================================
+ Coverage          44.90%   45.31%   +0.41%     
=================================================
  Files                 88       88              
  Lines               6456     6516      +60     
  Branches            1352     1369      +17     
=================================================
+ Hits                2899     2953      +54     
- Misses              3511     3513       +2     
- Partials              46       50       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@DarkIsDude DarkIsDude force-pushed the feature/S3UTILS-230/multi-crr branch from d5490b8 to f9fca0c Compare June 1, 2026 20:17
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@claude

This comment was marked as resolved.

@claude

This comment was marked as resolved.

@DarkIsDude DarkIsDude requested review from a team, delthas and maeldonn June 2, 2026 14:13
@bert-e

bert-e commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@maeldonn maeldonn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be simplified a lot by using arsenal logic

Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@bert-e

bert-e commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following reviewers are expecting changes from the author, or must review again:

Comment thread package.json
@claude

This comment was marked as resolved.

Comment thread package.json Outdated
@claude

This comment was marked as resolved.

@DarkIsDude DarkIsDude requested a review from maeldonn June 8, 2026 17:40
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@claude

This comment was marked as resolved.

Comment thread package.json Outdated
@claude

This comment was marked as resolved.

@DarkIsDude DarkIsDude force-pushed the feature/S3UTILS-230/multi-crr branch from 2649e03 to c03ea05 Compare June 15, 2026 13:47
Comment thread package.json Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@claude

This comment was marked as resolved.

Comment thread package.json Outdated
@claude

This comment was marked as resolved.

@maeldonn maeldonn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One final batch of reviews and we should be good to go 💪

Comment thread CRR/ReplicationStatusUpdater.js Outdated
*/
_isV1Format(repConfig) {
return repConfig.Rules.every(r => !r.Filter);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used once, could be inlined to simplify code reading.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was intended here. As the function is waterflow, with old style and very long, I specifically tried to split it. If it's a hard no for you, I can change 🙏

Comment on lines +200 to +204
_computeTopLevelStatus(backends) {
if (backends.some(b => b.status === 'FAILED')) { return 'FAILED'; }
if (backends.some(b => b.status === 'PENDING')) { return 'PROCESSING'; }
return 'COMPLETED';
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is only used once, it could be inlined. While extracting logic to an external function usually improves readability, creating small functions for single-use cases can actually make the code harder to follow

Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread package.json Outdated
@claude

This comment was marked as resolved.

Comment thread package.json Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@claude

This comment was marked as resolved.

Comment thread package.json Outdated
@claude

This comment was marked as resolved.

@DarkIsDude DarkIsDude requested a review from maeldonn June 17, 2026 13:14
@DarkIsDude DarkIsDude force-pushed the feature/S3UTILS-230/multi-crr branch from d1e9317 to dcbdaa0 Compare June 17, 2026 14:28
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants