Skip to content

ZSTAC-86444 backport external-storage split-brain guard#4438

Closed
MatheMatrix wants to merge 1 commit into
5.4.10from
sync/yaohua.wu/bugfix/ZSTAC-86444-5.4.10
Closed

ZSTAC-86444 backport external-storage split-brain guard#4438
MatheMatrix wants to merge 1 commit into
5.4.10from
sync/yaohua.wu/bugfix/ZSTAC-86444-5.4.10

Conversation

@MatheMatrix

Copy link
Copy Markdown
Owner

Root cause:

VM start on a new host could continue after the old storage client failed to deactivate or silently remained attached. On xinfini Vhost volumes this can leave the old QEMU and new QEMU using the same bdev.

Fix:

  • Report xinfini deactivate and bdev-delete failures instead of swallowing them.
  • Verify the bdev is really deleted after deletion polling.
  • Re-query active clients after deactivate reports success and fall back to blacklist when the old client is still active.
  • Make PrimaryStorageNodeSvc.blacklist synchronous so blacklist failures propagate out of beforeStartVmOnKvm and abort VM start.
  • Adapt xinfini, expon and zbs blacklist implementations to the new signature.

Verification:

  • git diff --check
  • mvn -pl header,plugin/externalStorage,plugin/xinfini,plugin/expon,plugin/zbs -am -DskipTests compile

Notes:

  • Backported behavior from 2ad18a5ad1, 57fe48cd27, e2d4f68bc4.
  • Source integration test changes were not forced into this MR because the target 5.4.10 test file layout differs and the pick conflicts as modify/delete.

Jira: ZSTAC-86444
Related: ZSTAC-85124

sync from gitlab !10399

Backport external-storage split-brain guards to 5.4.10.

1. Why?
Starting a VM on a new host may continue even when the old storage
client still holds the volume. For xinfini Vhost volumes this can leave
two QEMU processes using the same bdev.

2. How?
Report xinfini deactivate failures instead of swallowing them, verify
the bdev is really deleted, re-check active clients after deactivate
success, and make blacklist failures propagate out of beforeStartVmOnKvm.

3. Side effects?
Deactivate failure or a still-active old client now aborts VM start for
that volume instead of silently continuing. Happy-path deactivate keeps
the previous behavior.

# Summary of changes (by module):
- header: make PrimaryStorageNodeSvc.blacklist synchronous.
- externalStorage: post-check active clients and propagate blacklist failures.
- xinfini: surface deactivate/delete failures and abort unsupported isolation.
- expon/zbs: adapt blacklist implementations to the synchronous signature.

Related: ZSTAC-86444
Related: ZSTAC-85124
Change-Id: I59812c6d68eec6533e48a8c9cf05b7c55d6b85eb
@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 41 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: fb2f0c8d-b29b-407c-b00f-ad778c36874c

📥 Commits

Reviewing files that changed from the base of the PR and between 08177ca and d2b599d.

📒 Files selected for processing (6)
  • header/src/main/java/org/zstack/header/storage/addon/primary/PrimaryStorageNodeSvc.java
  • plugin/expon/src/main/java/org/zstack/expon/ExponStorageController.java
  • plugin/externalStorage/src/main/java/org/zstack/externalStorage/primary/kvm/ExternalPrimaryStorageKvmFactory.java
  • plugin/xinfini/src/main/java/org/zstack/xinfini/XInfiniApiHelper.java
  • plugin/xinfini/src/main/java/org/zstack/xinfini/XInfiniStorageController.java
  • plugin/zbs/src/main/java/org/zstack/storage/zbs/ZbsStorageController.java

Warning

.coderabbit.yaml has a parsing error

The CodeRabbit configuration file in this repository has a parsing error and default settings were used instead. Please fix the error(s) in the configuration file. You can initialize chat with CodeRabbit to get help with the configuration file.

💥 Parsing errors (1)
Could not fetch remote config from http://open.zstack.ai:20001/code-reviews/zstack-cloud.yaml: TimeoutError: The operation was aborted due to timeout
⚙️ Configuration instructions
  • Please see the configuration documentation for more information.
  • You can also validate your configuration using the online YAML validator.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sync/yaohua.wu/bugfix/ZSTAC-86444-5.4.10

Comment @coderabbitai help to get the list of available commands.

@MatheMatrix MatheMatrix closed this Jul 6, 2026
@MatheMatrix MatheMatrix deleted the sync/yaohua.wu/bugfix/ZSTAC-86444-5.4.10 branch July 6, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants