Skip to content

MCO-2380: MCO-2381: Expose MachineOSBuild Status Conditions on Paused MachineConfigPools and add E2E Tests for Paused Pool Builds#6282

Open
anandram2 wants to merge 1 commit into
openshift:mainfrom
anandram2:decouple-image-and-build
Open

MCO-2380: MCO-2381: Expose MachineOSBuild Status Conditions on Paused MachineConfigPools and add E2E Tests for Paused Pool Builds#6282
anandram2 wants to merge 1 commit into
openshift:mainfrom
anandram2:decouple-image-and-build

Conversation

@anandram2

@anandram2 anandram2 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

- What I did

  • Updated calculateStatus in the node controller to surface MachineOSBuild lifecycle state on paused MachineConfigPools via the Updating condition (waiting, prepared, in progress, success, and failure)
  • Added unit tests in status_test.go covering each paused-pool build phase and expected condition messages
  • Added TestPausedMCPSurfacesMachineOSBuildStatus e2e test to verify build status visibility while paused and rollout after unpause
  • Extended MachineOSBuildBuilder with WithBuildPrepared, WithBuildInProgress, and WithFailedBuild helpers for unit test fixtures
  • Added e2e helpers setMachineConfigPoolPaused and waitForMCPUpdatingCondition

- How to verify it

Run the unit tests:
go test ./pkg/controller/node/ -run TestCalculateStatusWithImageModeReporting

or

Run the e2e test (requires a cluster with OCL enabled):
go test -v ./test/e2e-ocl-2of2/ -run TestPausedMCPSurfacesMachineOSBuildStatus

Manual verification on a cluster:

  1. Pause a pool: oc patch mcp worker --type=merge -p '{"spec":{"paused":true}}'
  2. Create or update a MachineOSConfig for that pool
  3. Watch the MCP Updating condition: oc get mcp worker -o jsonpath='{.status.conditions[?(@.type=="Updating")].message}'
  4. You should see messages like Pool is paused; waiting for a new OS image build to start, ...prepared but will not rollout, ...build in progress, and ...build completed successfully as the build progresses
  5. Unpause the pool and confirm nodes roll out to the new image

- Description for the changelog
Surface MachineOSBuild status on paused MachineConfigPools and add e2e coverage

MCO-2380
MCO-2381

Summary by CodeRabbit

  • New Features

    • Improved MachineConfigPoolUpdating reporting when a MachineConfigPool is paused, surfacing distinct MachineOSBuild lifecycle stages (waiting, prepared, in progress) and clearer, stage-specific messages.
  • Bug Fixes

    • Ensures MachineConfigPoolUpdating correctly transitions to completion states (success/failure) while paused, with accurate condition status and messaging aligned to MachineOSBuild.
  • Tests

    • Expanded unit and end-to-end coverage for paused-pool scenarios, including finer MachineOSBuild lifecycle control and new helpers to set build condition states.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 8, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

@anandram2: This pull request references MCO-2380 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

- What I did

  • Updated calculateStatus in the node controller to surface MachineOSBuild lifecycle state on paused MachineConfigPools via the Updating condition (waiting, prepared, in progress, success, and failure)
  • Added unit tests in status_test.go covering each paused-pool build phase and expected condition messages
  • Added TestPausedMCPSurfacesMachineOSBuildStatus e2e test to verify build status visibility while paused and rollout after unpause
  • Extended MachineOSBuildBuilder with WithBuildPrepared, WithBuildInProgress, and WithFailedBuild helpers for unit test fixtures
  • Added e2e helpers setMachineConfigPoolPaused and waitForMCPUpdatingCondition

- How to verify it

Run the unit tests:
go test ./pkg/controller/node/ -run TestCalculateStatusWithImageModeReporting

or

Run the e2e test (requires a cluster with OCL enabled):
go test -v ./test/e2e-ocl-2of2/ -run TestPausedMCPSurfacesMachineOSBuildStatus

Manual verification on a cluster:

  1. Pause a pool: oc patch mcp worker --type=merge -p '{"spec":{"paused":true}}'
  2. Create or update a MachineOSConfig for that pool
  3. Watch the MCP Updating condition: oc get mcp worker -o jsonpath='{.status.conditions[?(@.type=="Updating")].message}'
  4. You should see messages like Pool is paused; waiting for a new OS image build to start, ...prepared but will not rollout, ...build in progress, and ...build completed successfully as the build progresses
  5. Unpause the pool and confirm nodes roll out to the new image

- Description for the changelog
Surface MachineOSBuild status on paused MachineConfigPools and add e2e coverage

MCO-2380
MCO-2381

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e2b74056-45bb-405b-a0a3-32bc8aeda8aa

📥 Commits

Reviewing files that changed from the base of the PR and between de3832d and 87498fa.

📒 Files selected for processing (4)
  • pkg/controller/node/status.go
  • pkg/controller/node/status_test.go
  • test/e2e-ocl-2of2/onclusterlayering_test.go
  • test/helpers/machineosbuildbuilder.go
🚧 Files skipped from review as they are similar to previous changes (4)
  • pkg/controller/node/status_test.go
  • test/helpers/machineosbuildbuilder.go
  • test/e2e-ocl-2of2/onclusterlayering_test.go
  • pkg/controller/node/status.go

Walkthrough

The node status controller now reports MachineOSBuild progress through MachineConfigPoolUpdating for paused pools, with separate handling for prepared, building, failed, succeeded, and missing-build states. Unit tests, helper builders, and an end-to-end test were updated to cover the paused-pool flow.

Changes

Paused pool MachineOSBuild status reporting

Layer / File(s) Summary
Controller status handling
pkg/controller/node/status.go
Separates prepared builds from building builds and adds paused-pool handling that maps MachineOSBuild state to MachineConfigPoolUpdating status and message.
Unit test inputs
pkg/controller/node/status_test.go
Adds per-case MachineOSBuild injection so paused-pool scenarios can supply specific build objects or nil.
Unit test cases
pkg/controller/node/status_test.go
Adds paused-pool cases covering missing, prepared, building, failed, and succeeded states with exact status and message assertions.
E2E paused-pool flow
test/helpers/machineosbuildbuilder.go, test/e2e-ocl-2of2/onclusterlayering_test.go
Adds MachineOSBuild status builders plus a paused MCP end-to-end test and helpers that verify the reported condition and final image rollout.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant E2ETest
  participant MachineConfigPool
  participant MachineOSBuildBuilder
  participant NodeImage
  E2ETest->>MachineConfigPool: pause pool
  E2ETest->>MachineOSBuildBuilder: create prepared/building/failed/successful build states
  E2ETest->>MachineConfigPool: poll MachineConfigPoolUpdating while paused
  E2ETest->>MachineConfigPool: unpause pool
  MachineConfigPool->>NodeImage: apply completed build image
Loading
🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Microshift Test Compatibility ⚠️ Warning New e2e test uses MachineConfigPool/MachineOSConfig/MachineOSBuild APIs, and no MicroShift skip/tag/guard was found. Add an [apigroup:machineconfiguration.openshift.io] or [Skipped:MicroShift] tag, or guard the test with exutil.IsMicroShiftCluster().
Ipv6 And Disconnected Network Test Compatibility ⚠️ Warning TestPausedMCPSurfacesMachineOSBuildStatus uses GetCowsayDockerfileForCluster, whose Dockerfiles start from quay.io/centos images, so it depends on public registry access. Use an internal/mirrored base image or cluster image stream for the Dockerfile, or skip the test on disconnected clusters.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: surfacing MachineOSBuild status on paused MachineConfigPools and adding E2E coverage.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The new and existing test titles are static and deterministic; no dynamic identifiers or generated values appear in test names.
Test Structure And Quality ✅ Passed New tests use t.Cleanup and existing cleanup helpers, explicit timeouts on all waits, and assertion style consistent with nearby e2e tests.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The new e2e test only uses one worker node and checks its own image change; it has no multi-node/HA assumptions or SNO skip requirement.
Topology-Aware Scheduling Compatibility ✅ Passed Changes only update MCP status/messages and tests; no pod affinity, replica, nodeSelector, toleration, or topology-based scheduling logic was added.
Ote Binary Stdout Contract ✅ Passed PASS: the PR adds only flag init and test helpers; no new fmt.Print/println, os.Stdout, or logging-to-stdout in process-level code appears in the touched files.
No-Weak-Crypto ✅ Passed No MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret/token comparisons were introduced in the touched files.
Container-Privileges ✅ Passed The PR only changes Go controller/tests; no added or modified manifests or container specs contain privileged, hostPID/Network/IPC, SYS_ADMIN, or allowPrivilegeEscalation settings.
No-Sensitive-Data-In-Logs ✅ Passed New logs only report MCP/MOSC/MOSB names and paused/build state; no passwords, tokens, PII, or other sensitive data are logged.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: anandram2
Once this PR has been reviewed and has the lgtm label, please assign proietfb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot requested review from djoshy and proietfb July 8, 2026 13:52
@anandram2

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
test/helpers/machineosbuildbuilder.go (1)

107-141: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use apihelpers condition helpers here too. MachineOSBuildRunningConditions() and MachineOSBuildFailedConditions() already encode these slices, so switching WithBuildInProgress() and WithFailedBuild() to them avoids condition drift and keeps the builder consistent with WithBuildPrepared().

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/helpers/machineosbuildbuilder.go` around lines 107 - 141, The manual
condition slice in WithBuildInProgress is duplicating MachineOSBuild status
logic and can drift from the shared helpers. Update WithBuildInProgress to use
the existing apihelpers condition helper, matching the pattern already used by
WithBuildPrepared and the failed-state builder, so the MachineOSBuildBuilder
stays consistent with MachineOSBuildRunningConditions and
MachineOSBuildFailedConditions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controller/node/status.go`:
- Around line 286-287: The switch in node status handling has an empty
IsBuildPrepared() branch, which causes the non-paused pool path to skip setting
the Updating condition; update the logic in the status computation around the
MOSB state switch so the IsBuildPrepared() case uses the same build-specific
Updating message as the building path, and remove the now-dead IsBuildPrepared()
check from the following IsBuilding() case. Also add a unit test covering the
non-paused branch when the MOSB is in the prepared state to verify the
build-specific condition is returned.

---

Nitpick comments:
In `@test/helpers/machineosbuildbuilder.go`:
- Around line 107-141: The manual condition slice in WithBuildInProgress is
duplicating MachineOSBuild status logic and can drift from the shared helpers.
Update WithBuildInProgress to use the existing apihelpers condition helper,
matching the pattern already used by WithBuildPrepared and the failed-state
builder, so the MachineOSBuildBuilder stays consistent with
MachineOSBuildRunningConditions and MachineOSBuildFailedConditions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e610e29c-b093-4ea2-bbc1-bb7b5aba4ffb

📥 Commits

Reviewing files that changed from the base of the PR and between f2f78c2 and fb16efd.

📒 Files selected for processing (4)
  • pkg/controller/node/status.go
  • pkg/controller/node/status_test.go
  • test/e2e-ocl-2of2/onclusterlayering_test.go
  • test/helpers/machineosbuildbuilder.go

Comment thread pkg/controller/node/status.go Outdated
@anandram2 anandram2 force-pushed the decouple-image-and-build branch from de3832d to 87498fa Compare July 9, 2026 13:54
@openshift-ci

openshift-ci Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

@anandram2: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants