Skip to content

azl4: rollback and servicing pipeline support#698

Draft
bfjelds wants to merge 7 commits into
user/bfjelds/azl4-2-imagesfrom
user/bfjelds/azl4-3-pipelines
Draft

azl4: rollback and servicing pipeline support#698
bfjelds wants to merge 7 commits into
user/bfjelds/azl4-2-imagesfrom
user/bfjelds/azl4-3-pipelines

Conversation

@bfjelds

@bfjelds bfjelds commented Jun 24, 2026

Copy link
Copy Markdown
Member

Summary

Part 3 of the stacked azl4 series. Builds on the test-image PR. Wires the rollback and servicing pipelines to build and exercise azl4 grub test images.

  • Shared common_tasks templates:
    • download-azl4-base-vhdx.yml downloads the azl4 base VHDX (azl4_qemu_guest) and stages the AZL4 netplan/systemd override RPMs (netplan 1.2.1) into artifacts/rpm-overrides so the image builder always picks them up.
    • prepare-testimage-requirements.yml moves the base VHDX, RPM overrides, and Trident RPMs into the locations the builder expects. Optional SSH key staging (stageSshKeys, default false) stages only the public key (id_rsa.pub) into the testimage tree; the private key is never baked into the image.
  • Refactored build-image and trident-testimg templates to use the shared tasks. For AZL4 preview, build-image-template.yml fails fast if the resolved base image type is not azl4_qemu_guest.
  • Rollback/servicing testing templates build and exercise the azl4 grub test images. netplan runtime testing remains enabled on grubazl4 (the earlier netplan/systemd incompatibility is fixed by the netplan 1.2.1 override RPM). Secure boot is skipped for grubazl4 because the azl4 base image does not support it yet.

Review note: only the third commit is new here.

@bfjelds bfjelds force-pushed the user/bfjelds/azl4-3-pipelines branch from d659796 to 795f9dd Compare June 24, 2026 23:47
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-2-images branch from 084346c to b76b575 Compare June 24, 2026 23:47
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-3-pipelines branch from 795f9dd to baaed75 Compare June 25, 2026 00:25
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-2-images branch from b76b575 to 490d136 Compare June 25, 2026 00:25
@bfjelds bfjelds requested a review from Copilot June 26, 2026 21:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the Azure DevOps pipeline templates to support Azure Linux 4 (“4.0-preview”) test images and wires servicing + rollback VM test stages to build and exercise the new grubazl4 flavor, reusing shared “common_tasks” templates for base-image download and testimage prerequisite staging.

Changes:

  • Add AZL4 (4.0-preview) as a selectable base-image version and introduce the grubazl4 flavor in servicing/rollback test templates.
  • Route AZL4 base-image acquisition through a dedicated blob-download template, and centralize “requirements staging” (VHDX/RPM movement + SSH key staging) in a shared template.
  • Adjust rollback/servicing runtime flags to skip secure-boot on grubazl4, and force skipping netplan runtime testing for grubazl4 rollback runs.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
.pipelines/templates/stages/trident_images/trident-testimg-template.yml Adds 4.0-preview support and switches requirement staging to shared common_tasks templates.
.pipelines/templates/stages/testing_servicing/vm-testing.yml Adds includeAzl4 and a new build/test stage wiring for grubazl4 servicing runs.
.pipelines/templates/stages/testing_servicing/testing-template.yml Skips --secure-boot flag for grubazl4 flavor.
.pipelines/templates/stages/testing_rollback/vm-testing.yml Adds includeAzl4 and wires rollback test jobs to use grubazl4 artifacts/stages.
.pipelines/templates/stages/testing_rollback/testing-template.yml Skips secure-boot on grubazl4 and forces skip-netplan-runtime-testing on grubazl4.
.pipelines/templates/stages/common_tasks/prepare-testimage-requirements.yml New shared template to stage SSH keys and move base images/RPM artifacts into expected locations.
.pipelines/templates/stages/common_tasks/download-azl4-base-vhdx.yml New AZL4 base VHDX + cached RPM download from blob storage.
.pipelines/templates/stages/build_image/build-image.yml Adds azureLinuxVersion parameter and passes it through to the build template.
.pipelines/templates/stages/build_image/build-image-template.yml Adds 4.0-preview support and integrates shared AZL4 download + requirements staging templates.
.pipelines/templates/e2e-template.yml Enables includeAzl4 for rollback and servicing VM test stages.

Comment thread .pipelines/templates/stages/common_tasks/download-azl4-base-vhdx.yml Outdated
Comment thread .pipelines/templates/stages/build_image/build-image-template.yml Outdated
Comment thread .pipelines/templates/stages/testing_rollback/testing-template.yml Outdated
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-3-pipelines branch from 5cd9e74 to 547d478 Compare June 26, 2026 21:33
if [ "${{ parameters.flavor }}" != "uki" ]; then
if [[ "${{ parameters.testSecureBoot }}" == 'True' ]]; then
STORM_DYNAMIC_FLAGS="$STORM_DYNAMIC_FLAGS --secure-boot"
# grubazl4 is skipped: its base image does not support secure boot yet.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revisit

if [ "${{ parameters.flavor }}" != "uki" ]; then
if [[ "${{ parameters.testSecureBoot }}" == 'True' ]]; then
FLAGS="$FLAGS --secure-boot"
# grubazl4 is skipped: its base image does not support secure boot yet.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revisit

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

.pipelines/templates/stages/testing_rollback/vm-testing.yml:116

  • RuntimeUpdateAndRollbackTesting depends on BuildImagesQemuGrubAzl4, but this template never defines a BuildImagesQemuGrubAzl4 stage. This will fail pipeline compilation when includeAzl4: true (e.g. via .pipelines/templates/e2e-template.yml). Add a BuildImagesQemuGrubAzl4 stage (similar to the one in stages/testing_servicing/vm-testing.yml) or remove the dependency and ensure the azl4 images are produced another way.
          - BuildingTools
          - ${{ if eq(parameters.includeQemuGrub, true) }}:
              - BuildImagesQemuGrub
          - ${{ if eq(parameters.includeAzl4, true) }}:
              - BuildImagesQemuGrubAzl4
          - ${{ if eq(parameters.includeQemu, true) }}:
              - BuildImagesQemu
          - ${{ if eq(parameters.includeUKI, true) }}:
              - BuildImagesUKI

Comment thread .pipelines/templates/stages/common_tasks/download-azl4-base-vhdx.yml Outdated
Comment thread .pipelines/templates/stages/testing_rollback/vm-testing.yml
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-3-pipelines branch 2 times, most recently from 85add72 to 951a287 Compare June 26, 2026 22:12
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-2-images branch from 9fdf734 to 9e50bd5 Compare June 26, 2026 22:16
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-3-pipelines branch from 951a287 to 8783943 Compare June 26, 2026 22:18
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-2-images branch from e0ce882 to 8559e3d Compare June 26, 2026 22:49
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-3-pipelines branch 7 times, most recently from f8b2dd7 to 3bc3bbe Compare June 27, 2026 01:06
bfjelds and others added 4 commits June 27, 2026 01:15
Wire the rollback and servicing test pipelines to build and exercise Azure
Linux 4 grub test images: shared common_tasks templates to download the azl4
base VHDX and prepare test-image requirements (including SSH key staging,
which replaces the simple inline staging step from the previous change),
refactored build-image and trident-testimg templates, and the
rollback/servicing testing templates (skip netplan runtime testing on
grubazl4; document the secure-boot gap on the azl4 base image).

Stacked on the azl4 test-image PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… step timeout

Remove the grubazl4 force-skip of netplan runtime testing now that the
early-boot systemd generator deadlock is fixed upstream (azurelinux
PR #17791), and raise the rollback test step timeout from 5 to 10 minutes
so the longer grubazl4 rollback suite can complete.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bfjelds and others added 2 commits June 27, 2026 01:15
…-online stall

The rollback netplan runtime test injects a dummy<N> netplan device. By
default systemd-networkd-wait-online waits for every managed link to be
routable; the dummy virtual interface never becomes routable, so on AZL4
(netplan generate/configure split) wait-online can block for its full
120s timeout. That delays network-online.target -> trident.service and
the post-update commit (ab-update-finalized -> provisioned), inflating
each rollback subtest by up to two minutes.

Setting optional: true emits RequiredForOnline=no so wait-online ignores
the test dummy. The real uplink (vmeth*/eth0) still gates online.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bfjelds bfjelds force-pushed the user/bfjelds/azl4-3-pipelines branch from 3bc3bbe to 5b34e5d Compare June 27, 2026 01:15
@bfjelds bfjelds requested a review from Copilot June 27, 2026 01:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Comment thread .pipelines/templates/stages/common_tasks/prepare-testimage-requirements.yml Outdated
Comment thread .pipelines/templates/stages/build_image/build-image-template.yml
- SSH staging: stage only id_rsa.pub into the testimage tree; do not
  copy or chmod the private key, which must not be baked into images.
- download-azl4-base-vhdx: download OS override RPMs into the dedicated
  artifacts/rpm-overrides channel (always included by the builder)
  instead of bin/RPMS (Trident-only, gated on requires_trident).
- prepare-testimage-requirements: harden the prepare/move block with
  set -euxo pipefail.
- build-image-template: fail fast for AZL4 preview if the resolved base
  image type is not azl4_qemu_guest (the only type download-azl4 fetches).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comment thread .pipelines/templates/stages/testing_rollback/vm-testing.yml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants