pcluster imagebuilder fails on Rocky 9.7 amis

[ezrizhu]

pcluster imagebuilder pulls the latest kmod for lustre client, which is only built for RHEL/Rocky9.8, which does not work on Rocky as Rocky 9.8 has yet to be released as of now.

https://github.com/aws/aws-parallelcluster-cookbook/blob/release-3.15/cookbooks/aws-parallelcluster-environment/resources/lustre/partial/_install_lustre_centos_redhat.rb

https://github.com/aws/aws-parallelcluster-cookbook/blob/release-3.15/cookbooks/aws-parallelcluster-environment/resources/lustre/lustre_rocky8.rb

$ dnf --disablerepo='*' --enablerepo=aws-fsx info kmod-lustre-client
Last metadata expiration check: 1 day, 7:24:13 ago on Sun May 24 21:47:19 2026.
Installed Packages
Name         : kmod-lustre-client
Version      : 2.15.6
Release      : 1.fsx30.el9
Architecture : x86_64
Size         : 16 M
Source       : lustre-client-2.15.6-1.fsx30.el9.src.rpm
Repository   : @System
From repo    : aws-fsx
Summary      : lustre-client kernel module(s)
URL          : https://wiki.whamcloud.com/
License      : GPL
Description  : This package provides the lustre-client kernel modules built for
             : the Linux kernel 5.14.0-611.49.1.el9_7.x86_64 for the x86_64
             : family of processors.

Available Packages
Name         : kmod-lustre-client
Version      : 2.15.6
Release      : 1.fsx31.el9
Architecture : x86_64
Size         : 2.7 M
Source       : lustre-client-2.15.6-1.fsx31.el9.src.rpm
Repository   : aws-fsx
Summary      : lustre-client kernel module(s)
URL          : https://wiki.whamcloud.com/
License      : GPL
Description  : This package provides the lustre-client kernel modules built for
             : the Linux kernel 5.14.0-687.5.3.el9_8.x86_64 for the x86_64
             : family of processors.

Resulting in below error

Stdout: ---- Begin output of ["dnf", "-y", "install", "kmod-lustre-client-0:2.15.6-1.fsx31.el9.x86_64", "lustre-client-0:2.15.6-1.fsx31.el9.x86_64"] ----
2026-05-25T10:20:08.289Z
Stdout: STDOUT: Last metadata expiration check: 0:00:20 ago on Mon 25 May 2026 10:19:11 AM UTC.
2026-05-25T10:20:08.289Z
Stdout: (try to add '--skip-broken' to skip uninstallable packages)
2026-05-25T10:20:08.289Z
Stdout: STDERR: Error:
2026-05-25T10:20:08.289Z
Stdout: Problem: conflicting requests
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(kmalloc_caches) = 0x4344f508 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(kmalloc_trace) = 0x544be8df needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(kthread_create_on_node) = 0x73ea27e3 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(module_layout) = 0x3075a628 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(pcpu_hot) = 0xd294630d needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(wake_up_process) = 0xa06cd0d4 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__alloc_pages) = 0x3f53e658 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__cpuhp_remove_state) = 0xe544a11f needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__cpuhp_setup_state) = 0xdaff2ec8 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__d_drop) = 0x46dd881a needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__ethtool_get_link_ksettings) = 0x6881b6c5 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__filemap_set_wb_err) = 0xf7de2cbd needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__folio_batch_release) = 0x2a6f7d57 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__folio_lock) = 0x8b781cb4 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__folio_put) = 0x07ae575c needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__free_pages) = 0xa763ceef needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
<snip>

[nyetsche]

Rocky 9.8 is released: https://docs.rockylinux.org/10/release_notes/9_8/

ImageBuilder fails but for a slightly different reason now; the Chef cookbook creates a yum repo (aws-fsx) with the major release embedded in the URL

https://github.com/aws/aws-parallelcluster-cookbook/blob/develop/cookbooks/aws-parallelcluster-environment/resources/lustre/partial/_install_lustre_centos_redhat.rb#L19

sets base_url defined at

https://github.com/aws/aws-parallelcluster-cookbook/blob/develop/cookbooks/aws-parallelcluster-environment/resources/lustre/partial/_install_lustre_centos_redhat.rb#L21

defined as [...]/el/#{node['platform_version']}/$basearch

which ends up as on disk as:

baseurl=https://fsx-lustre-client-repo.s3.amazonaws.com/el/9.7/$basearch

But there is nothing (I've found) in https://fsx-lustre-client-repo.s3.amazonaws.com/el/9.8/

[ezrizhu]

I was also able to reproduce a failure, @nyetsche is this also what you saw?

2026-06-01T22:02:58.183Z
Stdout: == Installing EFA packages ==
2026-06-01T22:02:58.183Z
Stdout: dkms is already installed
2026-06-01T22:02:58.183Z
Stdout: infiniband-diags-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: libibumad-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: libibverbs-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: libibverbs-utils-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: librdmacm-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: librdmacm-utils-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: rdma-core-devel-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: Alternative rdma-core is available
2026-06-01T22:02:58.183Z
Stdout: Last metadata expiration check: 0:19:49 ago on Mon Jun 1 21:42:30 2026.
2026-06-01T22:02:58.183Z
Stdout: No match for argument: infiniband-diags-compat
2026-06-01T22:02:58.183Z
Stdout: STDERR: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Error: Unable to find a match: kernel-devel-5.14.0-611.5.1.el9_7.x86_64
2026-06-01T22:02:58.183Z
Stdout: Error: Failed to install packages.
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Error: Unable to find a match: infiniband-diags-compat
2026-06-01T22:02:58.183Z
Stdout: Error: Failed to install packages.
2026-06-01T22:02:58.183Z
Stdout: Error: failed to install EFA packages, exiting
2026-06-01T22:02:58.183Z
Stdout: ---- End output of "bash" ----
2026-06-01T22:02:58.183Z
Stdout: Ran "bash" returned 1
2026-06-01T22:02:58.226Z
CmdExecution: Command execution has been completed
2026-06-01T22:02:58.226Z
CmdExecution: [ ERROR ] Command execution has resulted in an error
2026-06-01T22:02:58.226Z
CmdExecution: Stderr: + . /opt/parallelcluster/system_info ++ OS=rocky9 ++ PLATFORM=RHEL ++ RELEASE=rocky.9.8 ++ VERSION_ID=9.8 ++ COOKBOOK_URL=https://us-gov-west-1-aws-parallelcluster.s3.us-gov-west-1.amazonaws.com/parallelcluster/3.15.0/cookbooks/aws-parallelcluster-cookbook-3.15.0.tgz ++ CINC_URL=https://us-gov-west-1-aws-parallelcluster.s3.us-gov-west-1.amazonaws.com/archives/cinc/cinc-install-1.4.0.sh ++ COOKBOOK_NAME=aws-parallelcluster-cookbook-3.15.0 ++ KERNEL_PACKAGES_PREFIX=kernel + echo 'Calling chef-client with /etc/parallelcluster/image_dna.json' + cat /etc/parallelcluster/image_dna.json + cinc-client --local-mode --config /etc/chef/client.rb --log_level info --force-formatter --no-color --chef-zero-port 8889 --json-attributes /etc/parallelcluster/image_dna.json --override-runlist aws-parallelcluster-entrypoints::install
2026-06-01T22:02:58.226Z
CmdExecution: ExitCode 1
2026-06-01T22:02:58.226Z

[nyetsche]

Not quite, your error is a combination of failing to install (via dnf):

infiniband-diags-compat

and

kernel-devel-5.14.0-611.5.1.el9_7.x86_64

The infiniband-diags-compat issue I had to work around in a forked version of aws-parallelcluster (since we can't update to a newer version); during the build process something tries to do a dnf install of rdma-core, libibverbs, and so on, and infiniband-diags-compat is included in the list.

The Rocky repos provide most of those packages, but with a later revision - rdma-core-61.0-2, not rdma-core-61.0-1 (as in your output). Ex: https://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/r/rdma-core-61.0-2.el9.x86_64.rpm

There's no infiniband-diags-compat in the Rocky repos though, it's instead part of the aws-efa-installer tarball ( ex: https://efa-installer.amazonaws.com/aws-efa-installer-1.34.0.tar.gz but version varies across the pcluster version ).

In my branch, I hacked around by downloading https://efa-installer.amazonaws.com/aws-efa-installer-1.34.0.tar.gz and manually installing the RPM dnf install /tmp/aws-efa-installer/RPMS/ROCKYLINUX9/x86_64/rdma-core/infiniband-diags-compat-52.0-1.el9.x86_64.rpm since I'm not relying on these advanced RDMA/networking features anyway.

[nyetsche]

The second error (if fixing the infiniband-diags-compat doesn't end up skipping over it) is that Rocky doesn't keep older package versions in the default repos. So at https://dl.rockylinux.org/pub/rocky/9/AppStream/x86_64/os/Packages/k/ you'll see there's just the two latest kernel-devel:

kernel-devel-5.14.0-687.10.1.el9_8.0.1.x86_64.rpm
kernel-devel-5.14.0-687.12.1.el9_8.x86_64.rpm

I'm guessing that the base AMI you're booting from is 5.14.0-611.5.1.el9_7.x86_64 and it doesn't install the latest 5.14.0-687.12.1.el9_8.x86_64.rpm (mirror with missing package?). I'm less sure how to fix that for you, except maybe try another official Rocky9 AMI.