[DNM] [env_op_images] Add pulled images report to env_op_images role by nemarjan · Pull Request #3820 · openstack-k8s-operators/ci-framework

nemarjan · 2026-03-31T20:03:39Z

Cross-references pod images against ICSP/IDMS mirror rules to report which images have a mirror configured and which pull directly from the original registry.

openshift-ci · 2026-03-31T20:03:42Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

softwarefactory-project-zuul · 2026-03-31T22:36:36Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/44f99764da904637b15366619bfe386e

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 31m 41s
❌ podified-multinode-edpm-deployment-crc POST_FAILURE in 1h 27m 37s
❌ cifmw-crc-podified-edpm-baremetal POST_FAILURE in 1h 36m 20s
❌ cifmw-crc-podified-edpm-baremetal-minor-update POST_FAILURE in 2h 16m 34s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 43s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 18s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 19s
❌ cifmw-molecule-env_op_images FAILURE in 25m 57s

openshift-ci · 2026-04-08T13:06:02Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from nemarjan. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Cross-references pod images against ICSP/IDMS mirror rules to report which images have a mirror configured and which pull directly from the original registry. Co-authored-by: Cursor <cursor@cursor.com> Signed-off-by: nemarjan <nemarjan@redhat.com>

softwarefactory-project-zuul · 2026-04-14T14:28:41Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/024650ccc68343159b52144850714fdf

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 10m 48s
❌ podified-multinode-edpm-deployment-crc POST_FAILURE in 1h 29m 24s
❌ cifmw-crc-podified-edpm-baremetal POST_FAILURE in 1h 26m 50s
❌ cifmw-crc-podified-edpm-baremetal-minor-update POST_FAILURE in 1h 57m 05s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 07s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 02s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 55s
✔️ cifmw-molecule-env_op_images SUCCESS in 21m 01s

softwarefactory-project-zuul · 2026-04-15T17:18:43Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/f3a860921ee749599949a7c159a39ceb

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 11m 49s
❌ podified-multinode-edpm-deployment-crc POST_FAILURE in 1h 24m 03s
❌ cifmw-crc-podified-edpm-baremetal POST_FAILURE in 1h 45m 16s
❌ cifmw-crc-podified-edpm-baremetal-minor-update POST_FAILURE in 1h 58m 14s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 34s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 12s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 30s
✔️ cifmw-molecule-env_op_images SUCCESS in 20m 42s

softwarefactory-project-zuul · 2026-04-16T12:47:30Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/554dcc5512254d6a8a0feb15801e1fa1

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 17m 05s
❌ podified-multinode-edpm-deployment-crc POST_FAILURE in 1h 25m 32s
❌ cifmw-crc-podified-edpm-baremetal POST_FAILURE in 1h 33m 02s
❌ cifmw-crc-podified-edpm-baremetal-minor-update POST_FAILURE in 2h 04m 02s
✔️ cifmw-pod-zuul-files SUCCESS in 8m 57s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 35s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 01s
✔️ cifmw-molecule-env_op_images SUCCESS in 23m 16s

softwarefactory-project-zuul · 2026-04-16T15:47:22Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/93c073f390bc4396a0e93727c8669a15

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 16m 30s
❌ podified-multinode-edpm-deployment-crc POST_FAILURE in 1h 26m 25s
❌ cifmw-crc-podified-edpm-baremetal POST_FAILURE in 1h 29m 44s
❌ cifmw-crc-podified-edpm-baremetal-minor-update POST_FAILURE in 2h 03m 59s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 43s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 39s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 37s
✔️ cifmw-molecule-env_op_images SUCCESS in 23m 44s

softwarefactory-project-zuul · 2026-04-17T00:45:11Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/f2bc5cae3ee1449ea3ee8de2d237c38f

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 17m 18s
❌ podified-multinode-edpm-deployment-crc POST_FAILURE in 1h 23m 24s
❌ cifmw-crc-podified-edpm-baremetal POST_FAILURE in 1h 27m 32s
❌ cifmw-crc-podified-edpm-baremetal-minor-update POST_FAILURE in 2h 05m 25s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 29s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 36s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 14s
✔️ cifmw-molecule-env_op_images SUCCESS in 22m 53s

softwarefactory-project-zuul · 2026-04-17T05:59:26Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/afbaedd8aa234832bd0cf661af1b280a

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 45m 38s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 27m 29s
❌ cifmw-crc-podified-edpm-baremetal POST_FAILURE in 1h 33m 33s
❌ cifmw-crc-podified-edpm-baremetal-minor-update RETRY_LIMIT in 21m 48s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 54s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 39s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 47s
✔️ cifmw-molecule-env_op_images SUCCESS in 20m 50s

softwarefactory-project-zuul · 2026-04-17T08:34:20Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/f5e8239a29ee4f3aad864ac5f1b2a101

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 05m 16s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 07s
❌ cifmw-crc-podified-edpm-baremetal POST_FAILURE in 1h 29m 38s
❌ cifmw-crc-podified-edpm-baremetal-minor-update POST_FAILURE in 1h 53m 01s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 56s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 31s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 33s
✔️ cifmw-molecule-env_op_images SUCCESS in 22m 17s

softwarefactory-project-zuul · 2026-04-17T10:19:22Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/eae473ac41d74049b9ea13b955db9e8d

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 40m 47s
❌ podified-multinode-edpm-deployment-crc POST_FAILURE in 1h 18m 21s
❌ cifmw-crc-podified-edpm-baremetal POST_FAILURE in 1h 28m 51s
❌ cifmw-crc-podified-edpm-baremetal-minor-update RETRY_LIMIT in 36m 43s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 35s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 58s
✔️ cifmw-pod-pre-commit SUCCESS in 10m 39s
✔️ cifmw-molecule-env_op_images SUCCESS in 23m 51s

Cross-reference the pulled-images report with CRI-O journal logs from cluster nodes to confirm which images were actually pulled by the container runtime. Runs automatically when kubeconfig is defined, same as the pulled-images report itself. Co-authored-by: Cursor <cursor@cursor.com> Signed-off-by: nemarjan <nemarjan@redhat.com>

evallesp

I like the way is getting. And functionally now is working as expected.
Check my suggestions. Feel free to close them specially those marked as "non-blocking"

evallesp · 2026-04-17T13:16:36Z

+          oc get nodes failed (rc={{ _verify_crio_nodes_json.rc }}); cannot fetch CRI-O logs.
+
+    # Filename is sanitised to avoid path-traversal with unusual node names.
+    - name: Fetch CRI-O unit logs per node


(blocking) suggestion: let's move this task.

One per take the result and another to write in the logs.

Use for the first one: ansible.builtin.command -> cmd: oc adm node-logs "{{ item }}" -u crio --since=-2h

(From previous command, we should think if 2h is enough.

Then following task should be using ansible.builtin.copy where dest is the log path, and origin is stdout from previous task registered variable.

evallesp · 2026-04-17T13:20:07Z

@@ -0,0 +1,298 @@
+#!/usr/bin/python


(non-blocking) requirement: We need to add tests for this script.
You can check crawl_n_mask tests at: tests/unit/modules

I'm Ok if we prepare following work to address this (reason why it's marked as non-blocking) but probably we can check the output of a AI model.

evallesp · 2026-04-17T13:22:47Z

+      failed_when: false
+
+    # Flatten all loop results, then one report row per container + initContainer.
+    - name: Build per-pod pulled images report


(blocking) suggestion: This should go in a jinja2 temlate file under template folder in the role.

evallesp · 2026-04-17T13:23:26Z

+          {%         set match.expected_pull_basis = 'mirror' %}
+          {%       endif %}
+          {%     endfor %}
+          {%     set _ = entries.append({


(non-blocking) suggestion: I'm unsure if we want these all values.

evallesp · 2026-04-17T13:24:53Z

@@ -0,0 +1 @@
+../../../plugins/modules/verify_pulled_report_crio.py


this requires new line.

evallesp · 2026-04-17T13:28:04Z

@@ -4,6 +4,9 @@ A role to gather the container images used in the openstack deployment with spec
 ## Parameters
 * `cifmw_env_op_images_dir`: (String) Directory where the operator_images.yaml will be stored. Defaults to `~/ci-framework-data/artifacts`
 * `cifmw_env_op_images_file`: (String) Name of the file storing the operator images and tags. Defaults to `operator_images.yaml`


(blocking) suggestion: I think we should add proper doc about this new feature.

evallesp · 2026-04-17T13:34:27Z

+        state: directory
+        mode: "0755"
+
+    # Legacy OpenShift mirror CRD; empty or error is OK (parsed as no items).


(non-blocking) suggestion: I'd remove some of these comments. Just leave where tasks have some lofic like thsoe jinja templates.
Also I'd reword them to feel more natural.

evallesp · 2026-04-17T13:35:13Z

+        mode: "0644"
+
+    # Console visibility: split rows by how ICSP/IDMS classification turned out.
+    - name: Images with expected_pull_basis source


(non-blocking) question: Do we need these debugs statements?

evallesp · 2026-04-17T13:36:32Z

+      loop_control:
+        label: "{{ item.item | default('') }}"
+
+    - name: Find fetched CRI-O log files


(non-blockign) suggestion: If we split the task into two, one for executing oc adm node-logs other for retrieving the logs, this might not be necessary anymore.

evallesp · 2026-04-17T13:38:36Z

+
+    # One flat list of rules for templating: each mirror list entry becomes
+    # its own row so prefix matching can use the same loop for all mirrors.
+    - name: Build source-to-mirror mapping from ICSP/IDMS


(blocking) suggestion: I think this comment has room for improvement. Maybe an example of the output?

nemarjan requested review from Valkyrie00 and evallesp March 31, 2026 20:03

openshift-ci bot added the do-not-merge/work-in-progress label Mar 31, 2026

nemarjan self-assigned this Mar 31, 2026

nemarjan force-pushed the logic-multiple-registries branch 5 times, most recently from 2718747 to 5ed5de8 Compare April 14, 2026 12:16

nemarjan force-pushed the logic-multiple-registries branch 2 times, most recently from 4282933 to 5ae5fab Compare April 15, 2026 15:05

nemarjan force-pushed the logic-multiple-registries branch 3 times, most recently from c07895d to 80115fd Compare April 16, 2026 10:29

nemarjan force-pushed the logic-multiple-registries branch from 80115fd to 4530a09 Compare April 16, 2026 13:29

nemarjan force-pushed the logic-multiple-registries branch 5 times, most recently from 4bf52d0 to 5449b94 Compare April 16, 2026 22:26

nemarjan force-pushed the logic-multiple-registries branch from 5449b94 to e4a83cd Compare April 17, 2026 02:05

nemarjan force-pushed the logic-multiple-registries branch 2 times, most recently from 6390c54 to 334fb7f Compare April 17, 2026 04:12

nemarjan force-pushed the logic-multiple-registries branch from 334fb7f to b2e0d06 Compare April 17, 2026 06:28

nemarjan force-pushed the logic-multiple-registries branch from b2e0d06 to 4f0376f Compare April 17, 2026 08:37

nemarjan force-pushed the logic-multiple-registries branch from 4f0376f to bbd3d23 Compare April 17, 2026 10:20

evallesp reviewed Apr 17, 2026

View reviewed changes

		@@ -0,0 +1 @@
		../../../plugins/modules/verify_pulled_report_crio.py No newline at end of file

Conversation

nemarjan commented Mar 31, 2026

Uh oh!

openshift-ci bot commented Mar 31, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 31, 2026

Uh oh!

openshift-ci bot commented Apr 8, 2026

Uh oh!

softwarefactory-project-zuul bot commented Apr 14, 2026

Uh oh!

softwarefactory-project-zuul bot commented Apr 15, 2026

Uh oh!

softwarefactory-project-zuul bot commented Apr 16, 2026

Uh oh!

softwarefactory-project-zuul bot commented Apr 16, 2026

Uh oh!

softwarefactory-project-zuul bot commented Apr 17, 2026

Uh oh!

softwarefactory-project-zuul bot commented Apr 17, 2026

Uh oh!

softwarefactory-project-zuul bot commented Apr 17, 2026

Uh oh!

softwarefactory-project-zuul bot commented Apr 17, 2026

Uh oh!

evallesp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants