feat: remove product-config & clean up by razvan · Pull Request #692 · stackabletech/spark-k8s-operator

razvan · 2026-06-05T13:04:44Z

merging

razvan · 2026-06-10T10:21:01Z

--- PASS: kuttl (1336.26s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-4.1.1_ny-tlc-report-0.3.0 (136.48s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-true (137.32s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.0.1 (278.24s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-4.1.1 (130.02s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-hbase-connector-3.5.8_hbase-2.6.4_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (200.23s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_spark-connect-4.1.1_s3-use-tls-true (137.33s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-4.1.1_ny-tlc-report-0.3.0 (322.81s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-4.1.1 (66.59s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-true (192.76s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-true (165.33s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-4.1.1_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (175.59s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-4.0.1 (64.51s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-4.1.1 (61.82s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-4.1.1 (116.69s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-4.1.1 (141.93s)
        --- PASS: kuttl/harness/product-config-compat_openshift-false_spark-regression-3.5.8 (192.06s)
PASS

…ct-config

adwk67 · 2026-06-17T11:43:35Z

--- FAIL: kuttl (3221.28s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-4.1.1 (434.10s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-true (531.54s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-true (315.27s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-4.1.1_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (315.65s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-hbase-connector-3.5.8_hbase-2.6.4_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (809.44s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-4.1.1 (88.17s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-4.1.1_ny-tlc-report-0.3.0 (603.49s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-true (658.15s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-4.1.1 (180.93s)
        --- PASS: kuttl/harness/product-config-compat_openshift-false_spark-regression-3.5.8 (194.22s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-4.1.1_iceberg-latest-1.11.0 (98.75s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-4.1.1 (151.20s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-4.1.1 (230.70s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-4.1.1_ny-tlc-report-0.3.0 (166.80s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-true (174.24s)
        --- FAIL: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (974.16s)
FAIL

Re-run failed:

--- PASS: kuttl (304.70s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (304.68s)
PASS

adwk67 · 2026-06-23T12:34:06Z

Following op-rs bump:

--- PASS: kuttl (1497.44s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-4.1.1 (57.88s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-4.1.1 (121.61s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-4.1.1_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (295.98s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-4.1.1_ny-tlc-report-0.3.0 (140.07s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-4.1.1 (90.13s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (308.06s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-hbase-connector-3.5.8_hbase-2.6.4_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (425.17s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-4.1.1 (127.30s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-4.1.1 (126.02s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-4.1.1_iceberg-latest-1.11.0 (57.50s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-true (178.69s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-true (144.74s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-4.1.1_ny-tlc-report-0.3.0 (340.62s)
        --- PASS: kuttl/harness/product-config-compat_openshift-false_spark-regression-3.5.8 (35.93s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-true (137.35s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-true (326.84s)
PASS

…ogging test

maltesander · 2026-06-24T14:10:55Z

--- FAIL: kuttl (2612.82s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.8_s3-use-tls-true (122.35s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-3.5.8_s3-use-tls-false (202.58s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-4.1.1 (102.14s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-3.5.8_ny-tlc-report-0.3.0 (265.29s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-4.1.1_ny-tlc-report-0.3.0 (287.99s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-false (147.39s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-true (150.43s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-3.5.8_s3-use-tls-false (136.17s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-3.5.8_s3-use-tls-true (151.52s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-3.5.8_iceberg-latest-1.11.0 (46.32s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-4.1.1_iceberg-latest-1.11.0 (60.32s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-3.5.8 (100.09s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-false (118.03s)
        --- FAIL: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (967.57s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-true (122.84s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-3.5.8_hdfs-latest-3.5.0_zookeeper-latest-3.9.5 (164.40s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-4.1.1_hdfs-latest-3.5.0_zookeeper-latest-3.9.5 (187.27s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-3.5.8 (36.41s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-4.1.1 (42.44s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.8_s3-use-tls-false (120.21s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-3.5.8_s3-use-tls-false (135.84s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-hbase-connector-3.5.8_hbase-2.6.4_hdfs-latest-3.5.0_zookeeper-latest-3.9.5 (266.73s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-true (138.03s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-3.5.8_s3-use-tls-true (108.38s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-3.5.8 (94.90s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-4.1.1_ny-tlc-report-0.3.0 (107.25s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-3.5.8_ny-tlc-report-0.3.0 (104.00s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-3.5.8 (182.37s)
        --- PASS: kuttl/harness/product-config-compat_openshift-false_spark-regression-3.5.8 (36.06s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-4.1.1 (98.20s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-true (207.97s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-3.5.8 (50.55s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-4.1.1 (67.31s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-false (121.76s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-4.1.1 (105.07s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-3.5.8 (101.38s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-3.5.8_s3-use-tls-true (213.09s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-false (208.50s)
FAIL
ERROR:root:kuttl failed

--- PASS: kuttl (221.96s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (221.95s)
PASS

adwk67 · 2026-06-30T09:45:17Z

adwk67 · 2026-06-30T10:15:39Z

We have added vector config to the executor template: is this intentional? (I think it is, but am not 100% sure)
Update: yes, intentional as it was missing

adwk67 · 2026-06-30T13:11:38Z

🟢 Openshift tests

razvan added 2 commits June 5, 2026 14:55

feat: remove product-config & clean up

517fe43

update op-rs patch

dfa3307

razvan mentioned this pull request Jun 5, 2026

tracking: Smooth Operator Iteration 2: Remove product-config stackabletech/issues#851

Open

16 tasks

razvan added 3 commits June 5, 2026 17:04

add regression test

af9492a

fix merging properties for the history controller

b6b1452

update changelog

f210846

maltesander self-requested a review June 8, 2026 07:43

maltesander reviewed Jun 8, 2026

View reviewed changes

razvan added 2 commits June 8, 2026 11:04

implement review feedback

15f0295

update pull policy in some tests

3e009b8

maltesander reviewed Jun 8, 2026

View reviewed changes

Comment thread rust/operator-binary/src/crd/history.rs Outdated

razvan added 5 commits June 8, 2026 16:08

remove shs argument from build_configmap()

d75ec31

Merge branch 'main' into refactor/remove-product-config

f283256

remove Merge impl for history ConfigOverrides.

76931d7

update tests

9bba56a

fix tests and config overrides

1f64bf2

merging

razvan marked this pull request as ready for review June 9, 2026 11:56

maltesander reviewed Jun 9, 2026

View reviewed changes

Comment thread rust/operator-binary/src/crd/history.rs Outdated

Comment thread rust/operator-binary/src/spark_k8s_controller.rs Outdated

Comment thread rust/operator-binary/src/history/controller/validate.rs Outdated

Comment thread rust/operator-binary/src/history/controller/validate.rs Outdated

razvan added 5 commits June 9, 2026 17:25

more cleanups

a24fc24

change namespace type

b410525

replace owner reference with name and uid

791044e

validated history: add metadata and impl Resource

70c7d78

connect validated: make it in line with the history structure

10efca5

maltesander and others added 6 commits June 11, 2026 15:00

fix: use v2 get name/namespace/uid helpers.

8fa3170

fix: cleanup product-config left overs & remove allow dead_code

59411fa

bump op-rs

ffd814e

Option<KeyValueConfigOverrides> -> KeyValueConfigOverrides

77a6aa5

thread metadata through via validated object

ad14fb1

Merge remote-tracking branch 'origin/main' into refactor/remove-produ…

2ca90b8

…ct-config

fix: test should not assume bash-isms

5d19467

adwk67 and others added 16 commits June 23, 2026 14:36

bump op-rs to 0.112

e372c1d

Merge branch 'main' into refactor/remove-product-config

e847741

fix: use v2 prometheus annotations and label helper

8430d4b

refactor: use v2 container builder

59246b3

refactor: consolidate logging, remove errors

fe720a8

refactor: switch to v2 ListenerClassName

5ff6b43

refactor: switch vector config map name to ConfigMapName v2 type

1569570

refactor: switch internal constants to VolumeName

e39a874

refactor: validate config overrides upfront

60b0ad1

refactor: remove redundant parameters from Validated* structs

53829d8

refactor: consolidate logging for connect

71d04c5

docs: consolidate upstream / v2 comments

744fd18

chore: bump tokio 1.52, clap 4.6

b7694a6

fix: remove vector file tests

07cd672

fix: consolidate vector logging, adapt snapshot tests, add executor l…

4d1a2ed

…ogging test

Merge branch 'main' into refactor/remove-product-config

e6001c0

fix merge conflicts

e6ef4a7

bump anyhow to 1.0.103 due to RUSTSEC advisory

4490865

adwk67 reviewed Jun 30, 2026

View reviewed changes

Comment thread rust/operator-binary/src/history/controller/build/resource/config_map.rs Outdated

adwk67 requested changes Jun 30, 2026

View reviewed changes

revert block-owner-deletion to original value of None

420b2bf

Merge branch 'main' into refactor/remove-product-config

1cc63c4

adwk67 approved these changes Jun 30, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

razvan commented Jun 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

razvan commented Jun 10, 2026

Uh oh!

adwk67 commented Jun 17, 2026

Uh oh!

adwk67 commented Jun 23, 2026

Uh oh!

maltesander commented Jun 24, 2026

Uh oh!

adwk67 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

adwk67 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adwk67 commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adwk67 commented Jun 30, 2026 •

edited

Loading

adwk67 commented Jun 30, 2026 •

edited

Loading