Deleting a tenant with a Kafka app hangs the namespace in Terminating (KafkaTopic strimzi.io/topic-operator finalizer)

[IvanHunters]

Summary

Deleting a tenant that has a Kafka app leaves the tenant namespace stuck in Terminating indefinitely. The KafkaTopic CR keeps the finalizer strimzi.io/topic-operator, but the Strimzi topic-operator is torn down together with the Kafka app, so nothing ever removes the finalizer and the namespace never finalizes.

Environment

• Cozystack v1.6.1-rc.1
• A Kafka app (Strimzi-based) with at least one topic

Steps to reproduce

1. Create a tenant.
2. Deploy a Kafka app with a topic (e.g. topic orders).
3. Delete the tenant (this deletes the Kafka app and the namespace).

Actual result

The tenant namespace hangs in Terminating:

NamespaceContentRemaining:    Some resources are remaining: kafkatopics.kafka.strimzi.io has 1 resource instances
NamespaceFinalizersRemaining: Some content in the namespace has finalizers remaining: strimzi.io/topic-operator in 1 resource instances

The KafkaTopic (e.g. kafka-kafka-orders) still carries finalizers: [strimzi.io/topic-operator]. The Strimzi topic-operator has already been deleted together with the Kafka app, so no controller is left to drain that finalizer. The namespace stays in Terminating forever.

Reproduced on every tenant that had a Kafka app (multiple tenants stuck the same way).

Expected result

Deleting the tenant / Kafka app should clean up KafkaTopic resources (or drain their finalizers) so the namespace terminates cleanly, without manual finalizer surgery.

Workaround

Remove the finalizer manually, then the namespace finalizes within seconds:

kubectl -n tenant-<name> patch kafkatopic <topic-name> \
  --type=merge -p '{"metadata":{"finalizers":null}}'

Suggested fix

Order the teardown so the Strimzi topic-operator outlives its KafkaTopic resources (drain the strimzi.io/topic-operator finalizer before the operator itself is removed), or have the Kafka app / tenant deletion explicitly delete KafkaTopic CRs and wait for their finalizers to clear before removing the operator.

[dosubot]

Good write-up. This is the same class of "operator torn down before its finalizer-owning CRs" bug that has already been fixed for several other apps in the repo, and the mechanism is well understood.

Root cause

The Kafka chart's topics.yaml creates KafkaTopic CRs with no finalizers field [1] — Strimzi's entity-operator adds the strimzi.io/topic-operator finalizer at runtime when it takes ownership. When the Kafka app (and the entity-operator pod) is torn down first, nothing remains to drain that finalizer, so the KafkaTopic objects block namespace GC indefinitely. This was already observed as a side-effect in PR #2934, which noted that a CrashLoopBackOff entity-operator leaves KafkaTopics stuck with the same finalizer.

No pre-delete hook exists for Kafka today

The packages/apps/kafka/templates directory contains only kafka.yaml, topics.yaml, and monitoring/config templates — there is no delete.yaml or any pre-delete hook [2]. The tenant-level cleanup-job.yaml deletes HelmReleases in the namespace with a 90s bounded wait, but that only removes the Kafka HelmRelease — it has no authority over KafkaTopic CRs or their Strimzi finalizers.

Existing patterns to follow

The repo already has two merged fixes for the identical problem class:

1. PR #3078 — the packages/apps/kubernetes chart's delete.yaml deletes the TenantControlPlane with --wait=true --timeout=120s to let Kamaji drain its own finalizer, then as a backstop patches metadata.finalizers: null on the datastore-config secret if it still exists.
2. PR #3598 — the tenant pre-delete hook with bounded waits, activeDeadlineSeconds: 270, backoffLimit: 0, and restartPolicy: Never, all sized to fit inside helm-controller's 300s uninstall budget.

Suggested fix direction

Add a Helm pre-delete hook Job to the kafka chart (mirroring the kubernetes app's delete.yaml) that:

1. Deletes all KafkaTopic CRs in the namespace with --wait=true --timeout=<N>s — while the entity-operator is still alive to process the deletion and drain its finalizer.
2. As a backstop after the timeout, patches metadata.finalizers: null on any surviving KafkaTopic CRs — safe because the Kafka CR and its operator are being torn down in the same release.
3. Carries activeDeadlineSeconds set below the HelmRelease uninstall budget, backoffLimit: 0, and restartPolicy: Never, following the same arithmetic used in the tenant and kubernetes hooks.

Your manual workaround (patching finalizers: null) is correct in the interim.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.