Skip to content

feat(chart): add deploymentAnnotations for Deployment metadata - #54

Open
louisgls wants to merge 2 commits into
mainfrom
fix/chart-deployment-annotations
Open

feat(chart): add deploymentAnnotations for Deployment metadata#54
louisgls wants to merge 2 commits into
mainfrom
fix/chart-deployment-annotations

Conversation

@louisgls

Copy link
Copy Markdown
Collaborator

Context

Found while investigating a production incident on kube02: every Loki read
query failed with

tls: failed to verify certificate: x509: certificate has expired or is not yet
valid: current time 2026-07-29T12:18:00Z is after 2026-07-26T16:05:14Z

against cockpit-s3proxy.cockpit.svc.cluster.local:4433.

cert-manager had renewed the certificate correctly — the secret held a cert
valid until 2026-09-24 — but the running pods were still serving the previous
one, issued in April and expired on July 26. The pods had started on
2026-06-25, hours before the renewal, and never picked up the new material.

Two things combined:

  1. The cert secret is mounted with subPath. Kubelet never refreshes a
    subPath volume in place, so the file inside the container stays frozen at
    whatever the secret held when the pod started. A pod restart is the only
    way to pick up a renewed certificate.
  2. The chart exposes podAnnotations only, which renders onto the pod
    template. Stakater Reloader reads reloader.stakater.com/auto from
    Deployment.metadata.annotations, so there was no way to opt the workload
    into automatic restarts through values.

The consuming chart had already set s3proxy.annotations.reloader.stakater.com/auto: "true"
in its values, assuming it would land on the Deployment. No template consumed
that key, so it was silently dropped and the rotation went unnoticed until the
old certificate expired a month later.

Change

Add a deploymentAnnotations value (default {}, declared in
values.schema.json) rendered onto the Deployment's own
metadata.annotations:

deploymentAnnotations:
  reloader.stakater.com/auto: "true"

Chart bumped 1.9.3 → 1.10.0. No behavior change when the value is unset.

Verification

  • helm lint — passes.
  • helm template without the value — Deployment metadata carries no
    annotations block, byte-identical to 1.9.3.
  • helm template with the value — annotation renders on the Deployment
    metadata, correctly quoted.

🤖 Generated with Claude Code

louisgls and others added 2 commits July 29, 2026 14:29
The chart only exposed `podAnnotations`, which renders onto the pod
template. Controllers that watch the workload object itself had no way to
be configured through values.

The concrete failure: the TLS cert secret is mounted with `subPath`, and
kubelet never refreshes a subPath volume in place. When cert-manager
rotates the certificate, the running pods keep serving the old one until
they are restarted. Stakater Reloader does exactly that, but it reads
`reloader.stakater.com/auto` from `Deployment.metadata.annotations` — which
the chart could not render. The proxy therefore served an expired
certificate and every TLS client failed with `x509: certificate has
expired`.

Add `deploymentAnnotations` (default `{}`, declared in values.schema.json)
and render it onto the Deployment metadata. No behavior change when unset.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 1.10.0 entry was inserted in place of the 1.9.3 heading line, leaving
1.9.3's dashboard sub-bullets attached to 1.10.0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant