OCPBUGS-98575: watch APIServer TLS profile and restart on change#3867
OCPBUGS-98575: watch APIServer TLS profile and restart on change#3867tmshort wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
This PR updates the package-server startup behavior on OpenShift to monitor the cluster-wide APIServer.config.openshift.io/cluster TLS profile and trigger a process restart when it changes, so the package-server can come back up using the updated TLS serving settings (mirroring the existing restart-on-OLMConfig-change pattern).
Changes:
- Add an APIServer informer/queue to detect TLS profile changes and exit(0) to trigger a restart.
- Introduce a new operator workqueue dedicated to APIServer TLS change handling.
- Compare current serving TLS settings against the APIServer TLS profile to decide whether a restart is necessary.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // If the TLS profile was self-applied at startup (no --tls-min-version flag), | ||
| // watch APIServer for changes and restart so the new profile is picked up. | ||
| // When flags are externally injected (e.g. by PSM), this watch is skipped; | ||
| // the external manager is responsible for triggering restarts on profile changes. | ||
| if o.SecureServing.MinTLSVersion != "" && len(o.SecureServing.CipherSuites) > 0 { |
| newMinVersion, newCipherSuites := olmapiserver.GetSecurityProfileConfig(apiServer.Spec.TLSSecurityProfile) | ||
| newMinVersionStr := libcrypto.TLSVersionToNameOrDie(newMinVersion) | ||
| newCipherNames := libcrypto.CipherSuitesToNamesOrDie(newCipherSuites) | ||
|
|
||
| currentMinVersion := op.options.SecureServing.MinTLSVersion | ||
| currentCipherNames := op.options.SecureServing.CipherSuites | ||
|
|
||
| if newMinVersionStr == currentMinVersion && slices.Equal(newCipherNames, currentCipherNames) { | ||
| return nil | ||
| } | ||
|
|
||
| log.Warnf("APIServer TLS profile changed (minVersion: %s → %s), restarting to apply new profile", | ||
| currentMinVersion, newMinVersionStr) | ||
| os.Exit(0) | ||
| return nil |
| // syncAPIServerTLSProfile is called by the APIServer informer whenever the | ||
| // cluster singleton APIServer object changes. If the TLS profile differs from | ||
| // the one applied at startup the process exits so it can be restarted with the | ||
| // updated profile. This watch is only active when --tls-min-version was not | ||
| // supplied via flags; when flags are externally managed (e.g. by PSM) this | ||
| // handler is never registered. | ||
| func (op *Operator) syncAPIServerTLSProfile(obj interface{}) error { |
db27d68 to
853f1ab
Compare
| selfApplied := o.SecureServing.MinTLSVersion == "" | ||
| if selfApplied { | ||
| // Warn and continue on failure: the API server may not be reachable | ||
| // during initial startup (e.g. cluster bootstrap). Packageserver starts | ||
| // with secure TLS defaults; operators can supply --tls-min-version and |
|
|
||
| newMinVersion, newCipherSuites := olmapiserver.GetSecurityProfileConfig(apiServer.Spec.TLSSecurityProfile) | ||
| newMinVersionStr := libcrypto.TLSVersionToNameOrDie(newMinVersion) | ||
| newCipherNames := libcrypto.CipherSuitesToNamesOrDie(newCipherSuites) |
There was a problem hiding this comment.
Panic risk in watch handler: TLSVersionToNameOrDie and CipherSuitesToNamesOrDie both call panic() when the input value is not in their known maps (verified in vendor/github.com/openshift/library-go/pkg/crypto/crypto.go). The same functions are used at startup in applyClusterTLSProfileWithClients, but a panic there fails the process once and is caught by the pod restart policy. Here, this handler fires on every APIServer CR update, so if the cluster admin changes to a Custom profile that contains a cipher suite or TLS version not yet in library-go's map, the handler will panic on each reconcile, causing a crash loop rather than a clean error.
Consider wrapping with a recover, or switching to returning-an-error variants (e.g. TLSVersionToName/CipherSuiteToName) so the handler can return an error that the queue will log and retry safely:
newMinVersionStr, err := libcrypto.TLSVersionToName(newMinVersion)
if err != nil {
return fmt.Errorf("unknown TLS version %d in APIServer profile: %w", newMinVersion, err)
}| } | ||
|
|
||
| log.Warnf("APIServer TLS profile changed (minVersion: %s → %s, ciphers: %v → %v), restarting to apply new profile", | ||
| currentMinVersion, newMinVersionStr, currentCipherNames, newCipherNames) |
There was a problem hiding this comment.
Log shows unsorted ciphers, comparison uses sorted: The restart decision is correctly made on sorted copies (newSorted/currentSorted), but the Warnf on the next line logs the original unsorted slices (currentCipherNames, newCipherNames). When a restart fires, an operator reading the log sees the raw cipher order, which may differ from how the sorted comparison actually saw them. Logging the sorted versions makes the log easier to reconcile against the comparison:
| currentMinVersion, newMinVersionStr, currentCipherNames, newCipherNames) | |
| log.Warnf("APIServer TLS profile changed (minVersion: %s → %s, ciphers: %v → %v), restarting to apply new profile", | |
| currentMinVersion, newMinVersionStr, currentSorted, newSorted) |
853f1ab to
d1b788a
Compare
| // ensuring that externally-managed instances (e.g. PSM-injected flags) are not | ||
| // restarted by this code path. Providing --tls-cipher-suites without | ||
| // --tls-min-version would create a mixed state that the watch cannot reconcile | ||
| // safely, so the watch is gated on both flags being absent. | ||
| selfApplied := o.SecureServing.MinTLSVersion == "" && len(o.SecureServing.CipherSuites) == 0 |
| apiServerInformer := configinformers.NewSharedInformerFactory(cfgClient, 0).Config().V1().APIServers() | ||
| apiServerQueueInformer, qiErr := queueinformer.NewQueueInformer( | ||
| ctx, | ||
| queueinformer.WithInformer(apiServerInformer.Informer()), | ||
| queueinformer.WithQueue(op.apiServerTLSQueue), |
| apiServer, ok := obj.(*apiconfigv1.APIServer) | ||
| if !ok { | ||
| return fmt.Errorf("casting APIServer failed") | ||
| } |
d1b788a to
effa3b8
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
pkg/package-server/server/server.go:452
- The applyClusterTLSProfile godoc still states it is only used when
--tls-min-versionis not provided, but the startup logic now calls it when either TLS field is unset (min version or cipher suites). Please update this comment to reflect the current behavior.
// This is the fallback path used when --tls-min-version is not provided via flags
// (i.e. before the PSM has had a chance to inject them).
| verbs: | ||
| - get | ||
| - list | ||
| - watch |
| verbs: | ||
| - get | ||
| - list | ||
| - watch |
| // updated profile. This watch is only active when --tls-min-version was not | ||
| // supplied via flags; when flags are externally managed (e.g. by PSM) this | ||
| // handler is never registered. |
| // Snapshot the applied profile as raw uint16 values so the watch | ||
| // handler can compare without calling TLSVersionToNameOrDie or | ||
| // CipherSuitesToNamesOrDie, which panic on unrecognised inputs. | ||
| apiServerCR, getErr := cfgClient.ConfigV1().APIServers().Get(ctx, "cluster", metav1.GetOptions{}) |
effa3b8 to
ebe053e
Compare
| apiServer, ok := obj.(*apiconfigv1.APIServer) | ||
| if !ok { | ||
| return fmt.Errorf("expected *configv1.APIServer, got %T", obj) | ||
| } |
| newVersionStr := libcrypto.TLSVersionToNameOrDie(newMinVersion) | ||
| currentVersionStr := libcrypto.TLSVersionToNameOrDie(op.appliedTLSMinVersion) | ||
| newCipherNames := libcrypto.CipherSuitesToNamesOrDie(newSortedCiphers) | ||
| currentCipherNames := libcrypto.CipherSuitesToNamesOrDie(currentSortedCiphers) | ||
| log.Warnf("APIServer TLS profile changed (minVersion: %s → %s, ciphers: %v → %v), restarting to apply new profile", | ||
| currentVersionStr, newVersionStr, currentCipherNames, newCipherNames) |
When --tls-min-version is not supplied via flags, packageserver now sets up a watch on APIServer.config.openshift.io/cluster after applying the initial TLS profile at startup. If the profile changes, the process exits so it restarts with the updated settings — the same restart pattern used for OLMConfig interval changes. The watch is intentionally skipped when --tls-min-version is present (i.e. flags are injected externally). In that case the external manager is responsible for triggering restarts on profile changes, avoiding unnecessary restarts.
ebe053e to
7c44b60
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
pkg/package-server/server/server.go:451
- The comment above
applyClusterTLSProfilestill describes this as a fallback only when--tls-min-versionis unset, but the call site now invokes it when either the min version or cipher suites are unset. Updating the comment will prevent future confusion about when this path runs.
// applyClusterTLSProfile fetches the cluster-wide APIServer TLS security profile
// and applies it to the SecureServingOptions. It is a no-op on non-OpenShift clusters.
// This is the fallback path used when --tls-min-version is not provided via flags
// (i.e. before the PSM has had a chance to inject them).
| if getErr != nil { | ||
| log.WithError(getErr).Warn("Failed to snapshot APIServer TLS profile; watch will not be registered") | ||
| } else { | ||
| op.appliedTLSMinVersion, op.appliedTLSCipherSuites = olmapiserver.GetSecurityProfileConfig(apiServerCR.Spec.TLSSecurityProfile) |
| // The APIServer watch is only registered when *neither* TLS flag was originally | ||
| // supplied and the CR lookup succeeded. When flags are externally injected (e.g. | ||
| // by PSM), the watch is skipped; the external manager is responsible for | ||
| // triggering restarts on profile changes. | ||
| selfApplied := minVersionWasEmpty && cipherSuitesWasEmpty && | ||
| o.SecureServing.MinTLSVersion != "" && len(o.SecureServing.CipherSuites) > 0 |
When --tls-min-version is not supplied via flags, packageserver now sets up a watch on APIServer.config.openshift.io/cluster after applying the initial TLS profile at startup. If the profile changes, the process exits so it restarts with the updated settings — the same restart pattern used for OLMConfig interval changes.
The watch is intentionally skipped when --tls-min-version is present (i.e. flags are injected externally). In that case the external manager is responsible for triggering restarts on profile changes, avoiding unnecessary restarts.