Use metrics to display runner statuses instead of status field for EphemeralRunnerSet and AutoscalingRunnerSet#4557
Conversation
|
Hello! Thank you for your contribution. Please review our contribution guidelines to understand the project's testing and code conventions. |
d138dfd to
5223a44
Compare
There was a problem hiding this comment.
Pull request overview
This PR moves ephemeral runner lifecycle aggregation visibility from status fields on EphemeralRunnerSet / AutoscalingRunnerSet into Prometheus metrics, and introduces a shared lifecycle bucketing helper to keep the aggregation deterministic.
Changes:
- Add Prometheus gauges for additional EphemeralRunner lifecycle buckets (succeeded/outdated/deleting) and a new
SetEphemeralRunnerCountsByLifecyclesetter API. - Remove runner-count fields from
EphemeralRunnerSetStatus/AutoscalingRunnerSetStatus(types + CRDs) and update controllers/tests to stop relying on those status fields. - Introduce
AggregateEphemeralRunnerLifecycleto consistently bucket EphemeralRunners (with deletion timestamp precedence) and update controller logic to use it.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| controllers/actions.github.com/metrics/metrics.go | Adds new lifecycle gauges and replaces the old 3-bucket setter with a 6-bucket lifecycle setter. |
| controllers/actions.github.com/metrics/metrics_test.go | Adds unit tests for the new metrics setter and CommonLabels contract. |
| controllers/actions.github.com/metrics/lifecycle_metrics_integration_test.go | Adds an integration-style test that gathers metrics from the controller-runtime registry. |
| controllers/actions.github.com/ephemeralrunnerset_controller.go | Stops publishing ephemeral runner count metrics from the ERS reconciler and simplifies ERS status to Phase only. |
| controllers/actions.github.com/ephemeralrunnerset_controller_test.go | Updates tests to assert runner counts by listing EphemeralRunner objects instead of using ERS status fields. |
| controllers/actions.github.com/ephemeralrunner_lifecycle.go | Adds a shared lifecycle bucketing helper with deletion-timestamp precedence. |
| controllers/actions.github.com/ephemeralrunner_lifecycle_test.go | Adds thorough unit tests for lifecycle bucketing precedence and totals. |
| controllers/actions.github.com/ephemeralrunner_controller.go | Emits lifecycle metrics on runner lifecycle/status changes and adds a PublishMetrics gate to the reconciler. |
| controllers/actions.github.com/autoscalingrunnerset_controller.go | Replaces “active runners” check based on ERS status fields with a list+aggregate of EphemeralRunners. |
| controllers/actions.github.com/autoscalingrunnerset_controller_test.go | Updates tests to stop asserting autoscaling status runner-count fields. |
| config/crd/bases/actions.github.com_ephemeralrunnersets.yaml | Removes ERS status count fields/printcolumns from the CRD schema. |
| config/crd/bases/actions.github.com_autoscalingrunnersets.yaml | Removes ARS status count fields/printcolumns from the CRD schema. |
| charts/gha-runner-scale-set-controller/crds/actions.github.com_ephemeralrunnersets.yaml | Mirrors the ERS CRD schema changes in the Helm chart CRDs. |
| charts/gha-runner-scale-set-controller/crds/actions.github.com_autoscalingrunnersets.yaml | Mirrors the ARS CRD schema changes in the Helm chart CRDs. |
| charts/gha-runner-scale-set-controller-experimental/crds/actions.github.com_ephemeralrunnersets.yaml | Mirrors the ERS CRD schema changes in the experimental chart CRDs. |
| charts/gha-runner-scale-set-controller-experimental/crds/actions.github.com_autoscalingrunnersets.yaml | Mirrors the ARS CRD schema changes in the experimental chart CRDs. |
| apis/actions.github.com/v1alpha1/ephemeralrunnerset_types.go | Removes ERS status count fields and associated kubebuilder printcolumns. |
| apis/actions.github.com/v1alpha1/autoscalingrunnerset_types.go | Removes ARS status count fields and associated kubebuilder printcolumns. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
f1ca790 to
7b50f20
Compare
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…hemeralRunnerSet and AutoscalingRunnerSet
7b50f20 to
dbdad1b
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
controllers/actions.github.com/autoscalingrunnerset_controller.go:304
- The previous logic that deferred EphemeralRunnerSet spec updates while runners were pending/running (and instead deleted the listener to drain jobs) has been removed. With this change, spec updates are applied immediately on integrity hash change, which can cause in-flight jobs to be disrupted if the EphemeralRunnerSet reacts to spec changes by replacing/deleting runners. Consider restoring a drain gate by checking for active runners via EphemeralRunner objects/Pods (or another reliable source) before patching the EphemeralRunnerSet, and only proceed once the set is drained.
if ephemeralRunnerSet.Annotations[annotationKeyIntegrityHash] != desired.Annotations[annotationKeyIntegrityHash] {
original := ephemeralRunnerSet.DeepCopy()
ephemeralRunnerSet.Spec.EphemeralRunnerMetadata = desired.Spec.EphemeralRunnerMetadata
ephemeralRunnerSet.Spec.EphemeralRunnerSpec = desired.Spec.EphemeralRunnerSpec
ephemeralRunnerSet.Labels = r.filterAndMergeLabels(ephemeralRunnerSet.Labels, desired.Labels)
ephemeralRunnerSet.Annotations = r.mergeAnnotations(ephemeralRunnerSet.Annotations, desired.Annotations)
log.Info("Updating ephemeral runner set spec to match the desired spec")
if err := r.Patch(ctx, &ephemeralRunnerSet, client.MergeFrom(original)); err != nil {
log.Error(err, "Failed to patch ephemeral runner set to match the desired spec")
return ctrl.Result{}, err
| type EphemeralRunnerSetSpec struct { | ||
| // Replicas is the number of desired EphemeralRunner resources in the k8s namespace. | ||
| // +optional | ||
| Replicas int `json:"replicas,omitempty"` | ||
| // PatchID is the unique identifier for the patch issued by the listener app | ||
| // +optional | ||
| PatchID int `json:"patchID"` | ||
| // EphemeralRunnerSpec is the spec of the ephemeral runner | ||
| // +optional | ||
| EphemeralRunnerSpec EphemeralRunnerSpec `json:"ephemeralRunnerSpec,omitempty"` |
| commonLabels, err := ephemeralRunnerMetricLabels(ephemeralRunner) | ||
| if err != nil { | ||
| log.Error(err, "Failed to build ephemeral runner metric labels") | ||
| return | ||
| } | ||
|
|
||
| key := types.NamespacedName{Namespace: ephemeralRunner.Namespace, Name: ephemeralRunner.Name} | ||
|
|
||
| ephemeralRunnerPhaseMetrics.Lock() | ||
| defer ephemeralRunnerPhaseMetrics.Unlock() | ||
|
|
||
| previousPhase, ok := ephemeralRunnerPhaseMetrics.phases[key] | ||
| if ok && previousPhase == phase { | ||
| return | ||
| } |
There was a problem hiding this comment.
This will be addressed in the follow up PR that introduces caching
There is no need for updating the status at points to display information about ephemeral runner state aggregations. Metrics are way more useful place to do it. Therefore, we should move this information to a metric.