Skip to content

Use metrics to display runner statuses instead of status field for EphemeralRunnerSet and AutoscalingRunnerSet#4557

Open
nikola-jokic wants to merge 21 commits into
masterfrom
nikola-jokic/ephemeral-runner-stats-to-metrics
Open

Use metrics to display runner statuses instead of status field for EphemeralRunnerSet and AutoscalingRunnerSet#4557
nikola-jokic wants to merge 21 commits into
masterfrom
nikola-jokic/ephemeral-runner-stats-to-metrics

Conversation

@nikola-jokic

Copy link
Copy Markdown
Collaborator

There is no need for updating the status at points to display information about ephemeral runner state aggregations. Metrics are way more useful place to do it. Therefore, we should move this information to a metric.

Copilot AI review requested due to automatic review settings July 6, 2026 13:18
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Hello! Thank you for your contribution.

Please review our contribution guidelines to understand the project's testing and code conventions.

@nikola-jokic nikola-jokic force-pushed the nikola-jokic/ephemeral-runner-stats-to-metrics branch from d138dfd to 5223a44 Compare July 6, 2026 13:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR moves ephemeral runner lifecycle aggregation visibility from status fields on EphemeralRunnerSet / AutoscalingRunnerSet into Prometheus metrics, and introduces a shared lifecycle bucketing helper to keep the aggregation deterministic.

Changes:

  • Add Prometheus gauges for additional EphemeralRunner lifecycle buckets (succeeded/outdated/deleting) and a new SetEphemeralRunnerCountsByLifecycle setter API.
  • Remove runner-count fields from EphemeralRunnerSetStatus / AutoscalingRunnerSetStatus (types + CRDs) and update controllers/tests to stop relying on those status fields.
  • Introduce AggregateEphemeralRunnerLifecycle to consistently bucket EphemeralRunners (with deletion timestamp precedence) and update controller logic to use it.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
controllers/actions.github.com/metrics/metrics.go Adds new lifecycle gauges and replaces the old 3-bucket setter with a 6-bucket lifecycle setter.
controllers/actions.github.com/metrics/metrics_test.go Adds unit tests for the new metrics setter and CommonLabels contract.
controllers/actions.github.com/metrics/lifecycle_metrics_integration_test.go Adds an integration-style test that gathers metrics from the controller-runtime registry.
controllers/actions.github.com/ephemeralrunnerset_controller.go Stops publishing ephemeral runner count metrics from the ERS reconciler and simplifies ERS status to Phase only.
controllers/actions.github.com/ephemeralrunnerset_controller_test.go Updates tests to assert runner counts by listing EphemeralRunner objects instead of using ERS status fields.
controllers/actions.github.com/ephemeralrunner_lifecycle.go Adds a shared lifecycle bucketing helper with deletion-timestamp precedence.
controllers/actions.github.com/ephemeralrunner_lifecycle_test.go Adds thorough unit tests for lifecycle bucketing precedence and totals.
controllers/actions.github.com/ephemeralrunner_controller.go Emits lifecycle metrics on runner lifecycle/status changes and adds a PublishMetrics gate to the reconciler.
controllers/actions.github.com/autoscalingrunnerset_controller.go Replaces “active runners” check based on ERS status fields with a list+aggregate of EphemeralRunners.
controllers/actions.github.com/autoscalingrunnerset_controller_test.go Updates tests to stop asserting autoscaling status runner-count fields.
config/crd/bases/actions.github.com_ephemeralrunnersets.yaml Removes ERS status count fields/printcolumns from the CRD schema.
config/crd/bases/actions.github.com_autoscalingrunnersets.yaml Removes ARS status count fields/printcolumns from the CRD schema.
charts/gha-runner-scale-set-controller/crds/actions.github.com_ephemeralrunnersets.yaml Mirrors the ERS CRD schema changes in the Helm chart CRDs.
charts/gha-runner-scale-set-controller/crds/actions.github.com_autoscalingrunnersets.yaml Mirrors the ARS CRD schema changes in the Helm chart CRDs.
charts/gha-runner-scale-set-controller-experimental/crds/actions.github.com_ephemeralrunnersets.yaml Mirrors the ERS CRD schema changes in the experimental chart CRDs.
charts/gha-runner-scale-set-controller-experimental/crds/actions.github.com_autoscalingrunnersets.yaml Mirrors the ARS CRD schema changes in the experimental chart CRDs.
apis/actions.github.com/v1alpha1/ephemeralrunnerset_types.go Removes ERS status count fields and associated kubebuilder printcolumns.
apis/actions.github.com/v1alpha1/autoscalingrunnerset_types.go Removes ARS status count fields and associated kubebuilder printcolumns.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread controllers/actions.github.com/ephemeralrunner_controller.go
Comment thread controllers/actions.github.com/ephemeralrunner_controller.go Outdated
Comment thread controllers/actions.github.com/ephemeralrunner_controller.go Outdated
Comment thread controllers/actions.github.com/metrics/metrics.go Outdated
Comment thread controllers/actions.github.com/metrics/metrics.go
@nikola-jokic nikola-jokic force-pushed the nikola-jokic/ephemeral-runner-stats-to-metrics branch 5 times, most recently from f1ca790 to 7b50f20 Compare July 10, 2026 10:19
@nikola-jokic nikola-jokic force-pushed the nikola-jokic/ephemeral-runner-stats-to-metrics branch from 7b50f20 to dbdad1b Compare July 10, 2026 20:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

controllers/actions.github.com/autoscalingrunnerset_controller.go:304

  • The previous logic that deferred EphemeralRunnerSet spec updates while runners were pending/running (and instead deleted the listener to drain jobs) has been removed. With this change, spec updates are applied immediately on integrity hash change, which can cause in-flight jobs to be disrupted if the EphemeralRunnerSet reacts to spec changes by replacing/deleting runners. Consider restoring a drain gate by checking for active runners via EphemeralRunner objects/Pods (or another reliable source) before patching the EphemeralRunnerSet, and only proceed once the set is drained.
		if ephemeralRunnerSet.Annotations[annotationKeyIntegrityHash] != desired.Annotations[annotationKeyIntegrityHash] {
			original := ephemeralRunnerSet.DeepCopy()
			ephemeralRunnerSet.Spec.EphemeralRunnerMetadata = desired.Spec.EphemeralRunnerMetadata
			ephemeralRunnerSet.Spec.EphemeralRunnerSpec = desired.Spec.EphemeralRunnerSpec
			ephemeralRunnerSet.Labels = r.filterAndMergeLabels(ephemeralRunnerSet.Labels, desired.Labels)
			ephemeralRunnerSet.Annotations = r.mergeAnnotations(ephemeralRunnerSet.Annotations, desired.Annotations)

			log.Info("Updating ephemeral runner set spec to match the desired spec")
			if err := r.Patch(ctx, &ephemeralRunnerSet, client.MergeFrom(original)); err != nil {
				log.Error(err, "Failed to patch ephemeral runner set to match the desired spec")
				return ctrl.Result{}, err

Comment on lines 24 to 30
type EphemeralRunnerSetSpec struct {
// Replicas is the number of desired EphemeralRunner resources in the k8s namespace.
// +optional
Replicas int `json:"replicas,omitempty"`
// PatchID is the unique identifier for the patch issued by the listener app
// +optional
PatchID int `json:"patchID"`
// EphemeralRunnerSpec is the spec of the ephemeral runner
// +optional
EphemeralRunnerSpec EphemeralRunnerSpec `json:"ephemeralRunnerSpec,omitempty"`
Comment on lines +891 to +905
commonLabels, err := ephemeralRunnerMetricLabels(ephemeralRunner)
if err != nil {
log.Error(err, "Failed to build ephemeral runner metric labels")
return
}

key := types.NamespacedName{Namespace: ephemeralRunner.Namespace, Name: ephemeralRunner.Name}

ephemeralRunnerPhaseMetrics.Lock()
defer ephemeralRunnerPhaseMetrics.Unlock()

previousPhase, ok := ephemeralRunnerPhaseMetrics.phases[key]
if ok && previousPhase == phase {
return
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be addressed in the follow up PR that introduces caching

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants