Skip to content

minor: Add metrics to costBased auto-scaler#19631

Merged
kfaraz merged 1 commit into
apache:masterfrom
kfaraz:add_autoscaler_metrics
Jun 26, 2026
Merged

minor: Add metrics to costBased auto-scaler#19631
kfaraz merged 1 commit into
apache:masterfrom
kfaraz:add_autoscaler_metrics

Conversation

@kfaraz

@kfaraz kfaraz commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Changes

  • Add metric task/autoScaler/costBased/avgProcessingRate
  • Add metric task/autoScaler/costBased/avgPollIdleRatio
  • Add metric task/autoScaler/costBased/lagWeight
  • Add metric task/autoScaler/costBased/costWeight
  • Remove metric task/autoScaler/costBased/lagCost since it did not add value in identifying why a decision was taken
  • Remove metric task/autoScaler/costBased/idleCost since it did not add value in identifying why a decision was taken
  • Minor cleanup

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.


At every evaluation interval, Druid computes the score for each candidate task count and picks the one with the lowest total cost.

Note: Kinesis is not supported yet, support is in progress.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested in comment #19378 (comment)

this.scaleDownDuringTaskRolloverOnly = Configs.valueOrDefault(scaleDownDuringTaskRolloverOnly, false);

if (this.enableTaskAutoScaler) {
if (useTaskCountBoundaries != null) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is a relatively new auto-scaler, we need not carry forward discontinued parameters for logging purposes as it adds to clutter.

@kfaraz

kfaraz commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @Fly-Style !

@kfaraz kfaraz merged commit 9c26bb7 into apache:master Jun 26, 2026
39 checks passed
@kfaraz kfaraz deleted the add_autoscaler_metrics branch June 26, 2026 09:33
@github-actions github-actions Bot added this to the 38.0.0 milestone Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants