Optionally emit Dag tags as metric tags#68568
Open
sortega wants to merge 3 commits into
Open
Conversation
sortega
force-pushed
the
sortega/airflow-tagged-metrics
branch
from
80ec8c9 to
fe514c9
Compare
sortega
requested review from
XD-DENG,
amoghrajesh,
ashb,
kaxil and
potiuk
as code owners
ashb
reviewed
Jun 18, 2026
ashb
left a comment
ashb
left a comment
Member
There was a problem hiding this comment.
the behavior is gated behind a new [metrics] dag_tags_in_metrics option, disabled by default, to avoid surprise cardinality increases on existing installations
Perfect, I was thinking the same thing as I read the description.
Overall a good PR though
sortega
force-pushed
the
sortega/airflow-tagged-metrics
branch
3 times, most recently
from
36f5aca to
82d305f
Compare
hussein-awala
left a comment
hussein-awala
left a comment
Member
There was a problem hiding this comment.
Thanks for this, I like the idea, being able to slice metrics by dag tags (team/env/criticality) is genuinely useful.
I left a few inline comments
sortega
force-pushed
the
sortega/airflow-tagged-metrics
branch
from
82d305f to
2e27387
Compare
Member
|
@sortega — this PR has 6 unresolved review thread(s) that still look like they need your attention. Once you've addressed them (push changes and/or reply in-thread), please resolve the threads or reply to confirm, and give the reviewer a nudge for another look. Thanks! See the PR quality criteria. Automated first-pass triage note drafted by an AI-assisted tool — may get things wrong; once addressed, a real Apache Airflow maintainer takes the next look. (why automated) Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting |
Contributor
Author
|
I've addressed the feedback. @ashb, @hussein-awala could you take another look, please? |
sortega
force-pushed
the
sortega/airflow-tagged-metrics
branch
5 times, most recently
from
dbaff3b to
be52464
Compare
sortega
force-pushed
the
sortega/airflow-tagged-metrics
branch
from
be52464 to
666fcb5
Compare
potiuk
added
the
ready for maintainer review
ashb
approved these changes
Jul 7, 2026
ashb
left a comment
ashb
left a comment
Member
There was a problem hiding this comment.
A few nits and small things, but close.
Please ping me again on Slack when you've had another pass.
sortega
force-pushed
the
sortega/airflow-tagged-metrics
branch
3 times, most recently
from
4256096 to
58b977b
Compare
sortega
force-pushed
the
sortega/airflow-tagged-metrics
branch
3 times, most recently
from
efd4519 to
6831198
Compare
ashb
changed the title
ashb
reviewed
Jul 14, 2026
ashb
reviewed
Jul 14, 2026
sortega
force-pushed
the
sortega/airflow-tagged-metrics
branch
from
4aa2660 to
0abe042
Compare
Add a [metrics] dag_tags_in_metrics option (default False). When enabled, each Dag tag becomes a metric tag on Dag-run and task-instance metrics: tags with a colon (e.g. env:prod) split into a key/value pair; plain tags (e.g. production) become standalone DogStatsd tags, or tag=true in InfluxDB line protocol. Built-in keys (dag_id, run_type, task_id, team_name) win on collision. Tags come from the DagRun's dag_model.tags. The scheduler hot loop (get_running_dag_runs_to_examine) and the executor-event failure path eager-load them (gated on the flag) to avoid per-DagRun queries; other in-session emission paths fall back to a lazy load, so the tags reach all Dag-run and task-instance metrics. DagRun.dag_tags_for_stats swallows SQLAlchemyError so a detached/expired instance degrades to no tags rather than breaking the caller. TaskInstance.stats_tags reuses DagRun.dag_tags_for_stats and adds task_id and run_type; the Task SDK worker reads the in-memory Dag and also adds run_type, so ti.* metrics carry a consistent tag set across the worker and scheduler emitters. The build_dag_metric_tags helper lives in the shared observability stats module.
Keep the scheduler hot path free of per-task conf reads. Warm the dag tags via the existing TaskInstance -> DagModel join rather than an extra dag_run hop, since the DagModel is shared in the identity map. Fold the repeated dogstatsd tag-list guard into a single helper, and consolidate the tag-formatting tests.
Make session a keyword-only argument on get_running_dag_runs_to_examine, in line with the convention for session parameters. Reduce TaskInstance.stats_tags to the dag run's tags plus the task-level ones, keeping the TI's transiently resolved team_name when present.
sortega
force-pushed
the
sortega/airflow-tagged-metrics
branch
from
0abe042 to
b312cbb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Airflow's Dag-run and task-instance metrics carry a small fixed set of tags (
dag_id,run_type,task_id, andteam_name). Dag authors already annotate their Dags with free-form tags likeproductionorenv:prod, but those never reach the metrics — so dashboards and alerts can't be sliced by team, environment, or criticality without hardcodingdag_idlists.This PR optionally surfaces each Dag tag as an individual metric tag on all Dag-run and task-instance metrics, across the StatsD, DogStatsD and InfluxDB backends:
:(e.g.env:prod) splits into a key/value pair (env→prod).production) becomes a standalone DogStatsD tag, orproduction=truein InfluxDB line protocol.dag_id,run_type,task_id) always win on collision.Because Dag tags are unbounded, user-defined strings, the behavior is gated behind a new
[metrics] dag_tags_in_metricsoption, disabled by default, to avoid surprise cardinality increases on existing installations. Tags are read from the in-memory serialized Dag, so no extra DB queries are issued and the worker never touches the metastore.This is the metrics counterpart of the goal in #37901 (routing observability by Dag tags), and builds on the
team_namework from #68108.related: #37901
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines