Skip to content

Flag missing conn-fields when hook-only files change in static checks#70007

Merged
eladkal merged 88 commits into
apache:mainfrom
dabla:feature/flag-missing-conn-fields
Jul 22, 2026
Merged

Flag missing conn-fields when hook-only files change in static checks#70007
eladkal merged 88 commits into
apache:mainfrom
dabla:feature/flag-missing-conn-fields

Conversation

@dabla

@dabla dabla commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Currently, the check-provider-yaml-valid prek hook only fires when a
provider.yaml file is modified. When a developer adds a new field to
get_connection_form_widgets() in a hook Python file — as in PR #64643
which added auth_protocol to SambaHook — the static check is never
triggered, so the mismatch between the hook and provider.yaml goes
undetected.

There was also a second gap: even when the check did run, a
connection-type entry with no conn-fields section at all was
silently skipped, regardless of how many form widgets the hook declared.

This PR closes both gaps:

1. Trigger on hook file changes

check-provider-yaml-valid in providers/.pre-commit-config.yaml now
also fires when any file matching
src/airflow/providers/**/hooks/*.py is staged.
check_provider_yaml_files.py maps each changed hook file to the
provider.yaml at the root of the same provider package and passes
that to the in-container validator, so the full conn-fields check runs
even when only the hook changes.

2. Flag absent conn-fields when the hook declares widgets

check_conn_fields_for_entry() previously returned early with no error
when conn-fields was absent from provider.yaml. It now treats an
absent conn-fields section as an empty dict, so every field returned
by get_connection_form_widgets() is reported as missing — exactly the
same error the existing bidirectional check already produces for a
present-but-incomplete conn-fields section.

Changes

File What changed
scripts/ci/prek/check_provider_conn_fields.py Treat absent conn-fields as {} instead of short-circuiting
scripts/ci/prek/check_provider_yaml_files.py Map hook Python files → provider.yaml before delegating to Breeze
providers/.pre-commit-config.yaml Expand trigger to include hooks/*.py files
scripts/tests/ci/prek/test_check_conn_fields_match_form_widgets.py Replace the "no-conn-fields-with-hook-widgets-is-ok" passing case with a failing case; add absent-conn-fields-with-hook-widgets error test

Reproduction

Without this fix, the following scenario produces no CI error:

# provider.yaml — no conn-fields section at all
connection-types:
  - hook-class-name: airflow.providers.samba.hooks.samba.SambaHook
    connection-type: samba
# samba.py
@classmethod
def get_connection_form_widgets(cls):
    return {"auth_protocol": SelectField(...)}  # invisible in React UI!

With this fix, the static check reports:

Mismatch between conn-fields in providers/samba/provider.yaml and
SambaHook.get_connection_form_widgets() for connection-type 'samba':
  Fields in get_connection_form_widgets() but NOT in provider.yaml conn-fields: auth_protocol
How to fix it: Add the missing key(s) to conn-fields in provider.yaml.

Was generative AI tooling used to co-author this PR?
  • [ x ] Yes (please specify the tool below)

Claude Sonnet 4.6


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@dabla
dabla force-pushed the feature/flag-missing-conn-fields branch from a44e38a to c94cf31 Compare July 17, 2026 14:15
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

uv.lock on main just moved via #70087 ("Add AWS services toolset for agents to access 1000+ APIs"), commit eddaa67 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

DjVinnii and others added 14 commits July 21, 2026 15:37
* Add missing  Dutch translations
…ache#69881)

When durable=True, tools supplied via a pydantic-ai Toolset capability
(capabilities=[Toolset(ts)]) bypassed the CachingToolset applied to
toolsets=, so their results re-executed on every retry instead of
replaying. Wrap the inner toolset of concrete Toolset capabilities with
the same CachingToolset. A Toolset capability backed by a callable
factory can't be wrapped (it resolves per run) and now logs a warning.

Also fixes two latent durable-execution bugs found in review:

- cleanup() ran before the message-history XCom push and output
  serialization. A failure in either wiped the cache, so the retry
  re-ran every already-completed step. Move cleanup to after them.

- the pre-3.3 ObjectStorage cache filename joined dag/task/run with "_",
  aliasing distinct tasks (dag "etl"/task "load_data" and dag
  "etl_load"/task "data" both mapped to one file, so one task could
  read, overwrite, or delete another's cache). Hash the identity
  components instead.
)

`DagBundlesManager.sync_bundles_to_db()` marks every stored bundle that is
not in the calling process's config as inactive. This assumes the caller
sees the complete bundle configuration.

When multiple dag-processors are each configured with a partial config (one
bundle per processor, a natural way to shard parsing), each processor treats
the other processors' bundles as "no longer found in config" and disables
them. Processor A disables B's bundle, B disables A's, and they flip
`dag_bundle.active` on every parse cycle -- the deployment never converges
and continuously logs "DAG bundle ... is no longer found in config and has
been disabled" for bundles that are actively configured elsewhere.

Add a `deactivate_missing` flag (default `True`, preserving existing
single-processor behavior) and have `DagFileProcessorManager.sync_bundles()`
pass `deactivate_missing=False` when the processor was started with a bundle
filter (`--bundle-name` / `bundle_names_to_parse`), i.e. when it only owns a
subset of the bundles.

closes: apache#69963
related: apache#69698

Generated-by: Claude Code following the guidelines
…ag_id (apache#69471)

* Authorize Dag reparse against the file's Dags, not the query-string dag_id

The PUT /parseDagFile/{file_token} endpoint authorized the caller through a route dependency that, with no dag_id path parameter, resolves the target from the query string, which is decoupled from the Dag file the signed file_token actually resolves to.

Add a requires_access_dag_from_file_token dependency that decodes the token, resolves the Dags defined in that file, and authorizes the caller against exactly those Dags, so authorization always matches the Dag being reparsed regardless of any request parameter.

* Update airflow-core/src/airflow/api_fastapi/core_api/security.py

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>

* Fix CI

---------

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
…he#68760)

* Restore graph task and group coloring via Chakra palette tokens

Custom operator and TaskGroup colors disappeared in Airflow 3: the
ui_color/ui_fgcolor attributes were kept but the new UI ignored them, so
teams that relied on color to parse large graphs at a glance lost that
signal (see the demand on the linked issue).

Bring the coloring back in a way that fits the new opinionated theming:
the value must be a Chakra palette token (e.g. blue.500), which resolves
through a theme-controlled CSS variable and therefore stays legible in
both light and dark mode and under custom themes -- the dark-mode and
accessibility concerns that drove the original removal. The graph paints
the node fill from ui_color and the operator label from ui_fgcolor, as in
Airflow 2.x, while the border keeps showing run state. Legacy hex/named
values can no longer adapt to the theme, so they are ignored by the graph
and emit a warning for user-authored operators and task groups.

* Color task groups in the graph view, including when expanded

ui_color and ui_fgcolor are documented as the fill and label colors of a
TaskGroup node, and Airflow 2.x painted them, but the graph excluded groups
from the custom fill and dropped the colors entirely once a group was
expanded. Treat groups like leaf tasks so a token-valued group color renders
its fill and label in both the collapsed and expanded states.

* Accept raw hex colors and Chakra tokens for graph node coloring

Review feedback asked to keep the Airflow 2 behavior where ui_color and ui_fgcolor accept a raw color (a hex code or CSS name) in addition to Chakra palette tokens, so existing Dags keep their graph colors without any change. Dark-mode color variants are deferred to a follow-up.

* Adjustments

* Fix unreadable graph task text on light custom colors in dark mode

The task icon and label fell back to the theme's default foreground for Chakra token fills, so a light token such as yellow.200 rendered light text on a light fill in dark mode. Deriving the text color from the resolved fill measures tokens too, so contrast holds for hex, CSS names, and tokens alike.

* Update task group graph test fixture for restored node colors

The graph serializer now emits each node's ui_color/ui_fgcolor for task
and group nodes, so the expected graph fixture must include them; join
nodes stay uncolored. The fixture was stale and omitted these fields,
failing test_task_group_to_dict_serialized_dag and
test_task_group_to_dict_alternative_syntax.

* Document semantic theme tokens for ui_color and ui_fgcolor

The graph node color attributes accept theme-aware semantic tokens (e.g.
brand.solid) that adapt to light and dark mode, in addition to palette tokens
and raw colors — but the docs only mentioned palette tokens, so users had no
pointer to the semantic tokens available in the UI theme.
go-sdk/*.md edits (README, ADRs) currently match the Go SDK file groups
and trigger the Go unit + e2e suites plus a prod image build, none of
which a documentation change can affect. Excluding .md also lets
non-.go build files (go.mod, go.sum, build config) trigger the unit
tests, which the previous .go-only pattern missed.
ts-sdk/*.md edits (e.g. README) currently match TS_SDK_FILES and keep
the check-ts-sdk-supervisor-schema prek hook from being skipped, so a
documentation-only change needlessly regenerates and diffs the
generated supervisor schema. Documentation does not affect it.
…ache#69977)

Under concurrent asset-event fan-out, the per-row SAVEPOINT loop used for
non-PostgreSQL backends deadlocks on MySQL/InnoDB and surfaces confusing
"SAVEPOINT ... does not exist" errors, and the deadlock is not retried.
Route MySQL to a single-statement INSERT ... ON DUPLICATE KEY UPDATE,
mirroring the existing PostgreSQL path, so the enqueue is atomic and holds
locks far more briefly.

closes: apache#69938
A mid-run DAG re-parse possibly creates a new DagVersion and bumps
not-yet-started task instances' dag_version_id, while the DagRun
stays pinned to its original bundle_version. ExecuteTask.make()
sourced the bundle version from the run's pin (dag_run.bundle_version)
but bundle version_data from the task's now-bumped dag_version,
so the workload shipped a hash and a manifest that describe
different versions. This double source was noticed for the callback
path in a previous PR, but the task path was missed.

Source version_data from the run's pinned created_dag_version so it always
matches the bundle version, mirroring ExecuteCallback.make() and the way Git
bundles resolve everything from the run's pinned commit.
* Allow configuring XCom sidecar container security context

* fix CI error

* fix CI error apache#2

* Update providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py

Co-authored-by: Przemysław Mirowski <17602603+Miretpl@users.noreply.github.com>

* fix CI error

---------

Co-authored-by: Przemysław Mirowski <17602603+Miretpl@users.noreply.github.com>
* Remove duplicated newsfragments

* Add missing newsfragments

* Consolidate Airflow version newsfragment
@eladkal
eladkal merged commit ffe49c7 into apache:main Jul 22, 2026
70 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test PR Link

dabla added a commit to dabla/airflow that referenced this pull request Jul 22, 2026
…hange

When only `scripts/ci/prek/check_provider_yaml_files.py` or
`providers/.pre-commit-config.yaml` changed, selective checks incorrectly
skipped `check-provider-yaml-valid` because those paths matched neither
`ALL_PROVIDERS_DISTRIBUTION_CONFIG_FILES` nor `ALL_PROVIDERS_PYTHON_FILES`.
This meant a bug introduced in the check script itself (like the path
resolution bug in apache#70007) would go undetected in CI.

Fix by:
- Including `providers/.pre-commit-config.yaml` in
  `ALL_PROVIDERS_DISTRIBUTION_CONFIG_FILES`
- Adding `PREK_FILES` as a third condition that keeps
  `check-provider-yaml-valid` from being skipped

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.