Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion airflow-core/docs/migrations-ref.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,9 @@ Here's the list of all the Database Migrations that are executed via when you ru
+-------------------------+------------------+-------------------+--------------------------------------------------------------+
| Revision ID | Revises ID | Airflow Version | Description |
+=========================+==================+===================+==============================================================+
| ``d2f4e1b3c5a7`` (head) | ``9ff64e1c35d3`` | ``3.3.0`` | Add partition_date to asset_partition_dag_run. |
| ``c4e7a1f9b2d0`` (head) | ``d2f4e1b3c5a7`` | ``3.3.0`` | Add index on asset.uri. |

@pierrejeambrun pierrejeambrun Jul 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Target version needs to be updated, 3.3.0 was just released, that would be for 3.4.0

+-------------------------+------------------+-------------------+--------------------------------------------------------------+
| ``d2f4e1b3c5a7`` | ``9ff64e1c35d3`` | ``3.3.0`` | Add partition_date to asset_partition_dag_run. |
+-------------------------+------------------+-------------------+--------------------------------------------------------------+
| ``9ff64e1c35d3`` | ``dd5f3a8e2b91`` | ``3.3.0`` | Add indexes on dag_run.created_dag_version_id and |
| | | | task_instance.dag_version_id. |
Expand Down
1 change: 1 addition & 0 deletions airflow-core/newsfragments/69489.improvement.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add an indexed exact-match ``uri`` query parameter to ``GET /api/v2/assets`` for fast single-asset lookup by URI (much faster than ``uri_pattern``, which uses an unindexed ``ILIKE '%...%'``).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be removed. (I know doc mentions that user facing change should have a fragment, but that's only true for significant / behavior change that needs a user warning basically).

16 changes: 16 additions & 0 deletions airflow-core/src/airflow/api_fastapi/common/parameters.py
Original file line number Diff line number Diff line change
Expand Up @@ -1578,6 +1578,22 @@ def _transform_ti_states(states: list[str] | None) -> list[TaskInstanceState | N
QueryUriPrefixPatternSearch = Annotated[
_PrefixSearchParam, Depends(prefix_search_param_factory(AssetModel.uri, "uri_prefix_pattern"))
]
QueryUriExactMatch = Annotated[
FilterParam[str | None],
Depends(
filter_param_factory(
AssetModel.uri,
str | None,
filter_name="uri",
description=(
"Exact-match filter on the full asset URI. Compiles to an indexed equality "
Comment on lines +1581 to +1589

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make this filter accept multiple values.

To allow ?uri=some_uri1&uri=some_uri2 to get both assets with those URI. (And it makes more sense to me for a list endpoint to accept multiple values, otherwise it's really close to a single GET endpoint and maybe we should instead have GET assets/{uri}, but lets not go there)

It's really a simple change, as an exemple for implementation:

QueryTIStateFilter = Annotated[
    FilterParam[list[str]],
    Depends(
        filter_param_factory(
            TaskInstance.state,
            list[str],
            FilterOptionEnum.ANY_EQUAL,
            default_factory=list,
            transform_callable=_transform_ti_states,
        )
    ),
]

"comparison (``uri = ...``), so it is far faster than ``uri_pattern`` (which uses "
"``ILIKE '%...%'`` and cannot use an index) for resolving a single asset by its "
"known URI."
),
)
),
]
Comment on lines +1581 to +1596

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove the piece that compares to search params. They have their own doc explaining the performance pitfalls, this can stay about 'exact match'.

Suggested change
QueryUriExactMatch = Annotated[
FilterParam[str | None],
Depends(
filter_param_factory(
AssetModel.uri,
str | None,
filter_name="uri",
description=(
"Exact-match filter on the full asset URI. Compiles to an indexed equality "
"comparison (``uri = ...``), so it is far faster than ``uri_pattern`` (which uses "
"``ILIKE '%...%'`` and cannot use an index) for resolving a single asset by its "
"known URI."
),
)
),
]
QueryUriExactMatch = Annotated[
FilterParam[str | None],
Depends(
filter_param_factory(
AssetModel.uri,
str | None,
filter_name="uri",
description=(
"Exact-match filter on the full asset URI. Compiles to an indexed equality "
"comparison (``uri = ...``).
),
)
),
]

QueryAssetAliasNamePatternSearch = Annotated[
_SearchParam, Depends(search_param_factory(AssetAliasModel.name, "name_pattern"))
]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,22 @@ paths:
\ stays index-compatible under locale-aware collations \u2014 e.g. `test_`\
\ effectively matches items starting with `test`, and `s3://` matches items\
\ starting with `s3`."
- name: uri
in: query
required: false
schema:
anyOf:
- type: string
- type: 'null'
description: Exact-match filter on the full asset URI. Compiles to an indexed
equality comparison (``uri = ...``), so it is far faster than ``uri_pattern``
(which uses ``ILIKE '%...%'`` and cannot use an index) for resolving a
single asset by its known URI.
title: Uri
description: Exact-match filter on the full asset URI. Compiles to an indexed
equality comparison (``uri = ...``), so it is far faster than ``uri_pattern``
(which uses ``ILIKE '%...%'`` and cannot use an index) for resolving a single
asset by its known URI.
- name: uri_pattern
in: query
required: false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
QueryAssetNamePrefixPatternSearch,
QueryLimit,
QueryOffset,
QueryUriExactMatch,
QueryUriPatternSearch,
QueryUriPrefixPatternSearch,
RangeFilter,
Expand Down Expand Up @@ -139,6 +140,7 @@ def get_assets(
offset: QueryOffset,
name_pattern: QueryAssetNamePatternSearch,
name_prefix_pattern: QueryAssetNamePrefixPatternSearch,
uri: QueryUriExactMatch,
uri_pattern: QueryUriPatternSearch,
uri_prefix_pattern: QueryUriPrefixPatternSearch,
dag_ids: QueryAssetDagIdPatternSearch,
Expand Down Expand Up @@ -184,7 +186,15 @@ def get_assets(

assets_select, total_entries = paginated_select(
statement=assets_select_statement,
filters=[only_active, name_pattern, name_prefix_pattern, uri_pattern, uri_prefix_pattern, dag_ids],
filters=[
only_active,
name_pattern,
name_prefix_pattern,
uri,
uri_pattern,
uri_prefix_pattern,
dag_ids,
],
order_by=order_by,
offset=offset,
limit=limit,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Add index on asset.uri.

Revision ID: c4e7a1f9b2d0
Revises: d2f4e1b3c5a7
Create Date: 2026-07-06 00:00:00.000000
"""

from __future__ import annotations

from alembic import op

# revision identifiers, used by Alembic.
revision = "c4e7a1f9b2d0"
down_revision = "d2f4e1b3c5a7"
branch_labels = None
depends_on = None
airflow_version = "3.3.0"


def upgrade():
"""Apply Add index on asset.uri."""
with op.batch_alter_table("asset", schema=None) as batch_op:
batch_op.create_index("idx_asset_uri", ["uri"], unique=False)


def downgrade():
"""Unapply Add index on asset.uri."""
with op.batch_alter_table("asset", schema=None) as batch_op:
batch_op.drop_index("idx_asset_uri")
Comment on lines +37 to +47

@pierrejeambrun pierrejeambrun Jul 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fine to have a 'filter/order_by' in the API that isn't backed by an index. Indeed it will be slow if the table grows out of control but does the table grows out of control in most common cases? I would avoid adding an index (costly) just for a single filter of a single endpoint.
I'm not sure here but I would say the table isn't huge most of the time. (millions of assets) and no index is probably fine for most? (the index probably wasn't there in 2.x as well)

If someone happen to have a huge asset table they can add their own index to improve performance following https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/performance.rst

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just checked, in 2.x there was an index indeed, so the table was worth indexing. Disregard my previous comment, this makes sense.

3 changes: 3 additions & 0 deletions airflow-core/src/airflow/models/asset.py
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,9 @@ class AssetModel(Base):
__tablename__ = "asset"
__table_args__ = (
Index("idx_asset_name_uri_unique", name, uri, unique=True),
# Single-column index so exact-match lookups by URI (GET /assets?uri=...) can use an
# index; the composite index above leads with ``name`` and cannot serve uri-only queries.
Comment on lines +336 to +337

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove this, to not mention the API layer in ORM models.

Index("idx_asset_uri", uri),
{"sqlite_autoincrement": True}, # ensures PK values not reused
)

Expand Down
5 changes: 3 additions & 2 deletions airflow-core/src/airflow/ui/openapi-gen/queries/common.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,18 @@ import { DagRunState, DagWarningType } from "../requests/types.gen";
export type AssetServiceGetAssetsDefaultResponse = Awaited<ReturnType<typeof AssetService.getAssets>>;
export type AssetServiceGetAssetsQueryResult<TData = AssetServiceGetAssetsDefaultResponse, TError = unknown> = UseQueryResult<TData, TError>;
export const useAssetServiceGetAssetsKey = "AssetServiceGetAssets";
export const UseAssetServiceGetAssetsKeyFn = ({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }: {
export const UseAssetServiceGetAssetsKeyFn = ({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }: {
dagIds?: string[];
limit?: number;
namePattern?: string;
namePrefixPattern?: string;
offset?: number;
onlyActive?: boolean;
orderBy?: string[];
uri?: string;
uriPattern?: string;
uriPrefixPattern?: string;
} = {}, queryKey?: Array<unknown>) => [useAssetServiceGetAssetsKey, ...(queryKey ?? [{ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }])];
} = {}, queryKey?: Array<unknown>) => [useAssetServiceGetAssetsKey, ...(queryKey ?? [{ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }])];
export type AssetServiceGetAssetAliasesDefaultResponse = Awaited<ReturnType<typeof AssetService.getAssetAliases>>;
export type AssetServiceGetAssetAliasesQueryResult<TData = AssetServiceGetAssetAliasesDefaultResponse, TError = unknown> = UseQueryResult<TData, TError>;
export const useAssetServiceGetAssetAliasesKey = "AssetServiceGetAssetAliases";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import * as Common from "./common";
*
* **Performance note:** this full-match pattern is evaluated as ``ILIKE '%term%'`` and most of the time prevents the database from using B-tree indexes, which can be very slow on large tables. Prefer the equivalent ``name_prefix_pattern`` parameter when possible.
* @param data.namePrefixPattern Prefix match — returns items whose value starts with the given string (case-sensitive, index-friendly). Use the pipe `|` operator for OR logic (e.g. `dag1|dag2`). Use `~` to match all. Wildcard characters (`%`, `_`) are treated as literal characters. Trailing non-alphanumeric characters in the prefix are stripped before matching so the range scan stays index-compatible under locale-aware collations — e.g. `test_` effectively matches items starting with `test`, and `s3://` matches items starting with `s3`.
* @param data.uri Exact-match filter on the full asset URI. Compiles to an indexed equality comparison (``uri = ...``), so it is far faster than ``uri_pattern`` (which uses ``ILIKE '%...%'`` and cannot use an index) for resolving a single asset by its known URI.
* @param data.uriPattern SQL LIKE expression — use `%` / `_` wildcards (e.g. `%customer_%`). Use the pipe `|` operator for OR logic (e.g. `dag1 | dag2`). Regular expressions are **not** supported.
*
* **Performance note:** this full-match pattern is evaluated as ``ILIKE '%term%'`` and most of the time prevents the database from using B-tree indexes, which can be very slow on large tables. Prefer the equivalent ``uri_prefix_pattern`` parameter when possible.
Expand All @@ -24,17 +25,18 @@ import * as Common from "./common";
* @returns AssetCollectionResponse Successful Response
* @throws ApiError
*/
export const ensureUseAssetServiceGetAssetsData = (queryClient: QueryClient, { dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }: {
export const ensureUseAssetServiceGetAssetsData = (queryClient: QueryClient, { dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }: {
dagIds?: string[];
limit?: number;
namePattern?: string;
namePrefixPattern?: string;
offset?: number;
onlyActive?: boolean;
orderBy?: string[];
uri?: string;
uriPattern?: string;
uriPrefixPattern?: string;
} = {}) => queryClient.ensureQueryData({ queryKey: Common.UseAssetServiceGetAssetsKeyFn({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }), queryFn: () => AssetService.getAssets({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }) });
} = {}) => queryClient.ensureQueryData({ queryKey: Common.UseAssetServiceGetAssetsKeyFn({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }), queryFn: () => AssetService.getAssets({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }) });
/**
* Get Asset Aliases
* Get asset aliases.
Expand Down
6 changes: 4 additions & 2 deletions airflow-core/src/airflow/ui/openapi-gen/queries/prefetch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import * as Common from "./common";
*
* **Performance note:** this full-match pattern is evaluated as ``ILIKE '%term%'`` and most of the time prevents the database from using B-tree indexes, which can be very slow on large tables. Prefer the equivalent ``name_prefix_pattern`` parameter when possible.
* @param data.namePrefixPattern Prefix match — returns items whose value starts with the given string (case-sensitive, index-friendly). Use the pipe `|` operator for OR logic (e.g. `dag1|dag2`). Use `~` to match all. Wildcard characters (`%`, `_`) are treated as literal characters. Trailing non-alphanumeric characters in the prefix are stripped before matching so the range scan stays index-compatible under locale-aware collations — e.g. `test_` effectively matches items starting with `test`, and `s3://` matches items starting with `s3`.
* @param data.uri Exact-match filter on the full asset URI. Compiles to an indexed equality comparison (``uri = ...``), so it is far faster than ``uri_pattern`` (which uses ``ILIKE '%...%'`` and cannot use an index) for resolving a single asset by its known URI.
* @param data.uriPattern SQL LIKE expression — use `%` / `_` wildcards (e.g. `%customer_%`). Use the pipe `|` operator for OR logic (e.g. `dag1 | dag2`). Regular expressions are **not** supported.
*
* **Performance note:** this full-match pattern is evaluated as ``ILIKE '%term%'`` and most of the time prevents the database from using B-tree indexes, which can be very slow on large tables. Prefer the equivalent ``uri_prefix_pattern`` parameter when possible.
Expand All @@ -24,17 +25,18 @@ import * as Common from "./common";
* @returns AssetCollectionResponse Successful Response
* @throws ApiError
*/
export const prefetchUseAssetServiceGetAssets = (queryClient: QueryClient, { dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }: {
export const prefetchUseAssetServiceGetAssets = (queryClient: QueryClient, { dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }: {
dagIds?: string[];
limit?: number;
namePattern?: string;
namePrefixPattern?: string;
offset?: number;
onlyActive?: boolean;
orderBy?: string[];
uri?: string;
uriPattern?: string;
uriPrefixPattern?: string;
} = {}) => queryClient.prefetchQuery({ queryKey: Common.UseAssetServiceGetAssetsKeyFn({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }), queryFn: () => AssetService.getAssets({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }) });
} = {}) => queryClient.prefetchQuery({ queryKey: Common.UseAssetServiceGetAssetsKeyFn({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }), queryFn: () => AssetService.getAssets({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }) });
/**
* Get Asset Aliases
* Get asset aliases.
Expand Down
6 changes: 4 additions & 2 deletions airflow-core/src/airflow/ui/openapi-gen/queries/queries.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import * as Common from "./common";
*
* **Performance note:** this full-match pattern is evaluated as ``ILIKE '%term%'`` and most of the time prevents the database from using B-tree indexes, which can be very slow on large tables. Prefer the equivalent ``name_prefix_pattern`` parameter when possible.
* @param data.namePrefixPattern Prefix match — returns items whose value starts with the given string (case-sensitive, index-friendly). Use the pipe `|` operator for OR logic (e.g. `dag1|dag2`). Use `~` to match all. Wildcard characters (`%`, `_`) are treated as literal characters. Trailing non-alphanumeric characters in the prefix are stripped before matching so the range scan stays index-compatible under locale-aware collations — e.g. `test_` effectively matches items starting with `test`, and `s3://` matches items starting with `s3`.
* @param data.uri Exact-match filter on the full asset URI. Compiles to an indexed equality comparison (``uri = ...``), so it is far faster than ``uri_pattern`` (which uses ``ILIKE '%...%'`` and cannot use an index) for resolving a single asset by its known URI.
* @param data.uriPattern SQL LIKE expression — use `%` / `_` wildcards (e.g. `%customer_%`). Use the pipe `|` operator for OR logic (e.g. `dag1 | dag2`). Regular expressions are **not** supported.
*
* **Performance note:** this full-match pattern is evaluated as ``ILIKE '%term%'`` and most of the time prevents the database from using B-tree indexes, which can be very slow on large tables. Prefer the equivalent ``uri_prefix_pattern`` parameter when possible.
Expand All @@ -24,17 +25,18 @@ import * as Common from "./common";
* @returns AssetCollectionResponse Successful Response
* @throws ApiError
*/
export const useAssetServiceGetAssets = <TData = Common.AssetServiceGetAssetsDefaultResponse, TError = unknown, TQueryKey extends Array<unknown> = unknown[]>({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }: {
export const useAssetServiceGetAssets = <TData = Common.AssetServiceGetAssetsDefaultResponse, TError = unknown, TQueryKey extends Array<unknown> = unknown[]>({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }: {
dagIds?: string[];
limit?: number;
namePattern?: string;
namePrefixPattern?: string;
offset?: number;
onlyActive?: boolean;
orderBy?: string[];
uri?: string;
uriPattern?: string;
uriPrefixPattern?: string;
} = {}, queryKey?: TQueryKey, options?: Omit<UseQueryOptions<TData, TError>, "queryKey" | "queryFn">) => useQuery<TData, TError>({ queryKey: Common.UseAssetServiceGetAssetsKeyFn({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }, queryKey), queryFn: () => AssetService.getAssets({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uriPattern, uriPrefixPattern }) as TData, ...options });
} = {}, queryKey?: TQueryKey, options?: Omit<UseQueryOptions<TData, TError>, "queryKey" | "queryFn">) => useQuery<TData, TError>({ queryKey: Common.UseAssetServiceGetAssetsKeyFn({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }, queryKey), queryFn: () => AssetService.getAssets({ dagIds, limit, namePattern, namePrefixPattern, offset, onlyActive, orderBy, uri, uriPattern, uriPrefixPattern }) as TData, ...options });
/**
* Get Asset Aliases
* Get asset aliases.
Expand Down
Loading
Loading