Providers: Make psycopg (v3) the default synchronous Postgres driver#69526
Open
Dev-iL wants to merge 6 commits into
Open
Providers: Make psycopg (v3) the default synchronous Postgres driver#69526Dev-iL wants to merge 6 commits into
Dev-iL wants to merge 6 commits into
Conversation
Dev-iL
requested review from
dheerajturaga,
hussein-awala,
o-nikolas and
shahar1
as code owners
boring-cyborg
Bot
added
area:providers
kind:documentation
provider:amazon
1 task
Mirrors the async default switch (apache#69089): PostgresHook no longer requires psycopg2 at import time. The psycopg2-specific connection, cursor, and execute_values imports are now lazy/guarded and raise a clear AirflowOptionalProviderFeatureException if psycopg2 genuinely needs to be used but isn't installed. psycopg2-binary moves to a new [psycopg2] extra. Part of the migration tracked in apache#68453.
The hardcoded fallback that derives [celery] result_backend from sql_alchemy_conn when a driverless postgresql:// URL is configured targeted postgresql+psycopg2:// explicitly. Update it to target postgresql+psycopg:// to match the sync Postgres driver default change in apache-airflow and apache-airflow-providers-postgres — but only when psycopg (v3) is actually importable, since providers/celery's own dependency floor doesn't guarantee SQLAlchemy 2.0+/psycopg, falling back to postgresql+psycopg2:// otherwise (the same guard PostgresHook itself uses). Part of the migration tracked in apache#68453.
register_adapter(list/dict, Json) is psycopg2-specific and previously ran unconditionally, forcing psycopg2 to be importable even when the resolved PostgresHook connection is on psycopg3 (a no-op there). Only import and call it when PostgresHook is actually on the psycopg2 path, so the module imports cleanly without psycopg2 installed. Part of the migration tracked in apache#68453.
from pgvector.psycopg2 import register_vector ran unconditionally at module level, forcing psycopg2 to be importable regardless of which driver the underlying PostgresHook connection actually uses, and was always called even when the connection is on psycopg3 — where it's the wrong helper entirely (psycopg3 has its own pgvector.psycopg registration API). Make the import lazy, call the helper matching PostgresHook.USE_PSYCOPG3, and raise a clear AirflowOptionalProviderFeatureException if the needed driver's pgvector integration is missing. Fixes apache#69443. Part of the migration tracked in apache#68453.
RedshiftSQLHook.get_sqlalchemy_engine() built its SQLAlchemy engine from a bare postgresql:// URI, relying on SQLAlchemy's implicit default of the psycopg2 DB-API. Now that apache-airflow-providers-postgres no longer bundles psycopg2-binary by default, that import fails wherever psycopg2 genuinely isn't installed. Resolve an explicit driver instead: prefer psycopg (v3), fall back to psycopg2, and raise a clear error only if neither is installed. Also updates the Batch, ECS, and Lambda executor setup guides' example postgresql+psycopg2:// connection strings to postgresql+psycopg://, since those stale examples would otherwise break for anyone following them as-is under the new default. Part of the migration tracked in apache#68453.
SQLAlchemy resolves a bare postgresql:// scheme to the psycopg2 DB-API by default, regardless of what Airflow's own config does. Any DbApiHook subclass whose connection type maps to a bare postgresql:// URI (rather than delegating to PostgresHook's own driver-aware sqlalchemy_url) broke once apache-airflow-providers-postgres stopped hard-requiring psycopg2. get_sqlalchemy_engine() now retries with an explicit driver — psycopg (v3) preferred, psycopg2 as fallback — only for that specific bare-scheme failure; any other scheme's error still propagates unchanged. Part of the migration tracked in apache#68453.
Dev-iL
force-pushed
the
2607/ppg3_providers
branch
from
f5b1337 to
c8ebd84
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes:
related:
What changed (6 commits, one per package)
providers/postgres—psycopg2-binarymoves from a hard dependency to an optional[psycopg2]extra. The psycopg2-specific connection/cursor/execute_valuesimports are now lazy/guarded, raising a clearAirflowOptionalProviderFeatureExceptionif genuinely needed but missing.providers/celery— the[celery] result_backendderivation from a driverlesssql_alchemy_connnow targetspostgresql+psycopg://(matching the new sync default) when psycopg (v3) is importable, falling back topostgresql+psycopg2://otherwise — mirroring the same guardPostgresHookuses.providers/google—BigQueryToPostgresOperator's psycopg2-specificregister_adapter(list/dict, Json)now only runs when the resolvedPostgresHookconnection is actually on psycopg2 (a no-op on psycopg3), so the module imports cleanly without psycopg2 installed.providers/pgvector—PgVectorIngestOperator._register_vectornow calls the registration helper matching the connection's actual driver (pgvector.psycopg2.register_vectorvs.pgvector.psycopg.register_vector) instead of always assuming psycopg2, fixing a pre-existing bug.providers/amazon—RedshiftSQLHook.get_sqlalchemy_engine()resolves an explicit driver (prefers psycopg, falls back to psycopg2, clear error if neither) instead of relying on SQLAlchemy's implicit psycopg2 default for a barepostgresql://URI.get_uri()'s own bare-scheme contract is untouched. Also updates the Batch/ECS/Lambda executor docs' example connection strings topostgresql+psycopg://.providers/common/sql—DbApiHook.get_sqlalchemy_engine()retries with an explicit driver only for the specific "barepostgresqlscheme, driver import failed" case; any other scheme's failure still propagates unchanged, keeping the class DB-agnostic. This is the general safety net: any current or futureDbApiHooksubclass with a postgres-compatibleconn_typethat doesn't overridesqlalchemy_urlis covered by this fix, not just the two concrete cases (amazon, google) found so far.Test plan
USE_PSYCOPG3branches in celery/google/pgvector,PgVectorIngestOperatorregistering against both a psycopg2-backed and psycopg3-backed connection, andget_sqlalchemy_engine's driver-resolution/retry/re-raise/no-retry-for-other-schemes behavior in both amazon and common.sql.Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.