Skip to content

Add IBM Db2 provider with Hook #67532

Open
Amitkumar293 wants to merge 28 commits into
apache:mainfrom
Amitkumar293:airflow-db2-adapter
Open

Add IBM Db2 provider with Hook #67532
Amitkumar293 wants to merge 28 commits into
apache:mainfrom
Amitkumar293:airflow-db2-adapter

Conversation

@Amitkumar293

@Amitkumar293 Amitkumar293 commented May 26, 2026

Copy link
Copy Markdown

Description

This PR adds a new provider for IBM Db2 database support in Apache Airflow.

What: New Db2 provider package with Hook and Operator
Why: Enable Airflow users to connect to and execute queries against IBM Db2 databases

Changes

  • Db2Hook: Extends DbApiHook for IBM Db2 connectivity with SSL support
  • Added Db2Dialect: - Mentioned the SQLAlchemy dialect component
  • Comprehensive unit tests (Tests covering all functionality)
  • Documentation and example DAGs
  • Provider metadata and configuration

Testing

  • ✅ All unit tests passing
  • ✅ Ruff linting and formatting checks passed
  • ✅ Follows Airflow coding standards

Important

🛠️ Maintainer triage note for @Amitkumar293 · by @potiuk · 2026-07-08 15:38 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed:

  • Unit tests failing (Basic tests / Breeze unit tests, Non-DB tests: core / Non-DB-core::3.10:API...CLI, Non-DB tests: core / Non-DB-core::3.11:API...CLI (+20 more)). See the contributor guide.
  • Pre-commit / static checks failing (CI image checks / Static checks). See the contributor guide.
  • Image build failing (CI image checks / Build documentation (--docs-only), CI image checks / Build documentation (--spellcheck-only)). See the contributor guide.
  • Provider tests failing (MyPy providers checks, provider distributions tests / Compat 2.11.1:P3.10:, provider distributions tests / Compat 3.0.6:P3.10: (+34 more)). See the contributor guide.

Full criteria: Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

@boring-cyborg

boring-cyborg Bot commented May 26, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@Amitkumar293
Amitkumar293 marked this pull request as ready for review May 27, 2026 05:59
Comment thread .github/boring-cyborg.yml Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Note, that some fomality with announcement/DISCUSS is neede to add a new provider, following https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst

@Amitkumar293

Copy link
Copy Markdown
Author

Note, that some fomality with announcement/DISCUSS is neede to add a new provider, following https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst

Thanks for pointing this out.

I’ve created a DISCUSS thread on the dev@airflow.apache.org mailing list for the new IBM Db2 provider, following the ACCEPTING_PROVIDERS.rst process.

Comment thread providers/ibm/db2/src/airflow/providers/ibm/db2/hooks/__init__.py
@github-actions

github-actions Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

uv.lock on main just moved via #69694 ("[main] Upgrade important CI environment"), commit c8d313a and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

@Amitkumar293
Amitkumar293 force-pushed the airflow-db2-adapter branch from 5d470ba to 7b1633e Compare June 1, 2026 10:03
Comment thread scripts/ci/docker-compose/tests-sources.yml Outdated
Comment thread providers/ibm/db2/docs/commits.rst Outdated
Comment thread providers/ibm/db2/docs/example-dags.rst
Comment thread providers/ibm/db2/docs/installing-providers-from-sources.rst Outdated
Comment thread providers/ibm/db2/docs/security.rst Outdated
Comment thread providers/ibm/db2/src/airflow/providers/ibm/db2/operators/db2.py Outdated
Comment thread providers/ibm/db2/provider.yaml Outdated
@Amitkumar293

Copy link
Copy Markdown
Author

@jscheffl Thanks for your review, very helpful. I am working on the review comments.

@eladkal

eladkal commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Was there a mailing list thread about this new provider?
https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst#examples-of-current-process

Amit Kumar and others added 28 commits July 17, 2026 14:41
Implements initial IBM Db2 provider package with:
- Db2Hook: Database connectivity with SSL support
- Db2Operator: SQL execution with Jinja templating
- Unit tests for both hook and operator
- Example DAG demonstrating usage
- Add Db2-specific connection test SQL using SYSIBM.SYSDUMMY1
- Add cleanup task to example DAG for idempotent runs
- Add autocommit=True to all DDL/DML operations in example DAG
- Ensure proper transaction handling for Db2 database operations
- Implement Db2Hook extending DbApiHook for IBM Db2 connectivity
- Add Db2Operator for SQL execution with parameter binding
- Include comprehensive unit tests for Hook and Operator
- Add documentation, example DAGs, and provider metadata
- Support SSL connections, autocommit, and batch execution
Move provider from `providers/db2/` to `providers/ibm/db2/` and update package name from `apache-airflow-providers-db2` to `apache-airflow-providers-ibm-db2`. Updates all import paths, documentation, tests, and workspace configuration.
- Change conn_type to 'ibmdb2' and conn_name_attr to 'conn_id'
- Add type hint for get_conn() return value
- Fix: Skip None values in connection strings
Move example DAG to tests/system/ibm/db2/ and update provider metadata to not-ready/incubation/0.1.0
Add dialects docs and clean up __init__.py files.
@Amitkumar293
Amitkumar293 force-pushed the airflow-db2-adapter branch from 3ca44bc to c2999aa Compare July 17, 2026 09:14

@dabla dabla left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Amitkumar293 You'll need to rebase, and then locally run following command with breeze:

breeze setup regenerate-command-images

As you added dependencies for db2, SBOM images need to be regenerated as well for those new dependencies.

This is why you see following failures in static checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants