Add Databricks SQL warehouse lifecycle operators

[Vamsi-klu]

This adds first-class operators for starting and stopping existing Databricks SQL warehouses, including optional polling until the requested lifecycle state is reached.

related: #21377

Problem

Airflow's Databricks provider can execute SQL against a warehouse, but it has no first-class way to manage an existing warehouse's start/stop lifecycle. Dag authors currently need custom REST calls around their SQL tasks.

What changed

• Add DatabricksHook methods for retrieving, starting, and stopping a warehouse through the Databricks SQL Warehouses API.
• Add a validated WarehouseState model for the six documented lifecycle states.
• Add DatabricksStartWarehouseOperator and DatabricksStopWarehouseOperator with idempotent pre-checks, optional waiting, monotonic deadlines, and explicit terminal-state errors.
• Register the operators in provider metadata and add a how-to guide plus a system-test example with unconditional stop cleanup.

Warehouse IDs are embedded in the documented REST paths; no request-body workaround or new dependency is introduced. The implementation follows the Databricks SQL Warehouses API.

Scope

This is the Phase 1 scope proposed on #21377: get/state/start/stop plus synchronous waiting. Create/delete, edit, warehouse-by-name resolution, async hooks, and deferrable operators remain outside this PR so the initial contribution stays reviewable and independently useful.

Behavior and compatibility

• Starting an already RUNNING warehouse and stopping an already STOPPED warehouse are no-ops.
• Existing STARTING/STOPPING transitions are reused instead of issuing duplicate requests.
• A start accepted while the warehouse still reports STOPPING continues polling; Databricks API transition rejections propagate unchanged.
• Waiting uses time.monotonic() and never accepts a target state observed after the configured deadline.
• The new operator validates templated warehouse IDs at execution time and remains import-compatible across supported Airflow versions through common.compat.sdk.

Validation

• breeze run pytest providers/databricks/tests/unit/databricks/operators/test_databricks_warehouse.py -xvs — 21 passed.
• breeze testing providers-tests --test-type "Providers[databricks]" — 838 passed, 12 skipped.
• breeze testing providers-tests --test-type "Providers[amazon,common.compat,common.sql,databricks,google,openlineage]" — 11,858 passed, 185 skipped.
• breeze run mypy providers/databricks/src/airflow/providers/databricks/exceptions.py providers/databricks/src/airflow/providers/databricks/hooks/databricks.py providers/databricks/src/airflow/providers/databricks/operators/databricks_warehouse.py — success, no issues.
• Explicit nine-file prek pre-commit checks — passed.
• Explicit nine-file prek manual checks — passed, including the providers mypy hook.
• breeze build-docs --docs-only --clean-build databricks — documentation build successful; the generated guide contains both lifecycle examples.
• breeze run pytest providers/databricks/tests/system/databricks/example_databricks_sql_warehouse.py --collect-only -q — 1 system test collected.
• breeze ci selective-check --commit-ref HEAD — selected provider unit/compatibility tests, provider mypy, docs, Python scans, and the system-test path; no UI tests selected.

Reviewer evidence

This PR has no UI surface, so before/after screenshots and browser validation are not applicable. The REST boundary is covered with autospecced request assertions, operator behavior is covered with a specced hook, and the system example is import-validated. No Databricks workspace credentials were used or required for these deterministic lifecycle tests.

The narrow Phase 1 scope was posted on the issue before implementation: #21377 (comment)

---

Was generative AI tooling used to co-author this PR?

• Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

[Vamsi-klu]

Local validation evidence for the Databricks SQL warehouse lifecycle implementation:

• Focused operator suite: 21 passed.
• Full Databricks provider suite: 838 passed, 12 skipped.
• Selective six-provider dependency matrix: 11,858 passed, 185 skipped.
• Changed-source mypy: Success: no issues found in 3 source files.
• Explicit pre-commit and manual prek checks: passed; the manual run included the providers mypy hook.
• Databricks docs build: successful; generated output contains both start and stop examples.
• System-test example: one test_run collected successfully.
• Selective-check analysis selected the expected provider unit/compatibility, provider mypy, docs, Python scan, and system-test jobs; it selected no UI work.

The tests assert the exact Databricks Warehouses API paths, idempotent start/stop behavior, transition-in-progress behavior, terminal failure states, templated-ID validation, and strict monotonic timeout handling.

There are no UI changes in this PR, so screenshots would not add reviewer signal. No Databricks credentials were used: REST calls are mocked at the hook boundary, while the system-test Dag is import-validated. Live workspace execution can be added later if a reviewer specifically requests it.

---

Drafted-by: Codex (GPT-5)

[Vamsi-klu]

@eladkal @moomindani Can i get some feedack/Stamp for the PR please? Thanks!