Contribute dbt-fabric to the Fabric Toolbox

[sdebruyn]

This PR adds tools/dbt-fabric/: one dbt adapter for both Microsoft Fabric compute engines (Data Warehouse and Lakehouse) in a single Python package.

I wrote most of the code that's now in microsoft/dbt-fabric. When Microsoft adopted the repository I kept maintaining a fork because customers were asking for things the official repo wasn't shipping. That fork (dbt-fabric-samdebruyn on PyPI) is what multiple organizations are running in production today.

I'm bringing it to the toolbox because the toolbox's multi-contributor model — the Fabric product team, the CAT team, and the community sharing maintenance — fits a dbt adapter better than a single-maintainer setup. The dbt ecosystem moves quickly (new dbt-core minors, community-package releases, dbt-tests-adapter Base* classes every cycle), and a shared codebase under the toolbox is better positioned to keep up.

---

What this gives users today

One pip install dbt-fabric and both Fabric engines work. No separate dbt-fabricspark package, no system ODBC driver to install: the bundled mssql-python driver handles the Data Warehouse side and ships ODBC Driver 18 + unixODBC inside the wheel.

On top of that, a long list of features the official adapters don't ship:

Microsoft Purview integration via API. A {{ purview_sync() }} macro that pushes model and column documentation, plus ref() and source() lineage, directly into Purview through the REST API. persist_docs-aware: models marked persist_docs: false are skipped, granular relation: true, columns: false only syncs what you asked for. No Purview scan configuration needed.

Python models on both engines. Standard model(dbt, spark) API with PySpark on both Data Warehouse and Lakehouse. microsoft/dbt-fabric doesn't support Python models at all; microsoft/dbt-fabricspark only supports them on the Lakehouse.

Compatibility with nine community packages, continuously tested. dbt-utils, dbt-date, dbt-codegen, dbt-expectations, dbt-audit-helper, dbt-external-tables, dbt-profiler, dbt-artifacts, and dbt-project-evaluator ship Postgres- or Snowflake-flavoured macros that fail on Fabric. This contribution writes the adapter-specific overrides through dbt's dispatch system and runs an integration test for each package on every PR. Neither official adapter ships overrides or tests for any of these.

And the rest:

• Warehouse snapshots as a callable Jinja macro ({{ create_or_update_fabric_warehouse_snapshot(...) }}) usable from on-run-start/on-run-end, any post-hook, or dbt run-operation.
• dbt-external-tables compatibility via dispatch, so OPENROWSET-backed files are regular source() references in the lineage graph.
• cluster_by as a standard model config.
• Manual statistics as model config.
• Catalog statistics in dbt docs generate output.
• Functions (dbt-core 1.11 scalar functions) on both engines.
• Workload identity (federated OIDC for CI/CD).
• 11 authentication methods through standard dbt profile keys, plus custom TokenCredential classes.
• One shared FabricTokenProvider covering both adapter types, so the same profile structure works for DW and Lakehouse.
• Auto host-resolution from the workspace name.
• A PEP 249–compliant cursor for Spark JSON results.

Every PR runs against real Fabric, and every release ships after the full integration suite has gone green.

---

Issues filed upstream

I tried contributing some of these fixes back to microsoft/dbt-fabric first, but the review-to-merge turnaround on PRs was long enough that I couldn't keep momentum that way. The fork picked up what the upstream couldn't absorb at that pace, and the gap has compounded since.

The issues and PRs listed below were filed in a single recent pass — a fresh attempt at backporting the bugfixes and smaller refactors now that the gap is documented. Backporting the larger feature work (Python models, the unified profile schema and shared auth stack, the community-package dispatch overrides, the dbt-spark inheritance for the Lakehouse adapter) isn't viable yet: it sits on top of dbt-core surface and a consolidated BaseFabricAdapter that the official repos don't have. Those backports only become tractable once the upstream baseline is closer to current.

microsoft/dbt-fabric

• microsoft/dbt-fabric#384 — varchar(8000) silently truncates long-text string columns (PR: microsoft/dbt-fabric#385)
• microsoft/dbt-fabric#373 — Case-sensitive Fabric Warehouses broken (missing _make_match_kwargs override) (PR: microsoft/dbt-fabric#375)
• microsoft/dbt-fabric#374 — apply_grants re-issues GRANTs on every run; query misses Entra-principal grants (PR: microsoft/dbt-fabric#376)
• microsoft/dbt-fabric#391 — Pre/post hooks fail because dbt-adapters' default run_hooks emits commit; (PR: microsoft/dbt-fabric#392)
• microsoft/dbt-fabric#377 — fabric__create_table_as wraps CTAS in EXEC('...') and manually escapes the query label (PR: microsoft/dbt-fabric#378)
• microsoft/dbt-fabric#389 — get_response returns hardcoded "OK", discards cursor messages + statement IDs (PR: microsoft/dbt-fabric#390)
• microsoft/dbt-fabric#399 — FabricAdapter.quote() doesn't escape ] — T-SQL injection vector (PR: microsoft/dbt-fabric#400)
• microsoft/dbt-fabric#381 — pyodbc pooling silently disabled; the right fix is landing the mssql-python PR (PR: microsoft/dbt-fabric#350)
• microsoft/dbt-fabric#395 — fabric__get_use_database_sql emits invalid USE [None]; from drop_schema_named (PR: microsoft/dbt-fabric#396)
• microsoft/dbt-fabric#397 — Incremental --full-refresh drop-then-recreate risks data loss on creation failure (PR: microsoft/dbt-fabric#398)
• microsoft/dbt-fabric#393 — fabric__get_incremental_microbatch_sql ignores unique_key (PR: microsoft/dbt-fabric#394)
• microsoft/dbt-fabric#382 — Custom fabric__snapshot_merge_sql UPDATE+INSERT instead of native MERGE
• microsoft/dbt-fabric#383 — delete_warehouse_snapshot is a return True stub
• microsoft/dbt-fabric#386 — apply_label macro emits debug log() on every call; should use query_header
• microsoft/dbt-fabric#387 — check_for_nested_cte macro parses SQL in Jinja (categorically wrong)
• microsoft/dbt-fabric#388 — PR #315 login_timeout=getattr(...) is a no-op on every call site
• microsoft/dbt-fabric#372 — Warehouse snapshots coupled to atexit + open() should be a Jinja macro
• microsoft/dbt-fabric#380 — Adapter-private delete_condition / delete_not_matched_by_source on incremental
• microsoft/dbt-fabric#379 — v1.9.10 retry wrapper at wrong layer (use add_query's retryable_exceptions)
• microsoft/dbt-fabric#370 — Module-level _TOKEN global — thread-safety and lifecycle issues
• microsoft/dbt-fabric#371 — Proposal: modernise build and tooling stack (PEP 621, uv, ruff, drop EOL Python)
• microsoft/dbt-fabric#230 — dbt docs generate should include catalog rowcounts (already delivered by this contribution)

microsoft/dbt-fabricspark

• microsoft/dbt-fabricspark#191 — Six __exit__ methods return True (silent exception swallowing) (PR: microsoft/dbt-fabricspark#193)
• microsoft/dbt-fabricspark#192 — Hardcoded expires_on = 1845972874 (year 2028) in int_tests auth path bypasses all token refresh
• microsoft/dbt-fabricspark#194 — _getLivySQL: re.DOTALL passed as positional count caps comment-stripping at 16 (PR: microsoft/dbt-fabricspark#195)
• microsoft/dbt-fabricspark#199 — Livy session cleanup bypasses dbt's close() lifecycle and uses atexit
• microsoft/dbt-fabricspark#202 — Dead Thrift exception handler from dbt-spark ancestry (PR: microsoft/dbt-fabricspark#203)
• microsoft/dbt-fabricspark#201 — Proposal: inherit from dbt-spark instead of being a standalone SQLAdapter
• microsoft/dbt-fabricspark#197 — botocore/boto3 DEBUG logging at module import time (PR: microsoft/dbt-fabricspark#198)
• microsoft/dbt-fabricspark#196 — _parse_retry_after duplicated 4× with deprecated datetime.utcnow() (PR: microsoft/dbt-fabricspark#200)

The structural cause is that the adapter is staffed as a sideline of a product role, with PyPI ownership on a personal account rather than a Microsoft organisational identity. A few of the issues read as AI-assisted PRs merged without dbt-domain review — a missing-review-step problem, not an AI problem. The recurring pattern across the rest is the same shape: features built next to dbt's conventional mechanisms instead of through them — dispatch sidestepped, profile keys camelCased, hooks reimplemented as atexit — the kind of mismatch you notice when you use dbt daily. The fix is shared maintenance under the toolbox with organisational package ownership and reviewers who use the adapter.

---

Why this stays maintainable long-term

The maintenance cost of a dbt adapter scales with two things: how much you reimplement that dbt's ecosystem already gives you, and how many private mechanisms you have to keep in sync with dbt-core across releases. The architecture here keeps both close to zero.

The Lakehouse adapter in this contribution inherits from dbt-spark. dbt-spark ships the Spark materializations, incremental strategies, Spark-aware column type handling, constraint handling, the Spark Python-model API, and a SparkAdapter base class. Spark-based adapters inherit from it and write only the parts specific to their engine — dbt-databricks does this.

microsoft/dbt-fabricspark doesn't. It's a standalone SQLAdapter, so every macro, materialization, type rule, incremental strategy, and Python-model path has to be implemented and maintained by hand. Python's multiple inheritance lets the FabricSpark adapter extend SparkAdapter and a shared BaseFabricAdapter at the same time, getting the dbt-spark machinery and the cross-adapter Fabric code in one class.

One auth stack, one Fabric API client, and one Livy session layer across both adapters. Auth, Fabric REST API access, workspace resolution, profile validation, and Livy session handling are the same problem on both engines — the Lakehouse runs everything through Livy, and the Data Warehouse needs the same Livy machinery for Python models.

microsoft/dbt-fabric and microsoft/dbt-fabricspark each maintain their own token logic, API client, workspace resolution, and profile validation, and they're already out of sync. The DW adapter uses workspace_id / workspace_name / access_token (snake_case). The Lakehouse adapter uses workspaceid / lakehouseid / accessToken (camelCase). The default auth method differs (ActiveDirectoryDefault vs CLI). A user running both can't share a profile structure. Every auth-related change has to be implemented twice.

This contribution has one FabricTokenProvider covering all 11 auth methods for both adapter types. One FabricApiClient for workspaces, warehouses, lakehouses, Livy, and snapshots. One Python-model submission path. One profile schema. A bug fix is a one-place change.

Test suite built on dbt-tests-adapter. Every dbt-core minor ships new Base* test classes that codify what an adapter needs to do to be compatible. Bumping dbt-tests-adapter is how this adapter picks up coverage for new dbt-core features automatically. About 430 adapter test classes here plus an extra ~110 community-package tests, all running against real Fabric. PRs run on Python 3.13; the full Python 3.11/3.12/3.13 matrix runs weekly.