Update Databricks integration setup docs (DOIO-94) by charlesmyu · Pull Request #38129 · DataDog/documentation

charlesmyu · 2026-07-13T17:34:12Z

What does this PR do?

Addresses user-reported friction in the Databricks integration setup docs (DOIO-94).

Files touched:

content/en/data_observability/jobs_monitoring/databricks/_index.md
content/en/data_observability/quality_monitoring/data_warehouses/databricks.md
content/en/cloud_cost_management/setup/saas_costs.md

Changes, mapped to the ticket items:

Item 1 — SP entitlements: Under Use a Service Principal for OAuth, folded the required workspace entitlements (Workspace access, Databricks SQL access, Admin access) into the service-principal creation step, replacing the separate "add to admins group" steps (Admin access is equivalent). Added a note that granular access (per the Permissions section under Advanced Configuration) is the fallback when Admin access can't be granted.
Item 3 — warehouse ID location: The SQL Warehouse setup is now a single step with ordered substeps (select a Pro/Serverless warehouse → copy the ID from the overview page → grant access). Also cross-links Quality Monitoring as something the warehouse ID powers.
Item 4 — ordering: Warehouse selection (including the Pro/Serverless requirement and dedicated-2XS recommendation) now comes before the access grants.
Item 5 — SQL Editor: Added an explicit "run in the SQL Editor" instruction before the system-table GRANT block (jobs-monitoring, quality-monitoring, saas_costs). (Screenshot not added — docs-repo limitation; follow-up.)
Item 6 — grant syntax: Replaced <service_principal> / <token_principal> with the backticked `<client-id>` and a "use the client ID, not the display name" note.
Surfaced the MANAGE on CATALOG system prerequisite as a warning callout next to the grant commands.

Intentionally not changed (per ticket-owner feedback during review):

Item 2 (SP MANAGE on itself): kept as-is.
Item 7 (required vs recommended): left the wording vague; the definitive "optional but recommended" framing belongs to the in-tile UI copy, not the docs.
Item 8 (env-var quotes): kept the DATABRICKS_WORKSPACE="..." quotes and the "enclose in double quotes if it contains whitespace" note, since quotes are needed for workspace names containing whitespace.

Out of scope for this docs repo (follow-ups for the owning team): the in-tile UI copy for items 3/7 and the SQL Editor screenshot for item 5.

Motivation

DOIO-94 — reduce onboarding friction for the Databricks integration.

Additional Notes

Draft — pending review, not intended to merge yet.

🤖 Generated with Claude Code

linear-code · 2026-07-13T17:34:16Z

DOIO-94

github-actions · 2026-07-13T17:38:49Z

Preview links (active after the `build_preview` check completes)

Modified Files

Reduce onboarding friction in the Databricks setup docs: - Fold the required workspace entitlements (Workspace access, Databricks SQL access, Admin access) into service principal creation, replacing the separate "add to admins group" steps; note the granular-permissions fallback when Admin access cannot be granted. - Restructure SQL Warehouse setup into a single step with ordered substeps: select a Pro/Serverless warehouse (recommend a dedicated 2XS), copy the ID, grant CAN USE, and grant system-table read access. - Document where to find the SQL Warehouse ID, and note it powers Quality Monitoring, with a cross-link to that page. - Tell users to run the system-table GRANT commands in the SQL Editor, and fix the grant syntax to use the backticked client ID instead of the service principal display name. - Surface the MANAGE-on-CATALOG-system prerequisite as a warning callout. Also updates the Cloud Cost Management (saas_costs) and Quality Monitoring Databricks docs to use the SQL Editor and backticked client ID for the system-table grants. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Databricks does not accept quotes of any kind in the Environment variables textbox (or the cluster-policy spark_env_vars Value field); whitespace in the workspace name is accepted without quoting. Remove the quotes and the "enclose in double quotes" note from those two UI-field contexts. The global init script remains a real shell script, where export VAR="..." quoting is valid and required for whitespace values, so it is left unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Just remove the quote instruction rather than explaining the behavior; the example already omits the quotes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

buraizu

Thanks for the PR, requesting a few updates. Also raising a few additional points:

Screenshot check: The screenshot at content/en/data_observability/jobs_monitoring/databricks/_index.md:376 (configure-databricks-cluster-init-script-quoted.png) — the filename suggests it may still show a quoted value in the Spark "Environment variables" textbox, which would now contradict the updated "do not enclose the value in quotes" instruction right below it. Worth confirming the screenshot matches the new guidance before merging.
GRANT statement ordering: The three GRANT statements for Unity Catalog system-table access are ordered differently between jobs_monitoring/databricks/_index.md (CATALOG, SELECT, SCHEMA) and cloud_cost_management/setup/saas_costs.md (CATALOG, SCHEMA, SELECT). This predates the PR, but since every line in these blocks is already being touched, it's a low-cost opportunity to standardize the order. Not blocking.
Callout ordering at _index.md:43-47: The new NOTE (Admin access fallback) is now stacked directly above a pre-existing WARNING about the service principal's Azure management type. The WARNING isn't actually related to entitlements — it's about a choice made earlier, when the service principal is created. Consider moving the WARNING up to where the service principal type is chosen (near "Click Add service principal, then click Add new"), so the entitlements section ends with just the one relevant callout. This is a pre-existing placement issue the PR happens to make more visible, not something introduced by this PR — optional follow-up.

buraizu · 2026-07-14T18:22:37Z

   | DD_API_KEY           | Your [Datadog API key][1].   |
   | DD_SITE              | Your [Datadog site][2].      |
-   | DATABRICKS_WORKSPACE | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Enclose the name in double quotes if it contains whitespaces. |
+   | DATABRICKS_WORKSPACE | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Do not enclose the value in quotes; Databricks does not accept quotes in this field. Whitespace in the name is allowed. |


Suggested change

| DATABRICKS_WORKSPACE | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Do not enclose the value in quotes; Databricks does not accept quotes in this field. Whitespace in the name is allowed. |

| DATABRICKS_WORKSPACE | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Do not enclose the value in quotes. Databricks does not accept quotes in this field. Whitespace in the name is allowed. |

buraizu · 2026-07-14T18:23:49Z

 | DD_API_KEY               | Your [Datadog API key][1].                                                                                                                                        |         |
 | DD_SITE                  | Your [Datadog site][2].                                                                                                                                           |         |
-| DATABRICKS_WORKSPACE     | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Enclose the name in double quotes if it contains whitespace. |         |
+| DATABRICKS_WORKSPACE     | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Do not enclose the value in quotes; Databricks does not accept quotes in this field. Whitespace in the name is allowed. |         |


Suggested change

| DATABRICKS_WORKSPACE | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Do not enclose the value in quotes; Databricks does not accept quotes in this field. Whitespace in the name is allowed. | |

| DATABRICKS_WORKSPACE | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Do not enclose the value in quotes. Databricks does not accept quotes in this field. Whitespace in the name is allowed. | |

…x.md Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

- Repoint the Spark env-vars screenshot to the unquoted-values image (configure-databricks-cluster-init-script.png) so it matches the "do not enclose the value in quotes" guidance - Standardize the Unity Catalog GRANT order to USE CATALOG, USE SCHEMA, SELECT, matching cloud_cost_management/setup/saas_costs.md - Move the Azure "Databricks managed" warning up to the service principal creation step, leaving the entitlements section ending with the Admin access note Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Update the remaining lowercase <client-id> placeholders in the cost-data GRANT block to match the uppercase <CLIENT-ID> used elsewhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

charlesmyu · 2026-07-14T19:04:10Z

@buraizu Thanks for the feedback! Updated, including a new screenshot. Let me know if this looks a bit better

buraizu · 2026-07-14T20:25:13Z

@charlesmyu thanks for following up. This LGTM, but unfortunately the deleted image needs to be re-added. This is because it will take some time for these changes to make their way into the translated docs — so deleting a screenshot now results in broken images in the translated pages. An automated process should remove outdated screenshots after some time. Let me know if you have any questions!

charlesmyu · 2026-07-14T21:13:18Z

@charlesmyu thanks for following up. This LGTM, but unfortunately the deleted image needs to be re-added. This is because it will take some time for these changes to make their way into the translated docs — so deleting a screenshot now results in broken images in the translated pages. An automated process should remove outdated screenshots after some time. Let me know if you have any questions!

@buraizu I believe we haven't deleted anything outright in this PR, but just replaced an image with a newer one - I think this is fine because it's functionally the same diagram, just updated for Databricks' new UI. Would this still be an issue?

buraizu · 2026-07-15T00:08:51Z

@charlesmyu apologies; you're right — no issues with the screenshot.

Separately, I've made commit 6b138dd to apply the wording from this suggestion to another instance of the previous phrasing. Since I made the change through the UI, it did something with the line endings that makes it look like the whole file is being replaced, when only one line changed. I reversed the change in 176a1b5, but happy to revert both my commits if you'd prefer — apologies again for the noise.

With that being said, this PR is still encountering a build_preview error which looks to be a transient issue in the pipeline. I'll merge in master for now to see if that resolves it.

…ation-docs

buraizu

Merging in the master branch resolved the GitLab error — LGTM

charlesmyu force-pushed the charlesyu/doio-94-update-databricks-integration-docs branch from a75c7c2 to 2b1b494 Compare July 13, 2026 18:45

charlesmyu marked this pull request as ready for review July 13, 2026 22:01

charlesmyu requested review from a team as code owners July 13, 2026 22:01

charlesmyu force-pushed the charlesyu/doio-94-update-databricks-integration-docs branch from 2b1b494 to fd25825 Compare July 13, 2026 22:01

This comment has been minimized.

Sign in to view

Drop the "enclose in quotes" note from Databricks UI-field env vars

dcb8895

Just remove the quote instruction rather than explaining the behavior; the example already omits the quotes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

buraizu requested changes Jul 14, 2026

View reviewed changes

charlesmyu and others added 3 commits July 14, 2026 14:42

Update content/en/data_observability/jobs_monitoring/databricks/_inde…

ed2e993

…x.md Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

Apply suggestions from code review

81c5dc0

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

github-actions Bot added the Images Images are added/removed with this PR label Jul 14, 2026

Standardize client-id placeholder casing to <CLIENT-ID>

f7c36f8

Update the remaining lowercase <client-id> placeholders in the cost-data GRANT block to match the uppercase <CLIENT-ID> used elsewhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

charlesmyu requested a review from buraizu July 14, 2026 19:05

buraizu self-assigned this Jul 14, 2026

buraizu added 2 commits July 14, 2026 16:15

Update phrasing

6b138dd

Normalize line endings to LF

176a1b5

Merge branch 'master' into charlesyu/doio-94-update-databricks-integr…

3b41caa

…ation-docs

buraizu approved these changes Jul 15, 2026

View reviewed changes

charlesmyu merged commit 8bbb031 into master Jul 15, 2026
20 checks passed

charlesmyu deleted the charlesyu/doio-94-update-databricks-integration-docs branch July 15, 2026 01:50

	\| DATABRICKS_WORKSPACE \| Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Do not enclose the value in quotes; Databricks does not accept quotes in this field. Whitespace in the name is allowed. \|
	\| DATABRICKS_WORKSPACE \| Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Do not enclose the value in quotes. Databricks does not accept quotes in this field. Whitespace in the name is allowed. \|

Uh oh!

Conversation

charlesmyu commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Uh oh!

linear-code Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Preview links (active after the build_preview check completes)

Modified Files

Uh oh!

This comment has been minimized.

buraizu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

buraizu Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

buraizu Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

charlesmyu commented Jul 14, 2026

Uh oh!

buraizu commented Jul 14, 2026

Uh oh!

charlesmyu commented Jul 14, 2026

Uh oh!

buraizu commented Jul 15, 2026

Uh oh!

buraizu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

charlesmyu commented Jul 13, 2026 •

edited

Loading

Preview links (active after the `build_preview` check completes)