Update Databricks integration setup docs (DOIO-94)

content/en/cloud_cost_management/setup/saas_costs.md

@@ -120,11 +120,11 @@ Notes:
 4. Enter a `System Tables SQL Warehouse ID` corresponding to your Databricks instance's warehouse to query for system table billing data.
 5. Click {{< ui >}}Save Databricks Workspace{{< /ui >}}.
 
-Your service principal requires read access to the [system tables](https://docs.databricks.com/aws/en/admin/system-tables/) within Unity Catalog.
+Your service principal requires read access to the [system tables](https://docs.databricks.com/aws/en/admin/system-tables/) within Unity Catalog. In Databricks, open the {{< ui >}}SQL Editor{{< /ui >}} and run the following commands, using the service principal's client ID (not its display name):
 ```sql
-GRANT USE CATALOG ON CATALOG system TO <service_principal>;
-GRANT USE SCHEMA ON CATALOG system TO <service_principal>;
-GRANT SELECT ON CATALOG system TO <service_principal>;
+GRANT USE CATALOG ON CATALOG system TO `<CLIENT-ID>`;
+GRANT USE SCHEMA ON CATALOG system TO `<CLIENT-ID>`;
+GRANT SELECT ON CATALOG system TO `<CLIENT-ID>`;
 ```
 
 Your Databricks cost data for the past 15 months can be accessed in Cloud Cost Management after 24 hours. To access the available data collected by each SaaS Cost Integration, see the [Data Collected section](#data-collected).

content/en/data_observability/jobs_monitoring/databricks/_index.md

@@ -37,9 +37,15 @@ Follow these steps to enable Data Observability: Jobs Monitoring for Databricks.
 1. As a **Databricks workspace admin**, go to {{< ui >}}Settings{{< /ui >}} by clicking your profile in the upper-right corner of the workspace.
 1. On the {{< ui >}}Identity and access{{< /ui >}} tab, click {{< ui >}}Manage{{< /ui >}} next to {{< ui >}}Service principals{{< /ui >}}.
 1. Click {{< ui >}}Add service principal{{< /ui >}}, then click {{< ui >}}Add new{{< /ui >}}.
-1. Enter a name, then click **Add**.
 
    <div class="alert alert-warning">For Azure Databricks, select the "Databricks managed" management type. Datadog does NOT support "Microsoft Entra ID managed" service principals.</div>
+1. Enter a name and enable the following workspace entitlements for the service principal:
+   - {{< ui >}}Workspace access{{< /ui >}}
+   - {{< ui >}}Databricks SQL access{{< /ui >}}
+   - {{< ui >}}Admin access{{< /ui >}}: Grants the workspace administrator access that Datadog requires. This is equivalent to adding the service principal to the `admins` group.
+
+   <div class="alert alert-info">If you cannot grant the <strong>Admin access</strong> entitlement, provision granular access instead, as described in the <a href="#permissions">Permissions</a> section under Advanced Configuration.</div>
+1. Click **Add**.
 
 1. Click on the name of your new service principal. Under the {{< ui >}}Secrets{{< /ui >}} tab, click {{< ui >}}Generate secret{{< /ui >}}.
    1. Set {{< ui >}}Lifetime (days){{< /ui >}} to the maximum value allowed (730).
@@ -51,26 +57,26 @@ Follow these steps to enable Data Observability: Jobs Monitoring for Databricks.
   {{< img src="data_jobs/databricks/client-id-secret.png" alt="In Databricks, a modal showing the client ID and secret associated with a new OAuth secret is displayed." style="width:70%;" >}}
 
 1. On the {{< ui >}}Permissions{{< /ui >}} tab, click {{< ui >}}Grant access{{< /ui >}}. Search for the new service principal, grant it the {{< ui >}}Manage{{< /ui >}} permission, and click {{< ui >}}Save{{< /ui >}}.
-1. Return to the {{< ui >}}Identity and access{{< /ui >}} tab and click {{< ui >}}Manage{{< /ui >}} next to {{< ui >}}Groups{{< /ui >}}.
-1. Click the {{< ui >}}admins{{< /ui >}} group, then click {{< ui >}}Add members{{< /ui >}} to add the new service principal.
 
 #### Add the Databricks workspace to Datadog
 
 1. In Datadog, open the Databricks integration tile.
 1. On the {{< ui >}}Configure{{< /ui >}} tab, click {{< ui >}}Add Databricks Workspace{{< /ui >}}.
 1. Enter a workspace name, your Databricks workspace URL, and the client ID and secret you generated.
    {{< img src="data_jobs/databricks/connect-workspace-form-m2m.png" alt="In the Datadog-Databricks integration tile, a Databricks workspace is displayed. This workspace has a name, URL, client ID, and client secret." style="width:100%;" >}}
-1. To gain visibility into your Databricks costs in Data Observability: Jobs Monitoring or [Cloud Cost Management][18], provide the ID of a [Databricks SQL Warehouse][19] that Datadog can use to query your [system tables][20].
-   - The service principal must have access to the SQL Warehouse. In the Warehouse configuration page, go to {{< ui >}}Permissions{{< /ui >}} (top right) and grant it `CAN USE` permission.
-   - Grant the service principal read access to the Unity Catalog [system tables][20] by running the following commands:
-   ```sql
-   GRANT USE CATALOG ON CATALOG system TO <service_principal>;
-   GRANT SELECT ON CATALOG system TO <service_principal>;
-   GRANT USE SCHEMA ON CATALOG system TO <service_principal>;
-   ```
-   The user granting these must have `MANAGE` privilege on `CATALOG system`.
-
-   -  The SQL Warehouse must be Pro or Serverless. Classic Warehouses are **NOT** supported. A 2XS warehouse is recommended, with Auto Stop set to 5-10 minutes to reduce cost.
+1. Provide the ID of a [Databricks SQL Warehouse][19] for Datadog to query. This gives you visibility into your Databricks costs in Jobs Monitoring or [Cloud Cost Management][18] and powers [Quality Monitoring][21].
+   1. In Databricks, go to {{< ui >}}SQL Warehouses{{< /ui >}} and select the warehouse for Datadog to use. It must be Pro or Serverless. Classic Warehouses are not supported. To reduce costs, use a dedicated 2XS warehouse, with Auto Stop configured for 5-10 minutes.
+   1. Copy the ID from the warehouse's overview page (it is also the last segment of the warehouse's URL) and enter it in the integration tile.
+   1. On the warehouse's {{< ui >}}Permissions{{< /ui >}} tab (top right), grant the service principal `CAN USE`.
+   1. Grant the service principal read access to the Unity Catalog [system tables][20]. In the {{< ui >}}SQL Editor{{< /ui >}}, run the following commands using the service principal's client ID (not its display name):
+
+      ```sql
+      GRANT USE CATALOG ON CATALOG system TO `<CLIENT-ID>`;
+      GRANT USE SCHEMA ON CATALOG system TO `<CLIENT-ID>`;
+      GRANT SELECT ON CATALOG system TO `<CLIENT-ID>`;
+      ```
+
+      <div class="alert alert-info">The user running these commands must have the <code>MANAGE</code> privilege on <code>CATALOG system</code>.</div>
 1. In the **Select products to set up integration** section, ensure that Data Observability: Jobs Monitoring is {{< ui >}}Enabled{{< /ui >}}.
 1. In the {{< ui >}}Datadog Agent Setup{{< /ui >}} section, choose either
     - [Managed by Datadog (recommended)](?tab=datadogmanagedglobalinitscriptrecommended#install-the-datadog-agent): Datadog installs and manages the Agent with a global init script in the workspace.
@@ -79,6 +85,7 @@ Follow these steps to enable Data Observability: Jobs Monitoring for Databricks.
 [18]: https://docs.datadoghq.com/cloud_cost_management/
 [19]: https://docs.databricks.com/aws/en/compute/sql-warehouse/
 [20]: https://docs.databricks.com/aws/en/admin/system-tables/
+[21]: /data_observability/quality_monitoring/data_warehouses/databricks/
 
 {{% /tab %}}
 
@@ -110,17 +117,19 @@ See [Private Link Connectivity (Preview)][15] for full setup instructions.
 1. On the {{< ui >}}Configure{{< /ui >}} tab, click {{< ui >}}Add Databricks Workspace{{< /ui >}}.
 1. Enter a workspace name, your Databricks workspace URL, and the Databricks token you generated.
    {{< img src="data_jobs/databricks/configure-workspace-form.png" alt="In the Datadog-Databricks integration tile, a Databricks workspace is displayed. This workspace has a name, URL, and API token." style="width:100%;" >}}
-1. To gain visibility into your Databricks costs in Data Observability: Jobs Monitoring or [Cloud Cost Management][18], provide the ID of a [Databricks SQL Warehouse][19] that Datadog can use to query your [system tables][20].
-
-   - The token's principal must have access to the SQL Warehouse. Give it `CAN USE` permission from **Permissions** at the top right of the Warehouse configuration page.
-   - Grant the service principal read access to the Unity Catalog [system tables][20] by running the following commands::
-   ```sql
-   GRANT USE CATALOG ON CATALOG system TO <token_principal>;
-   GRANT SELECT ON CATALOG system TO <token_principal>;
-   GRANT USE SCHEMA ON CATALOG system TO <token_principal>;
-   ```
-   The user granting these must have `MANAGE` privilege on `CATALOG system`.
-   -  The SQL Warehouse must be Pro or Serverless. Classic Warehouses are **NOT** supported. A 2XS size warehouse is recommended, with Auto Stop configured for 5-10 minutes to minimize cost.
+1. Provide the ID of a [Databricks SQL Warehouse][19] for Datadog to query. This gives you visibility into your Databricks costs in Jobs Monitoring or [Cloud Cost Management][18] and powers [Quality Monitoring][21].
+   1. In Databricks, go to {{< ui >}}SQL Warehouses{{< /ui >}} and select the warehouse for Datadog to use. It must be Pro or Serverless. Classic Warehouses are not supported. To reduce costs, use a dedicated 2XS warehouse, with Auto Stop configured for 5-10 minutes.
+   1. Copy the ID from the warehouse's overview page (it is also the last segment of the warehouse's URL) and enter it in the integration tile.
+   1. On the warehouse's {{< ui >}}Permissions{{< /ui >}} tab (top right), grant the token's principal `CAN USE`.
+   1. Grant the token's principal read access to the Unity Catalog [system tables][20]. In the {{< ui >}}SQL Editor{{< /ui >}}, run the following commands using the principal's client ID (not its display name):
+
+      ```sql
+      GRANT USE CATALOG ON CATALOG system TO `<CLIENT-ID>`;
+      GRANT USE SCHEMA ON CATALOG system TO `<CLIENT-ID>`;
+      GRANT SELECT ON CATALOG system TO `<CLIENT-ID>`;
+      ```
+
+      <div class="alert alert-info">The user running these commands must have the <code>MANAGE</code> privilege on <code>CATALOG system</code>.</div>
 1. In the **Select products to set up integration** section, make sure the Data Observability: Jobs Monitoring product is **Enabled**.
 1. In the {{< ui >}}Datadog Agent Setup{{< /ui >}} section, choose either
     - [Managed by Datadog (recommended)](?tab=datadogmanagedglobalinitscriptrecommended#install-the-datadog-agent): Datadog installs and manages the Agent with a global init script in the workspace.
@@ -133,6 +142,7 @@ See [Private Link Connectivity (Preview)][15] for full setup instructions.
 [18]: https://docs.datadoghq.com/cloud_cost_management
 [19]: https://docs.databricks.com/aws/en/compute/sql-warehouse/
 [20]: https://docs.databricks.com/aws/en/admin/system-tables/
+[21]: /data_observability/quality_monitoring/data_warehouses/databricks/
 
 
 {{% /tab %}}
@@ -231,7 +241,7 @@ This approach is recommended for clusters in **Standard** access mode.
    |----------------------|------------------------------|
    | DD_API_KEY           | Your [Datadog API key][1].   |
    | DD_SITE              | Your [Datadog site][2].      |
-   | DATABRICKS_WORKSPACE | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Enclose the name in double quotes if it contains whitespaces. |
+   | DATABRICKS_WORKSPACE | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). |
 
    1. For each of the above variables, in the {{< ui >}}Definition{{< /ui >}} section, click {{< ui >}}Add Definition{{< /ui >}}. In the modal that opens, fill in the fields:
        1. In the {{< ui >}}Field{{< /ui >}} dropdown, select {{< ui >}}spark_env_vars{{< /ui >}}.
@@ -363,14 +373,14 @@ Optionally, you can also set other init script parameters and Datadog environmen
 
 1. In Databricks, on the cluster configuration page, click the {{< ui >}}Advanced options{{< /ui >}} toggle.
 2. At the bottom of the page, go to the {{< ui >}}Spark{{< /ui >}} tab.
-   {{< img src="data_jobs/databricks/configure-databricks-cluster-init-script-quoted.png" alt="Databricks UI, cluster configuration advanced options, Spark tab. A textbox titled 'Environment variables' contains values for DD_API_KEY and DD_SITE." style="width:100%;" >}}
+   {{< img src="data_jobs/databricks/configure-databricks-cluster-init-script.png" alt="Databricks UI, cluster configuration advanced options, Spark tab. A textbox titled 'Environment variables' contains values for DD_API_KEY and DD_SITE." style="width:100%;" >}}
 
    In the {{< ui >}}Environment variables{{< /ui >}} textbox, provide the values for the init script parameters.
 
    ```text
    DD_API_KEY=<YOUR API KEY>
    DD_SITE=<YOUR DATADOG SITE>
-   DATABRICKS_WORKSPACE="<YOUR WORKSPACE NAME>"
+   DATABRICKS_WORKSPACE=<YOUR WORKSPACE NAME>
    ```
 
    Optionally, you can also set other init script parameters and Datadog environment variables here, such as `DD_ENV` and `DD_SERVICE`. The script can be configured using the following parameters:
@@ -379,7 +389,7 @@ Optionally, you can also set other init script parameters and Datadog environmen
 |--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
 | DD_API_KEY               | Your [Datadog API key][1].                                                                                                                                        |         |
 | DD_SITE                  | Your [Datadog site][2].                                                                                                                                           |         |
-| DATABRICKS_WORKSPACE     | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). Enclose the name in double quotes if it contains whitespace. |         |
+| DATABRICKS_WORKSPACE     | Name of your Databricks Workspace. It should match the name provided in the [Datadog-Databricks integration step](#configure-the-datadog-databricks-integration). |         |
 | DRIVER_LOGS_ENABLED      | Collect spark driver logs in Datadog.                                                                                                                          | false   |
 | WORKER_LOGS_ENABLED      | Collect spark workers logs in Datadog.                                                                                                                         | false   |
 | DD_TAGS                  | Add tags to Databricks cluster and Spark performance metrics. Comma or space separated key:value pairs. Follow [Datadog tag conventions][4]. For example: `env:staging,team:data_engineering` |         |
@@ -464,11 +474,11 @@ If you need more granular control, grant these minimal permissions to the follow
 
 Additionally, for Datadog to access your Databricks cost data in Data Observability: Jobs Monitoring or [Cloud Cost Management][26], the user or service principal used to query [system tables][27] must have the following permissions:
    - `CAN USE` permission on the SQL Warehouse.
-   - Read access to the [system tables][27] within Unity Catalog. This can be granted with:
+   - Read access to the [system tables][27] within Unity Catalog. In Databricks, open the {{< ui >}}SQL Editor{{< /ui >}} and run the following commands, using the service principal's client ID (not its display name):
    ```sql
-   GRANT USE CATALOG ON CATALOG system TO <service_principal>;
-   GRANT SELECT ON CATALOG system TO <service_principal>;
-   GRANT USE SCHEMA ON CATALOG system TO <service_principal>;
+   GRANT USE CATALOG ON CATALOG system TO `<CLIENT-ID>`;
+   GRANT USE SCHEMA ON CATALOG system TO `<CLIENT-ID>`;
+   GRANT SELECT ON CATALOG system TO `<CLIENT-ID>`;
    ```
    The user granting these must have `MANAGE` privilege on `CATALOG system`.
 

content/en/data_observability/quality_monitoring/data_warehouses/databricks.md

@@ -37,6 +37,8 @@ If your Databricks workspace restricts network access by IP, add the Datadog web
 
 ### Step 2 - Grant access
 
+In Databricks, open the {{< ui >}}SQL Editor{{< /ui >}} to run the following commands. Use the service principal's application (client) ID, not its display name, wherever `<application_id>` appears.
+
 First, grant access to system schemas for lineage:
 ```sql
 GRANT USE CATALOG ON CATALOG system TO `<application_id>`;