diff --git a/docs/cloud/integrations/bi/hex.mdx b/docs/cloud/integrations/bi/hex.mdx index fb5e5be30..84a560755 100644 --- a/docs/cloud/integrations/bi/hex.mdx +++ b/docs/cloud/integrations/bi/hex.mdx @@ -6,6 +6,18 @@ badge: "Context Engine" After you connect Hex, Elementary will automatically and continuously extend the lineage to the project & cell level. This will provide you end-to-end data lineage to understand your downstream dependencies, called exposures. +### Prerequisites + +The Hex integration relies on **query history** to discover which tables Hex projects query. It is supported on **BigQuery** and **Snowflake**. + +Elementary's service account must have permissions to read query history. Refer to your warehouse's setup page for the required roles: +- [BigQuery permissions](/cloud/integrations/dwh/bigquery#grant-service-user-access) +- [Snowflake permissions](/cloud/integrations/dwh/snowflake) + + +If Elementary cannot read query history, the Hex integration will not discover any exposures. + + ### Create API Token Elementary needs a workspace token in your account in order to access Hex's API on your behalf.
@@ -20,6 +32,15 @@ Choose the Hex connection and provide the following details to validate and comp - **Base URL**: Your Hex workspace URL. For example: `https://app.hex.tech/my-workspace/` - **Workspace Token**: The Hex workspace token you've created on the previous step. +### Hex queries on a different warehouse environment + +By default, Elementary reads query history from the warehouse environments (e.g., BigQuery projects, Snowflake accounts) referenced in your dbt project. If Hex executes queries through a **different environment** (e.g., a dedicated Hex execution environment), you will need to: + +1. **Grant query history permissions** — the same permissions required for your primary environment. See the [prerequisites](#prerequisites) above for links to your warehouse's permissions page. + +2. **Let us know** — contact Elementary support with the environment details so we can configure Elementary to also read query history from it. + +Without both steps, Elementary will not find any Hex queries in that environment and no Hex exposures will appear. ### Limitations diff --git a/docs/snippets/cloud/integrations/bigquery.mdx b/docs/snippets/cloud/integrations/bigquery.mdx index 3be5e6381..a3d661837 100644 --- a/docs/snippets/cloud/integrations/bigquery.mdx +++ b/docs/snippets/cloud/integrations/bigquery.mdx @@ -25,6 +25,10 @@ Select a tab below and follow the steps for your chosen method. +For BigQuery, this translates to: +- **BigQuery Data Viewer** on the Elementary dataset (for reading Elementary schema data). +- **BigQuery Metadata Viewer** and **BigQuery Resource Viewer** at the **project level** (for reading `INFORMATION_SCHEMA.JOBS` query history and table metadata). These project-level roles are required for BI integrations like Hex that rely on query history. + ### Fill the connection form Use the **Authentication method** toggle at the top of the form to select either **Service account** or **Workload Identity Federation**, matching the method you set up above. The credentials upload field changes based on your selection: diff --git a/docs/snippets/cloud/integrations/permissions-and-security.mdx b/docs/snippets/cloud/integrations/permissions-and-security.mdx index ed4e58c23..57bafc763 100644 --- a/docs/snippets/cloud/integrations/permissions-and-security.mdx +++ b/docs/snippets/cloud/integrations/permissions-and-security.mdx @@ -1,9 +1,9 @@ #### Permissions and security -Elementary cloud doesn't require read permissions to your tables and schemas, but only the following: +Elementary Cloud doesn't require read permissions to your tables and schemas, but only the following: -- Read-only access to the elementary schema. -- Access to read metadata in information schema and query history, related to the tables in your dbt project. +- Read-only access to the Elementary schema. +- Access to read metadata in `INFORMATION_SCHEMA` (table metadata and query history), related to the tables in your dbt project. It is recommended to create a user using the instructions specified above to avoid granting excess privileges. For more details, refer to [security and privacy](/cloud/general/security-and-privacy). diff --git a/docs/snippets/dwh/bigquery/grant_user_access_on_dataset_level.mdx b/docs/snippets/dwh/bigquery/grant_user_access_on_dataset_level.mdx index 5c1a22d5e..fec0be6fb 100644 --- a/docs/snippets/dwh/bigquery/grant_user_access_on_dataset_level.mdx +++ b/docs/snippets/dwh/bigquery/grant_user_access_on_dataset_level.mdx @@ -1,15 +1,39 @@ -### Grant service user access to specific datasets +### Grant service user access -In order for a service user to work with Elementary cloud, it requires the following permissions: +In order for a service user to work with Elementary Cloud, it requires the following permissions: -- Role "BigQuery Data Viewer" on your Elementary dataset. -- Roles "BigQuery Metadata Viewer", "BigQuery Resource Viewer" on the entire project and any external sources it is referencing. +- Role **"BigQuery Data Viewer"** on your Elementary dataset. +- Roles **"BigQuery Metadata Viewer"** and **"BigQuery Resource Viewer"** at the **project level** for the project containing your dbt models and any external projects it references. -To grant a role on a specific dataset, follow these steps: + +"BigQuery Metadata Viewer" and "BigQuery Resource Viewer" **must be granted at the project level** (via IAM & Admin), not at the dataset level. These roles include permissions such as `bigquery.jobs.listAll` which are required for reading query history and metadata from `INFORMATION_SCHEMA` — they cannot be granted on individual datasets. + -1. Go to your project in [BigQuery console](https://console.cloud.google.com/bigquery) +#### Step 1: Grant project-level roles -2. In the "Explorer" tab, find your desired dataset. +These roles are required for Elementary to read query history (`INFORMATION_SCHEMA.JOBS`) and table metadata. Without them, BI integrations (such as Hex) that rely on query history will not function. + +1. Go to [IAM & Admin](https://console.cloud.google.com/iam-admin/iam) in the Google Cloud Console. + +2. Select the project that contains your dbt models. + +3. Click **"Grant Access"** at the top. + +4. Fill out the form: + - In the "New principals" field, enter the **email address** of the Elementary service account. + - Add the role **"BigQuery Metadata Viewer"** (`roles/bigquery.metadataViewer`). + - Click **"Add another role"** and add **"BigQuery Resource Viewer"** (`roles/bigquery.resourceViewer`). + - Click **"Save"**. + +5. **Repeat for any additional GCP projects** referenced in your dbt sources or used by BI tools (e.g., Hex, Looker) for query execution. + +#### Step 2: Grant dataset-level role on the Elementary dataset + +This role provides read access to your Elementary schema data. + +1. Go to your project in [BigQuery console](https://console.cloud.google.com/bigquery). + +2. In the "Explorer" tab, find your Elementary dataset. 3. Click on the three dots icon next to the dataset name, then Share. @@ -26,13 +50,11 @@ To grant a role on a specific dataset, follow these steps: /> 5. Fill out the form: - - In the "New principals" textbox, write the **email address** of your user. - - In the "Select a role" dropdown menu, choose the desired role (BigQuery Data Viewer for your Elementary dataset, BigQuery Metadata Viewer, BigQuery Resource Viewer for your dbt dataset). + - In the "New principals" textbox, write the **email address** of your service account. + - In the "Select a role" dropdown menu, choose **"BigQuery Data Viewer"**. - Click "Save". Grant access - -Make sure to grant the correct access to your Elementary dataset **and** your dbt dataset.