Skip to content

Route bundle test clusters through the shared instance pool #5461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
renaudhartert-db wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Route bundle test clusters through the shared instance pool #5461

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
354063e
acceptance: route bundle test clusters through the shared instance pool
renaudhartert-db Jun 7, 2026
68fdd48
Drop job-spark-python-task from pool routing
renaudhartert-db Jun 7, 2026
3d5eb7e
Drop run_as/job_default from pool routing
renaudhartert-db Jun 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions acceptance/bundle/invariant/configs/cluster.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ resources:
cluster_name: test-cluster-$UNIQUE_NAME
spark_version: 13.3.x-scala2.12
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 1
1 change: 1 addition & 0 deletions acceptance/bundle/invariant/configs/job_with_task.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@ resources:
new_cluster:
spark_version: $DEFAULT_SPARK_VERSION
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 1
1 change: 1 addition & 0 deletions acceptance/bundle/resources/clusters/deploy/data_security_mode/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ resources:
cluster_name: test-cluster-$UNIQUE_NAME
spark_version: $DEFAULT_SPARK_VERSION
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 2
spark_conf:
"spark.executor.memory": "2g"
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/resources/clusters/deploy/local_ssd_count/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ resources:
cluster_name: test-cluster-$UNIQUE_NAME
spark_version: $DEFAULT_SPARK_VERSION
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 1
gcp_attributes:
# config from https://github.com/databricks/terraform-provider-databricks/issues/4089
Expand Down
8 changes: 2 additions & 6 deletions acceptance/bundle/resources/clusters/deploy/local_ssd_count/out.requests.direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,8 @@
"body": {
"autotermination_minutes": 60,
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"gcp_attributes": {
"availability": "PREEMPTIBLE_GCP",
"local_ssd_count": 0,
"zone_id": "auto"
},
"node_type_id": "[NODE_TYPE_ID]",
"gcp_attributes": {},
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 1,
"spark_version": "13.3.x-snapshot-scala2.12"
}
Expand Down
7 changes: 2 additions & 5 deletions acceptance/bundle/resources/clusters/deploy/local_ssd_count/out.requests.terraform.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,8 @@
"body": {
"autotermination_minutes": 60,
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"gcp_attributes": {
"availability": "PREEMPTIBLE_GCP",
"zone_id": "auto"
},
"node_type_id": "[NODE_TYPE_ID]",
"gcp_attributes": {},
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 1,
"spark_version": "13.3.x-snapshot-scala2.12"
}
Expand Down
3 changes: 0 additions & 3 deletions acceptance/bundle/resources/clusters/deploy/local_ssd_count/out.result.direct.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,2 @@
json.gcp_attributes.availability = "PREEMPTIBLE_GCP";
json.gcp_attributes.first_on_demand = 1;
json.gcp_attributes.local_ssd_count = 0;
json.gcp_attributes.use_preemptible_executors = false;
json.gcp_attributes.zone_id = "auto";
2 changes: 0 additions & 2 deletions acceptance/bundle/resources/clusters/deploy/local_ssd_count/out.result.terraform.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,2 @@
json.gcp_attributes.availability = "PREEMPTIBLE_GCP";
json.gcp_attributes.first_on_demand = 1;
json.gcp_attributes.use_preemptible_executors = false;
json.gcp_attributes.zone_id = "auto";
1 change: 1 addition & 0 deletions acceptance/bundle/resources/clusters/deploy/simple/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ resources:
cluster_name: test-cluster-$UNIQUE_NAME
spark_version: $DEFAULT_SPARK_VERSION
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 2
spark_conf:
"spark.executor.memory": "2g"
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/resources/clusters/deploy/update-after-create/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ resources:
cluster_name: test-cluster-$UNIQUE_NAME
spark_version: $DEFAULT_SPARK_VERSION
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 2
spark_conf:
"spark.executor.memory": "2g"
1 change: 1 addition & 0 deletions acceptance/bundle/resources/clusters/deploy/update-and-resize/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ resources:
cluster_name: test-cluster-$UNIQUE_NAME
spark_version: $DEFAULT_SPARK_VERSION
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 2
spark_conf:
"spark.executor.memory": "2g"
32 changes: 7 additions & 25 deletions acceptance/bundle/resources/clusters/deploy/update-and-resize/out.plan_.direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"value": {
"autotermination_minutes": 60,
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"node_type_id": "[NODE_TYPE_ID]",
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 2,
"spark_conf": {
"spark.executor.memory": "2g"
Expand All @@ -31,7 +31,7 @@
"value": {
"autotermination_minutes": 60,
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"node_type_id": "[NODE_TYPE_ID]",
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 3,
"spark_conf": {
"spark.executor.memory": "2g"
Expand All @@ -47,12 +47,11 @@
},
"cluster_id": "[CLUSTER_ID]",
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"driver_node_type_id": "[NODE_TYPE_ID]",
"enable_elastic_disk": false,
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"lifecycle": {
"started": true
},
"node_type_id": "[NODE_TYPE_ID]",
"num_workers": 2,
"spark_conf": {
"spark.executor.memory": "2g"
Expand All @@ -69,11 +68,6 @@
"zone_id": "us-east-1c"
}
},
"driver_node_type_id": {
"action": "skip",
"reason": "backend_default",
"remote": "[NODE_TYPE_ID]"
},
"enable_elastic_disk": {
"action": "skip",
"reason": "empty",
Expand Down Expand Up @@ -109,7 +103,7 @@
"value": {
"autotermination_minutes": 60,
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"node_type_id": "[NODE_TYPE_ID]",
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 4,
"spark_conf": {
"spark.executor.memory": "2g"
Expand All @@ -125,12 +119,11 @@
},
"cluster_id": "[CLUSTER_ID]",
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"driver_node_type_id": "[NODE_TYPE_ID]",
"enable_elastic_disk": false,
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"lifecycle": {
"started": true
},
"node_type_id": "[NODE_TYPE_ID]",
"num_workers": 3,
"spark_conf": {
"spark.executor.memory": "2g"
Expand All @@ -147,11 +140,6 @@
"zone_id": "us-east-1c"
}
},
"driver_node_type_id": {
"action": "skip",
"reason": "backend_default",
"remote": "[NODE_TYPE_ID]"
},
"enable_elastic_disk": {
"action": "skip",
"reason": "empty",
Expand Down Expand Up @@ -187,7 +175,7 @@
"value": {
"autotermination_minutes": 60,
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"node_type_id": "[NODE_TYPE_ID]",
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 5,
"spark_conf": {
"spark.executor.memory": "4g"
Expand All @@ -203,12 +191,11 @@
},
"cluster_id": "[CLUSTER_ID]",
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"driver_node_type_id": "[NODE_TYPE_ID]",
"enable_elastic_disk": false,
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"lifecycle": {
"started": true
},
"node_type_id": "[NODE_TYPE_ID]",
"num_workers": 4,
"spark_conf": {
"spark.executor.memory": "2g"
Expand All @@ -225,11 +212,6 @@
"zone_id": "us-east-1c"
}
},
"driver_node_type_id": {
"action": "skip",
"reason": "backend_default",
"remote": "[NODE_TYPE_ID]"
},
"enable_elastic_disk": {
"action": "skip",
"reason": "empty",
Expand Down
5 changes: 2 additions & 3 deletions acceptance/bundle/resources/clusters/deploy/update-and-resize/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,8 @@ Deployment complete!
},
"cluster_id": "[CLUSTER_ID]",
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"driver_node_type_id": "[NODE_TYPE_ID]",
"enable_elastic_disk": false,
"node_type_id": "[NODE_TYPE_ID]",
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 3,
"spark_conf": {
"spark.executor.memory": "2g"
Expand Down Expand Up @@ -87,7 +86,7 @@ Deployment complete!
"autotermination_minutes": 60,
"cluster_id": "[CLUSTER_ID]",
"cluster_name": "test-cluster-[UNIQUE_NAME]",
"node_type_id": "[NODE_TYPE_ID]",
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 5,
"spark_conf": {
"spark.executor.memory": "4g"
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/resources/clusters/lifecycle-started-terraform-error/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ resources:
cluster_name: $UNIQUE_NAME
spark_version: 15.4.x-scala2.12
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 1
lifecycle:
started: true
4 changes: 2 additions & 2 deletions acceptance/bundle/resources/clusters/lifecycle-started-terraform-error/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@
=== bundle plan fails with lifecycle.started on terraform engine
>>> errcode [CLI] bundle plan
Error: lifecycle.started is only supported in direct deployment mode
in databricks.yml:15:18
in databricks.yml:16:18


Exit code: 1

=== bundle deploy fails with lifecycle.started on terraform engine
>>> errcode [CLI] bundle deploy
Error: lifecycle.started is only supported in direct deployment mode
in databricks.yml:15:18
in databricks.yml:16:18


Exit code: 1
1 change: 1 addition & 0 deletions acceptance/bundle/resources/clusters/lifecycle-started-toggle/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ resources:
cluster_name: $UNIQUE_NAME
spark_version: 15.4.x-scala2.12
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 1
lifecycle:
started: false
2 changes: 1 addition & 1 deletion acceptance/bundle/resources/clusters/lifecycle-started-toggle/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Deployment complete!
"body": {
"autotermination_minutes": 60,
"cluster_name": "[UNIQUE_NAME]",
"node_type_id": "[NODE_TYPE_ID]",
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 1,
"spark_version": "15.4.x-scala2.12"
}
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/resources/clusters/lifecycle-started/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ resources:
cluster_name: $UNIQUE_NAME
spark_version: 15.4.x-scala2.12
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
num_workers: 1
lifecycle:
started: true
2 changes: 1 addition & 1 deletion acceptance/bundle/resources/clusters/lifecycle-started/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Deployment complete!
"body": {
"autotermination_minutes": 60,
"cluster_name": "[UNIQUE_NAME]",
"node_type_id": "[NODE_TYPE_ID]",
"instance_pool_id": "[TEST_INSTANCE_POOL_ID]",
"num_workers": 1,
"spark_version": "15.4.x-scala2.12"
}
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/run/app-with-job/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ resources:
num_workers: 1
spark_version: $DEFAULT_SPARK_VERSION
node_type_id: $NODE_TYPE_ID
instance_pool_id: $TEST_INSTANCE_POOL_ID
data_security_mode: USER_ISOLATION
spark_python_task:
python_file: ./hello_world.py
Loading