Skip to content

Commit e58e2c3

Browse files
denikIsaac
denik
and
Isaac
committed
acceptance: record that terraform resource-type references fail on direct
Bundles referencing resources as ${databricks_<type>.<name>...} deployed on terraform, but the direct engine only supports terraform field paths inside ${resources.*}: plan and deploy fail, both directly and after auto-migration from a replayed terraform deployment. Co-authored-by: Isaac <no-reply@databricks.com>
1 parent bd463dd commit e58e2c3

10 files changed

Lines changed: 418 additions & 0 deletions

File tree

‎acceptance/bundle/migrate/auto/tf-resource-type-references/databricks.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
bundle:
2+
name: test-bundle
3+
4+
resources:
5+
pipelines:
6+
src_pipeline:
7+
name: source-pipeline
8+
9+
jobs:
10+
dst_job:
11+
name: dst-job
12+
tasks:
13+
- task_key: main
14+
pipeline_task:
15+
# Terraform-style resource reference: databricks_pipeline is the terraform
16+
# resource type for DABs "pipelines".
17+
pipeline_id: ${databricks_pipeline.src_pipeline.id}

‎acceptance/bundle/migrate/auto/tf-resource-type-references/out.test.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Cloud = false
2+
EnvMatrix.DMS = [""]

‎acceptance/bundle/migrate/auto/tf-resource-type-references/output.txt‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
2+
=== Replay a prior terraform deployment
3+
>>> replay_tfstate.py terraform-requests.json terraform.tfstate
4+
5+
=== Plan after auto-migration
6+
>>> musterr [CLI] bundle plan
7+
Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
8+
Error: invalid dependency "${databricks_pipeline.src_pipeline.id}", no such node ""
9+
10+
11+
=== Deploy after auto-migration
12+
>>> musterr [CLI] bundle deploy
13+
Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
14+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
15+
Error: invalid dependency "${databricks_pipeline.src_pipeline.id}", no such node ""
16+
17+
Files: 1 uploaded, 0 deleted
18+
19+
>>> [CLI] bundle destroy --auto-approve
20+
Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
21+
The following resources will be deleted:
22+
delete resources.jobs.dst_job
23+
delete resources.pipelines.src_pipeline
24+
25+
This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
26+
Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
27+
delete resources.pipelines.src_pipeline
28+
29+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
30+
31+
Migrated 2 resources to direct deployment engine.
32+
Destroy: 2 deleted

‎acceptance/bundle/migrate/auto/tf-resource-type-references/script‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
export DATABRICKS_BUNDLE_ENGINE=
2+
3+
# The config references the pipeline by its terraform resource type (${databricks_pipeline...}),
4+
# which terraform resolved on the original deploy. The direct engine does not support this syntax
5+
# (only terraform field paths inside ${resources.*}), so after auto-migration plan and deploy fail.
6+
title "Replay a prior terraform deployment"
7+
trace replay_tfstate.py terraform-requests.json terraform.tfstate
8+
rm -f "$OUT_REQUESTS"
9+
10+
title "Plan after auto-migration"
11+
trace musterr $CLI bundle plan
12+
13+
title "Deploy after auto-migration"
14+
trace musterr $CLI bundle deploy
15+
rm -f "$OUT_REQUESTS"
16+
17+
trace $CLI bundle destroy --auto-approve
18+
rm -f "$OUT_REQUESTS"

‎acceptance/bundle/migrate/auto/tf-resource-type-references/terraform-requests.json‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
[
2+
{
3+
"method": "POST",
4+
"path": "/api/2.0/workspace/mkdirs",
5+
"body": {
6+
"path": "/Workspace/Users/tester@databricks.com/.bundle/test-bundle/default/state"
7+
}
8+
},
9+
{
10+
"method": "POST",
11+
"path": "/api/2.0/workspace/mkdirs",
12+
"body": {
13+
"path": "/Workspace/Users/tester@databricks.com/.bundle/test-bundle/default/artifacts/.internal"
14+
}
15+
},
16+
{
17+
"method": "POST",
18+
"path": "/api/2.0/workspace/mkdirs",
19+
"body": {
20+
"path": "/Workspace/Users/tester@databricks.com/.bundle/test-bundle/default/files"
21+
}
22+
},
23+
{
24+
"method": "POST",
25+
"path": "/api/2.0/pipelines",
26+
"body": {
27+
"channel": "CURRENT",
28+
"deployment": {
29+
"kind": "BUNDLE",
30+
"metadata_file_path": "/Workspace/Users/tester@databricks.com/.bundle/test-bundle/default/state/metadata.json"
31+
},
32+
"edition": "ADVANCED",
33+
"name": "source-pipeline"
34+
}
35+
},
36+
{
37+
"method": "POST",
38+
"path": "/api/2.2/jobs/create",
39+
"body": {
40+
"deployment": {
41+
"kind": "BUNDLE",
42+
"metadata_file_path": "/Workspace/Users/tester@databricks.com/.bundle/test-bundle/default/state/metadata.json"
43+
},
44+
"edit_mode": "UI_LOCKED",
45+
"format": "MULTI_TASK",
46+
"max_concurrent_runs": 1,
47+
"name": "dst-job",
48+
"queue": {
49+
"enabled": true
50+
},
51+
"tasks": [
52+
{
53+
"pipeline_task": {
54+
"pipeline_id": "7a00a8c6-b18f-59a8-7a00-a8c6b18f59a9"
55+
},
56+
"task_key": "main"
57+
}
58+
]
59+
}
60+
}
61+
]

‎acceptance/bundle/migrate/auto/tf-resource-type-references/terraform.tfstate‎

Lines changed: 244 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,244 @@
1+
{
2+
"version": 4,
3+
"terraform_version": "1.5.5",
4+
"serial": 3,
5+
"lineage": "feddf884-7b6a-3c90-968d-3dcc4761437e",
6+
"outputs": {},
7+
"resources": [
8+
{
9+
"mode": "managed",
10+
"type": "databricks_job",
11+
"name": "dst_job",
12+
"provider": "provider[\"registry.terraform.io/databricks/databricks\"]",
13+
"instances": [
14+
{
15+
"schema_version": 2,
16+
"attributes": {
17+
"always_running": false,
18+
"budget_policy_id": null,
19+
"continuous": [],
20+
"control_run_state": false,
21+
"dbt_task": [],
22+
"deployment": [
23+
{
24+
"deployment_id": "",
25+
"kind": "BUNDLE",
26+
"metadata_file_path": "/Workspace/Users/tester@databricks.com/.bundle/test-bundle/default/state/metadata.json",
27+
"version_id": ""
28+
}
29+
],
30+
"description": null,
31+
"edit_mode": "UI_LOCKED",
32+
"email_notifications": [
33+
{
34+
"no_alert_for_skipped_runs": false,
35+
"on_duration_warning_threshold_exceeded": [],
36+
"on_failure": [],
37+
"on_start": [],
38+
"on_streaming_backlog_exceeded": [],
39+
"on_success": []
40+
}
41+
],
42+
"environment": [],
43+
"existing_cluster_id": null,
44+
"format": "MULTI_TASK",
45+
"git_source": [],
46+
"health": [],
47+
"id": "8791212044006401000",
48+
"job_cluster": [],
49+
"library": [],
50+
"max_concurrent_runs": 1,
51+
"max_retries": 0,
52+
"min_retry_interval_millis": 0,
53+
"name": "dst-job",
54+
"new_cluster": [],
55+
"notebook_task": [],
56+
"notification_settings": [],
57+
"parameter": [],
58+
"parent_path": null,
59+
"performance_target": null,
60+
"pipeline_task": [],
61+
"provider_config": [
62+
{
63+
"workspace_id": "900800700600"
64+
}
65+
],
66+
"python_wheel_task": [],
67+
"queue": [
68+
{
69+
"enabled": true
70+
}
71+
],
72+
"retry_on_timeout": false,
73+
"run_as": [
74+
{
75+
"group_name": "",
76+
"service_principal_name": "",
77+
"user_name": "tester@databricks.com"
78+
}
79+
],
80+
"run_job_task": [],
81+
"schedule": [],
82+
"spark_jar_task": [],
83+
"spark_python_task": [],
84+
"spark_submit_task": [],
85+
"tags": null,
86+
"task": [
87+
{
88+
"ai_runtime_task": [],
89+
"alert_task": [],
90+
"clean_rooms_notebook_task": [],
91+
"compute": [],
92+
"condition_task": [],
93+
"dashboard_task": [],
94+
"dbt_cloud_task": [],
95+
"dbt_platform_task": [],
96+
"dbt_task": [],
97+
"depends_on": [],
98+
"description": "",
99+
"disable_auto_optimization": false,
100+
"disabled": false,
101+
"email_notifications": [
102+
{
103+
"no_alert_for_skipped_runs": false,
104+
"on_duration_warning_threshold_exceeded": [],
105+
"on_failure": [],
106+
"on_start": [],
107+
"on_streaming_backlog_exceeded": [],
108+
"on_success": []
109+
}
110+
],
111+
"environment_key": "",
112+
"existing_cluster_id": "",
113+
"for_each_task": [],
114+
"gen_ai_compute_task": [],
115+
"health": [],
116+
"job_cluster_key": "",
117+
"library": [],
118+
"max_retries": 0,
119+
"min_retry_interval_millis": 0,
120+
"new_cluster": [],
121+
"notebook_task": [],
122+
"notification_settings": [],
123+
"pipeline_task": [
124+
{
125+
"full_refresh": false,
126+
"full_refresh_selection": null,
127+
"parameters": null,
128+
"pipeline_id": "7a00a8c6-b18f-59a8-7a00-a8c6b18f59a9",
129+
"refresh_flow_selection": null,
130+
"refresh_selection": null,
131+
"reset_checkpoint_selection": null
132+
}
133+
],
134+
"power_bi_task": [],
135+
"python_operator_task": [],
136+
"python_wheel_task": [],
137+
"retry_on_timeout": false,
138+
"run_if": "ALL_SUCCESS",
139+
"run_job_task": [],
140+
"spark_jar_task": [],
141+
"spark_python_task": [],
142+
"spark_submit_task": [],
143+
"sql_task": [],
144+
"task_key": "main",
145+
"timeout_seconds": 0,
146+
"webhook_notifications": []
147+
}
148+
],
149+
"timeout_seconds": 0,
150+
"timeouts": null,
151+
"trigger": [],
152+
"triggers": [],
153+
"url": "[DATABRICKS_URL]/#job/8791212044006401000",
154+
"usage_policy_id": null,
155+
"webhook_notifications": [
156+
{
157+
"on_duration_warning_threshold_exceeded": [],
158+
"on_failure": [],
159+
"on_start": [],
160+
"on_streaming_backlog_exceeded": [],
161+
"on_success": []
162+
}
163+
]
164+
},
165+
"sensitive_attributes": [],
166+
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxODAwMDAwMDAwMDAwLCJ1cGRhdGUiOjE4MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjIifQ==",
167+
"dependencies": [
168+
"databricks_pipeline.src_pipeline"
169+
]
170+
}
171+
]
172+
},
173+
{
174+
"mode": "managed",
175+
"type": "databricks_pipeline",
176+
"name": "src_pipeline",
177+
"provider": "provider[\"registry.terraform.io/databricks/databricks\"]",
178+
"instances": [
179+
{
180+
"schema_version": 0,
181+
"attributes": {
182+
"allow_duplicate_names": false,
183+
"budget_policy_id": null,
184+
"catalog": null,
185+
"cause": null,
186+
"channel": "CURRENT",
187+
"cluster": [],
188+
"cluster_id": null,
189+
"configuration": null,
190+
"continuous": false,
191+
"creator_user_name": "tester@databricks.com",
192+
"deployment": [
193+
{
194+
"deployment_id": "",
195+
"kind": "BUNDLE",
196+
"metadata_file_path": "/Workspace/Users/tester@databricks.com/.bundle/test-bundle/default/state/metadata.json",
197+
"version_id": ""
198+
}
199+
],
200+
"development": false,
201+
"edition": "ADVANCED",
202+
"environment": [],
203+
"event_log": [],
204+
"expected_last_modified": 0,
205+
"filters": [],
206+
"gateway_definition": [],
207+
"health": null,
208+
"id": "7a00a8c6-b18f-59a8-7a00-a8c6b18f59a9",
209+
"ingestion_definition": [],
210+
"last_modified": 1791212043963,
211+
"latest_updates": null,
212+
"library": [],
213+
"name": "source-pipeline",
214+
"notification": [],
215+
"photon": false,
216+
"provider_config": [
217+
{
218+
"workspace_id": "900800700600"
219+
}
220+
],
221+
"restart_window": [],
222+
"root_path": null,
223+
"run_as": [],
224+
"run_as_user_name": "tester@databricks.com",
225+
"schema": null,
226+
"serverless": false,
227+
"serverless_compute_id": null,
228+
"state": "IDLE",
229+
"storage": "dbfs:/pipelines/7a00a8c6-b18f-59a8-7a00-a8c6b18f59a9",
230+
"tags": null,
231+
"target": null,
232+
"timeouts": null,
233+
"trigger": [],
234+
"url": "[DATABRICKS_URL]/#joblist/pipelines/7a00a8c6-b18f-59a8-7a00-a8c6b18f59a9",
235+
"usage_policy_id": null
236+
},
237+
"sensitive_attributes": [],
238+
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWZhdWx0IjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19"
239+
}
240+
]
241+
}
242+
],
243+
"check_results": null
244+
}

‎acceptance/bundle/resource_deps/tf_resource_type_ref/databricks.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
bundle:
2+
name: test-bundle
3+
4+
resources:
5+
pipelines:
6+
src_pipeline:
7+
name: source-pipeline
8+
9+
jobs:
10+
dst_job:
11+
name: dst-job
12+
tasks:
13+
- task_key: main
14+
pipeline_task:
15+
# Terraform-style resource reference: databricks_pipeline is the terraform
16+
# resource type for DABs "pipelines".
17+
pipeline_id: ${databricks_pipeline.src_pipeline.id}

0 commit comments

Comments
 (0)