Skip to content

Commit 10da8f6

Browse files
Cover best-effort migration through the environment flag
1 parent ed8989c commit 10da8f6

3 files changed

Lines changed: 35 additions & 29 deletions

File tree

‎acceptance/bundle/dms/existing-state/output.txt‎

Lines changed: 25 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -25,29 +25,37 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
2525
"features": null
2626
}
2727

28-
=== Enabling recording is refused because the existing deployment is not recorded
28+
=== The environment flag enables best-effort history for the existing deployment
2929
>>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true [CLI] bundle deploy
30-
Error: deployment history setting (true) does not match the existing state (false)
31-
32-
Update experimental.deployment_history to match the existing deployment, or run "databricks bundle destroy" to start over
33-
30+
Enabling deployment history...
31+
Deployment history enabled: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2
32+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/files...
33+
Updated jobs.one
34+
Files: 0 uploaded, 0 deleted
35+
Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
3436

3537
>>> print_requests.py --dms //api/2.0/bundle --oneline
38+
{"method": "POST", "path": "/api/2.0/bundle/deployments", "body": {"display_name": "dms-existing-state-[UNIQUE_NAME]", "initial_parent_path": "/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/state", "target_name": "default", "workspace_info": {"file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/files", "root_path": "/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default"}}}
39+
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "2"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_INITIAL_REGISTER", "resource_key": "jobs.one"}], "version_type": "VERSION_TYPE_DEPLOY"}}
40+
{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/operations/jobs.one", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[ONE_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"one\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
41+
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
42+
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "3"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.one"}], "previous_version_id": "2", "version_type": "VERSION_TYPE_DEPLOY"}}
43+
{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/operations/jobs.one", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[ONE_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"deployment_id\":\"[DEPLOYMENT_ID]\",\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/state/metadata.json\",\"version_id\":\"3\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"one\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
44+
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
3645

37-
=== Wiping the local cache still refuses after pulling the non-DMS state from the workspace
46+
=== Wiping the local cache pulls history-enabled file state without repeating registration
3847
>>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true [CLI] bundle deploy
39-
Error: deployment history setting (true) does not match the existing state (false)
40-
41-
Update experimental.deployment_history to match the existing deployment, or run "databricks bundle destroy" to start over
42-
48+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/files...
49+
Files: 2 uploaded, 0 deleted
50+
Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
4351

4452
=== A leftover local WAL with the wrong lineage is still refused
4553
>>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true [CLI] bundle deploy
4654
Error: reading state from [TEST_TMP_DIR]/.databricks/bundle/default/resources.json: WAL recovery failed: WAL lineage ("stale") does not match state lineage ("[UUID]")
4755

4856

4957
=== Destroy clears the deployment. With nothing recorded remotely the bundle is treated as new, so recording can be enabled once the stale WAL is gone
50-
>>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true [CLI] bundle destroy --auto-approve
58+
>>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false [CLI] bundle destroy --auto-approve
5159
The following resources will be deleted:
5260
delete resources.jobs.one
5361

@@ -62,15 +70,14 @@ Files: 2 uploaded, 0 deleted
6270
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
6371

6472
>>> print_requests.py --dms //api/2.0/bundle --oneline
65-
{"method": "POST", "path": "/api/2.0/bundle/deployments", "body": {"display_name": "dms-existing-state-[UNIQUE_NAME]", "initial_parent_path": "/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/state", "target_name": "default", "workspace_info": {"file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/files", "root_path": "/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default"}}}
66-
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "2"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_DELETE", "resource_key": "jobs.one"}], "version_type": "VERSION_TYPE_DESTROY"}}
67-
{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/operations/jobs.one", "q": {"update_mask": "state,error_message,status"}, "body": {"error_message": "", "sequence_id": 0, "status": "OPERATION_STATUS_SUCCEEDED"}}
68-
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
73+
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "4"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_DELETE", "resource_key": "jobs.one"}], "previous_version_id": "3", "version_type": "VERSION_TYPE_DESTROY"}}
74+
{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/4/operations/jobs.one", "q": {"update_mask": "state,error_message,status"}, "body": {"error_message": "", "sequence_id": 0, "status": "OPERATION_STATUS_SUCCEEDED"}}
75+
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/4/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
6976
{"method": "DELETE", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}
7077
{"method": "POST", "path": "/api/2.0/bundle/deployments", "body": {"display_name": "dms-existing-state-[UNIQUE_NAME]", "initial_parent_path": "/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/state", "target_name": "default", "workspace_info": {"file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/files", "root_path": "/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default"}}}
71-
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "1"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_CREATE", "resource_key": "jobs.one"}], "version_type": "VERSION_TYPE_DEPLOY"}}
72-
{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/1/operations/jobs.one", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[ONE_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"deployment_id\":\"[DEPLOYMENT_ID]\",\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/state/metadata.json\",\"version_id\":\"1\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"one\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
73-
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/1/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
78+
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID_2]/versions", "q": {"version_id": "1"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_CREATE", "resource_key": "jobs.one"}], "version_type": "VERSION_TYPE_DEPLOY"}}
79+
{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID_2]/versions/1/operations/jobs.one", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[ONE_ID_2]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"deployment_id\":\"[DEPLOYMENT_ID_2]\",\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/state/metadata.json\",\"version_id\":\"1\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"one\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
80+
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID_2]/versions/1/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
7481

7582
>>> find.py resources[.]json[.]wal --expect 0
7683

‎acceptance/bundle/dms/existing-state/script‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,29 +2,32 @@ envsubst < databricks.yml.tmpl > databricks.yml
22

33
title "Deploy without recording: ordinary direct-engine state tracking one job, unknown to DMS"
44
trace $CLI bundle deploy
5+
replace_ids.py
56
dms_plan
67
trace print_requests.py --dms //api/2.0/bundle --oneline
78
trace jq '{state_version, features}' .databricks/bundle/default/resources.json
89

9-
title "Enabling recording is refused because the existing deployment is not recorded"
10-
musterr trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy
10+
title "The environment flag enables best-effort history for the existing deployment"
11+
trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy
12+
deployment_id=$(MSYS_NO_PATHCONV=1 $CLI workspace get-status "/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-existing-state-${UNIQUE_NAME}/default/state/resources.deployment.json" | python3 -c 'import sys, json; print(json.load(sys.stdin)["object_id"])')
13+
add_repl "$deployment_id" DEPLOYMENT_ID
1114
trace print_requests.py --dms //api/2.0/bundle --oneline
1215

13-
title "Wiping the local cache still refuses after pulling the non-DMS state from the workspace"
16+
title "Wiping the local cache pulls history-enabled file state without repeating registration"
1417
rm -rf .databricks
15-
musterr trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy
18+
trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy
1619

1720
title "A leftover local WAL with the wrong lineage is still refused"
1821
mkdir -p .databricks/bundle/default
1922
printf '{"lineage":"stale","serial":99,"state_version":2,"features":{"deployment_history_best_effort":{}}}\n' > .databricks/bundle/default/resources.json.wal
2023
musterr trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy
2124

2225
title "Destroy clears the deployment. With nothing recorded remotely the bundle is treated as new, so recording can be enabled once the stale WAL is gone"
23-
# Clear the stale WAL first: it was refused, not deleted, and destroy runs without recording, so it
26+
# Clear the stale WAL first: it was refused, not deleted, so destroy
2427
# would otherwise try to replay it and fail on the lineage mismatch.
2528
rm -f .databricks/bundle/default/resources.json.wal
2629
# Keep the mismatched setting here to verify that destroy ignores configuration and follows state.
27-
trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle destroy --auto-approve
30+
trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false $CLI bundle destroy --auto-approve
2831
trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy
2932
trace print_requests.py --dms //api/2.0/bundle --oneline
3033
trace find.py 'resources[.]json[.]wal' --expect 0

‎acceptance/bundle/migrate/auto/dms-migration/output.txt‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,7 @@
44

55
=== Direct + deployment-history requested against a terraform state is rejected before commit
66
>>> DATABRICKS_BUNDLE_ENGINE=direct DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true errcode [CLI] bundle deploy
7-
Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
8-
Error: deployment history setting (true) does not match the existing state (false)
9-
10-
Update experimental.deployment_history to match the existing deployment, or run "databricks bundle destroy" to start over
11-
7+
Error: deployment history is not supported with Terraform state; first migrate to direct deployment with deployment history disabled
128

139
Exit code: 1
1410

0 commit comments

Comments
 (0)