Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
bundle:
name: dms-read-failure-$UNIQUE_NAME
experimental:
deployment_history: true

resources:
jobs:
foo:
name: foo
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/read-failure/deploy/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = ["true"]
55 changes: 55 additions & 0 deletions acceptance/bundle/dms/read-failure/deploy/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@

=== Plan with a failed read: the job keeps the deployment id the lookup found, so nothing changes

>>> fault.py GET /api/2.0/bundle/deployments/* 403 0 1 PERMISSION_DENIED

>>> [CLI] bundle plan
Warn: Could not read deployment history, not recording it for this run: Fault injected by test.
Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged

=== Deploy with a failed read: no history is recorded, so no second deployment is created, and the job keeps its deployment id

>>> update_file.py databricks.yml name: foo name: renamed

>>> fault.py GET /api/2.0/bundle/deployments/* 403 0 1 PERMISSION_DENIED

>>> [CLI] bundle deploy
Warn: Could not read deployment history, not recording it for this run: Fault injected by test.
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/files...
Updated jobs.foo
Files: 2 uploaded, 0 deleted
Resources: 0 created, 1 changed, 0 deleted, 0 unchanged

>>> print_requests.py --dms //api/2.0/bundle --get --oneline
{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}

>>> [CLI] jobs get [FOO_ID]
{
"deployment_id": "[DEPLOYMENT_ID]",
"kind": "BUNDLE",
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/state/metadata.json"
}

=== The next deploy resumes history on the same deployment and stamps the new version

>>> update_file.py databricks.yml name: renamed name: resumed

>>> [CLI] bundle deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/files...
Updated jobs.foo
Files: 1 uploaded, 0 deleted
Resources: 0 created, 1 changed, 0 deleted, 0 unchanged

>>> print_requests.py --dms //api/2.0/bundle --get --oneline
{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "3"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.foo"}], "previous_version_id": "1", "version_type": "VERSION_TYPE_DEPLOY"}}
{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/operations/jobs.foo", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[FOO_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"deployment_id\":\"[DEPLOYMENT_ID]\",\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/state/metadata.json\",\"version_id\":\"3\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"resumed\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}

>>> [CLI] jobs get [FOO_ID]
{
"deployment_id": "[DEPLOYMENT_ID]",
"kind": "BUNDLE",
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/state/metadata.json",
"version_id": "3"
}
19 changes: 19 additions & 0 deletions acceptance/bundle/dms/read-failure/deploy/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
deploy_recorded foo

title "Plan with a failed read: the job keeps the deployment id the lookup found, so nothing changes"
fail_deployment_read
trace $CLI bundle plan
rm -f "$OUT_REQUESTS"

title "Deploy with a failed read: no history is recorded, so no second deployment is created, and the job keeps its deployment id"
trace update_file.py databricks.yml 'name: foo' 'name: renamed'
fail_deployment_read
trace $CLI bundle deploy
trace print_requests.py --dms //api/2.0/bundle --get --oneline
trace $CLI jobs get "$foo_id" | jq .settings.deployment

title "The next deploy resumes history on the same deployment and stamps the new version"
trace update_file.py databricks.yml 'name: renamed' 'name: resumed'
trace $CLI bundle deploy
trace print_requests.py --dms //api/2.0/bundle --get --oneline
trace $CLI jobs get "$foo_id" | jq .settings.deployment
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
bundle:
name: dms-read-failure-$UNIQUE_NAME
experimental:
deployment_history: true

resources:
jobs:
foo:
name: foo
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/read-failure/destroy/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = ["true"]
22 changes: 22 additions & 0 deletions acceptance/bundle/dms/read-failure/destroy/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@

=== Destroy with a failed read: no history is recorded, so no second deployment is created and then deleted

>>> fault.py GET /api/2.0/bundle/deployments/* 403 0 1 PERMISSION_DENIED

>>> [CLI] bundle destroy --auto-approve
Warn: Could not read deployment history, not recording it for this run: Fault injected by test.
The following resources will be deleted:
delete resources.jobs.foo

All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default

Warn: Keeping the deployment history record, if any: it could not be read, so this destroy does not delete it
Destroy: 1 deleted

>>> print_requests.py --dms //api/2.0/bundle --get --oneline
{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}

=== The deployment record is kept

>>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments get-deployment deployments/[DEPLOYMENT_ID] -o json
{"last_version_id":"1"}
10 changes: 10 additions & 0 deletions acceptance/bundle/dms/read-failure/destroy/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
deploy_recorded

title "Destroy with a failed read: no history is recorded, so no second deployment is created and then deleted"
fail_deployment_read
trace $CLI bundle destroy --auto-approve
trace print_requests.py --dms //api/2.0/bundle --get --oneline

title "The deployment record is kept"
trace MSYS_NO_PATHCONV=1 $CLI bundle-deployments get-deployment "deployments/${deployment_id}" -o json | jq -c '{last_version_id}'
rm -f "$OUT_REQUESTS"
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
bundle:
name: dms-read-failure-$UNIQUE_NAME
experimental:
deployment_history: true

resources:
jobs:
foo:
name: foo
bar: {name: bar}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = ["true"]
31 changes: 31 additions & 0 deletions acceptance/bundle/dms/read-failure/failed-delete/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@

=== A delete that fails in a run with a failed read still fails the deploy and keeps the job in state

>>> update_file.py databricks.yml bar: {name: bar}

>>> fault.py GET /api/2.0/bundle/deployments/* 403 0 1 PERMISSION_DENIED

>>> fault.py POST /api/2.2/jobs/delete 400 0 1 INVALID_PARAMETER_VALUE

>>> musterr [CLI] bundle deploy
Warn: Could not read deployment history, not recording it for this run: Fault injected by test.
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/files...
Error: cannot delete resources.jobs.bar: deleting id=[BAR_ID]: Fault injected by test. (400 INVALID_PARAMETER_VALUE)

Endpoint: POST [DATABRICKS_URL]/api/2.2/jobs/delete
HTTP Status: 400 Bad Request
API error_code: INVALID_PARAMETER_VALUE
API message: Fault injected by test.

Files: 2 uploaded, 0 deleted

>>> print_requests.py --dms //api/2.0/bundle --get --oneline
{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}

>>> [CLI] jobs get [BAR_ID]
"bar"

>>> [CLI] bundle plan
delete jobs.bar

Plan: 0 to add, 0 to change, 1 to delete, 1 unchanged
11 changes: 11 additions & 0 deletions acceptance/bundle/dms/read-failure/failed-delete/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
deploy_recorded bar

title "A delete that fails in a run with a failed read still fails the deploy and keeps the job in state"
trace update_file.py databricks.yml 'bar: {name: bar}' ''
fail_deployment_read
trace fault.py "POST /api/2.2/jobs/delete" 400 0 1 INVALID_PARAMETER_VALUE
trace musterr $CLI bundle deploy
trace print_requests.py --dms //api/2.0/bundle --get --oneline
trace $CLI jobs get "$bar_id" | jq .settings.name
trace $CLI bundle plan
rm -f "$OUT_REQUESTS"
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
bundle:
name: dms-read-failure-$UNIQUE_NAME
experimental:
deployment_history: true

resources:
jobs:
foo:
name: foo
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/read-failure/not-found/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = ["true"]
19 changes: 19 additions & 0 deletions acceptance/bundle/dms/read-failure/not-found/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@

=== A deployment node without a record (404) is a half-created deployment: the deploy registers it again and records history

>>> update_file.py databricks.yml name: foo name: renamed

>>> fault.py GET /api/2.0/bundle/deployments/* 404 0 1 NOT_FOUND

>>> [CLI] bundle deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/files...
Updated jobs.foo
Files: 2 uploaded, 0 deleted
Resources: 0 created, 1 changed, 0 deleted, 0 unchanged

>>> print_requests.py --dms //api/2.0/bundle --get --oneline
{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}
{"method": "POST", "path": "/api/2.0/bundle/deployments", "body": {"display_name": "dms-read-failure-[UNIQUE_NAME]", "initial_parent_path": "/Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/state", "target_name": "default", "workspace_info": {"file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/files", "root_path": "/Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default"}}}
{"method": "POST", "path": "/api/2.0/bundle/deployments/[NEW_DEPLOYMENT_ID]/versions", "q": {"version_id": "2"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.foo"}], "version_type": "VERSION_TYPE_DEPLOY"}}
{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[NEW_DEPLOYMENT_ID]/versions/2/operations/jobs.foo", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[FOO_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"deployment_id\":\"[NEW_DEPLOYMENT_ID]\",\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-read-failure-[UNIQUE_NAME]/default/state/metadata.json\",\"version_id\":\"2\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"renamed\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
{"method": "POST", "path": "/api/2.0/bundle/deployments/[NEW_DEPLOYMENT_ID]/versions/2/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
9 changes: 9 additions & 0 deletions acceptance/bundle/dms/read-failure/not-found/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
deploy_recorded foo

title "A deployment node without a record (404) is a half-created deployment: the deploy registers it again and records history"
trace update_file.py databricks.yml 'name: foo' 'name: renamed'
trace fault.py "GET /api/2.0/bundle/deployments/*" 404 0 1 NOT_FOUND
trace $CLI bundle deploy
# The fake registers a new node, so the deployment gets a new id.
add_repl "$(deployment_node_id)" NEW_DEPLOYMENT_ID
trace print_requests.py --dms //api/2.0/bundle --get --oneline
25 changes: 25 additions & 0 deletions acceptance/bundle/dms/read-failure/script.prepare
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Each test starts from a recorded deployment (version 1). The setup output is in
# LOG.setup and its requests are drained, so output.txt shows only the step under test.
# Sets deployment_id, and <name>_id for each job name passed.
deploy_recorded() {
envsubst < databricks.yml.tmpl > databricks.yml
$CLI bundle deploy &> LOG.setup
deployment_id=$(deployment_node_id)
add_repl "$deployment_id" DEPLOYMENT_ID
local name
for name in "$@"; do
printf -v "${name}_id" '%s' "$(read_id.py "$name")"
done
rm -f "$OUT_REQUESTS"
}

# The deployment id is the id of the deployment's workspace node. Extracted with
# python, not jq: the id exceeds 2^53 and jq 1.6 rounds it.
deployment_node_id() {
MSYS_NO_PATHCONV=1 $CLI workspace get-status "/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-read-failure-${UNIQUE_NAME}/default/state/resources.deployment.json" | python3 -c 'import sys, json; print(json.load(sys.stdin)["object_id"])'
}

# A 403 is a read failure of unknown cause, and it is not retried.
fail_deployment_read() {
trace fault.py "GET /api/2.0/bundle/deployments/*" 403 0 1 PERMISSION_DENIED
}
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/read-failure/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Local only: fault.py registers its rule on the fake server.
Cloud = false
20 changes: 9 additions & 11 deletions acceptance/bundle/dms_fault/get_deployment/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,8 @@
>>> bundle deploy --auto-approve
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: Could not read deployment history: Fault injected by test.
Warn: Could not read deployment history, not recording it for this run: Fault injected by test.
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files...
Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2
Updated jobs.foo
Updated pipelines.bar
Files: 3 uploaded, 0 deleted
Expand All @@ -25,7 +24,7 @@
>>> bundle summary
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: Could not read deployment history: Fault injected by test.
Warn: Could not read deployment history, not recording it for this run: Fault injected by test.
Name: updated-dms-fault-get_deployment-commands-[UNIQUE_NAME]
Target: default
Workspace:
Expand Down Expand Up @@ -89,13 +88,11 @@
>>> bundle deploy --auto-approve
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: Could not read deployment history: Fault injected by test.
Warn: Could not read deployment history, not recording it for this run: Fault injected by test.
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files...
Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9
Updated jobs.foo
Updated pipelines.bar
Files: 6 uploaded, 0 deleted
Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
Resources: 0 created, 1 changed, 0 deleted, 1 unchanged

=== deployment_history: false

Expand All @@ -111,17 +108,17 @@
>>> bundle deploy --auto-approve
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: Could not read deployment history: Fault injected by test.
Warn: Could not read deployment history, not recording it for this run: Fault injected by test.
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files...
Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11
Updated jobs.foo
Updated pipelines.bar
Files: 2 uploaded, 0 deleted
Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
Resources: 0 created, 2 changed, 0 deleted, 0 unchanged

>>> bundle destroy --auto-approve
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
Warn: Could not read deployment history: Fault injected by test.
Warn: Could not read deployment history, not recording it for this run: Fault injected by test.
The following resources will be deleted:
delete resources.jobs.foo
delete resources.pipelines.bar
Expand All @@ -132,6 +129,7 @@

All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]

Warn: Keeping the deployment history record, if any: it could not be read, so this destroy does not delete it
Destroy: 2 deleted

=== deployment_history: false
Expand Down
5 changes: 3 additions & 2 deletions bundle/direct/apply.go
Original file line number Diff line number Diff line change
Expand Up @@ -309,8 +309,9 @@ func (d *DeploymentUnit) Delete(ctx context.Context, db *dstate.DeploymentState,
log.Warnf(ctx, "Ignoring permission error when deleting %s id=%s: %s", d.ResourceKey, oldID, err)
} else if d.deleteConfirmedGone(ctx, oldID) {
log.Warnf(ctx, "Treating %s id=%s as already deleted despite delete error: %s", d.ResourceKey, oldID, err)
} else if db.RecordsHistory() {
// When using the deployment metadata service, record the error.
} else if db.RecordsHistory() || db.HistorySkipped() {
// When using the deployment metadata service, record the error. A run that skips history
// after a failed DMS read still fails here, so a DMS outage does not change delete behavior.
err = fmt.Errorf("deleting id=%s: %w", oldID, err)
db.RecordFailure(d.ResourceKey, oldID, err)
return err
Expand Down
Loading
Loading