Skip to content

Commit ef609ea

Browse files
authored
Fixed immutable folder failing with DMS or top level permissions (#7001)
## Changes Fixed `experimental.immutable_folder` failing with `experimental.deployment_history` ("workspace_info.file_path must be an absolute workspace path") and failing `bundle validate` when top-level `permissions` are set. ## Why These paths are resolved later during the deployment phase and hence were not correct Fixes #6960 ## Tests Added acceptance tests <!-- If your PR needs to be included in the release notes for next release, add a changelog fragment: create .nextchanges/<section>/<name>.md with a one-line description (e.g. .nextchanges/cli/quickstart.md). See .nextchanges/README.md. -->
1 parent 170a1cc commit ef609ea

22 files changed

Lines changed: 280 additions & 53 deletions

File tree

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* Fixed `experimental.immutable_folder` failing with `experimental.deployment_history` ("workspace_info.file_path must be an absolute workspace path") and failing `bundle validate` when top-level `permissions` are set. ([#7001](https://github.com/databricks/cli/pull/7001))
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
bundle:
2+
name: dms-immutable-folder-$UNIQUE_NAME
3+
experimental:
4+
immutable_folder: true
5+
deployment_history: true
6+
7+
resources:
8+
jobs:
9+
foo:
10+
name: foo
11+
tasks:
12+
- task_key: main
13+
spark_python_task:
14+
python_file: ./src/main.py
15+
environment_key: env
16+
environments:
17+
- environment_key: env
18+
spec:
19+
environment_version: "4"
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Cloud = false
2+
EnvMatrix.DMS = ["true"]
3+
EnvMatrix.READPLAN = ["", "1"]
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
2+
=== Deploying with an immutable folder records the resolved snapshot path as the deployment's file_path
3+
>>> [CLI] bundle deploy
4+
Created internal_immutable_snapshots.immutable
5+
Created jobs.foo
6+
Files: 0 uploaded, 0 deleted
7+
Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
8+
9+
>>> MSYS_NO_PATHCONV=1 [CLI] api get /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
10+
{
11+
"workspace_info": {
12+
"root_path": "/Workspace/Users/[USERNAME]/.bundle/dms-immutable-folder-[UNIQUE_NAME]/default",
13+
"file_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files"
14+
}
15+
}
16+
17+
=== A new snapshot moves file_path, which is then updated on the deployment
18+
>>> [CLI] bundle deploy
19+
Recreated internal_immutable_snapshots.immutable
20+
Updated jobs.foo
21+
Files: 0 uploaded, 0 deleted
22+
Resources: 1 created, 1 changed, 1 deleted, 0 unchanged
23+
24+
>>> MSYS_NO_PATHCONV=1 [CLI] api get /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
25+
{
26+
"workspace_info": {
27+
"root_path": "/Workspace/Users/[USERNAME]/.bundle/dms-immutable-folder-[UNIQUE_NAME]/default",
28+
"file_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files"
29+
}
30+
}
31+
32+
=== An unchanged snapshot keeps the recorded file_path
33+
>>> [CLI] bundle deploy
34+
Files: 0 uploaded, 0 deleted
35+
Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
36+
37+
>>> MSYS_NO_PATHCONV=1 [CLI] api get /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
38+
{
39+
"workspace_info": {
40+
"root_path": "/Workspace/Users/[USERNAME]/.bundle/dms-immutable-folder-[UNIQUE_NAME]/default",
41+
"file_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files"
42+
}
43+
}
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
envsubst < databricks.yml.tmpl > databricks.yml
2+
3+
deployment_workspace_info() {
4+
deployment_id=$(MSYS_NO_PATHCONV=1 $CLI workspace get-status "/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-immutable-folder-${UNIQUE_NAME}/default/state/resources.deployment.json" -o json | python3 -c 'import sys,json; print(json.load(sys.stdin)["object_id"])')
5+
add_repl "$deployment_id" DEPLOYMENT_ID
6+
# MSYS_NO_PATHCONV: Git Bash on Windows would rewrite the leading-'/' API path.
7+
trace MSYS_NO_PATHCONV=1 $CLI api get "/api/2.0/bundle/deployments/${deployment_id}" | jq '{workspace_info}'
8+
}
9+
10+
deploy() {
11+
mkdir -p .databricks
12+
$CLI bundle plan -o json > .databricks/plan.json
13+
trace $CLI bundle deploy $(readplanarg .databricks/plan.json)
14+
rm .databricks/plan.json
15+
}
16+
17+
title "Deploying with an immutable folder records the resolved snapshot path as the deployment's file_path"
18+
deploy
19+
deployment_workspace_info
20+
21+
title "A new snapshot moves file_path, which is then updated on the deployment"
22+
echo 'print("changed")' > src/main.py
23+
deploy
24+
deployment_workspace_info
25+
26+
title "An unchanged snapshot keeps the recorded file_path"
27+
deploy
28+
deployment_workspace_info
29+
30+
rm -f "$OUT_REQUESTS"
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
print("hello")
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# The immutable folder API is not available against a real workspace yet.
2+
Cloud = false
3+
4+
EnvMatrix.READPLAN = ["", "1"]
5+
6+
Ignore = [
7+
'.databricks',
8+
'databricks.yml',
9+
]
10+
11+
# The snapshot path is content-addressed.
12+
[[Repls]]
13+
Old = '[0-9a-f]{64}'
14+
New = '[SNAPSHOT_HASH]'
15+
16+
# When READPLAN=1, "bundle deploy" is called as "bundle deploy --plan .databricks/plan.json".
17+
# Normalize so both variants produce identical output.
18+
[[Repls]]
19+
Old = ' --plan .databricks/plan.json'
20+
New = ''
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
bundle:
2+
name: my-bundle
3+
4+
experimental:
5+
immutable_folder: true
6+
7+
permissions:
8+
- level: CAN_VIEW
9+
user_name: viewer@example.com
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Cloud = false
2+
EnvMatrix.DMS = ["", "true"]
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
2+
>>> [CLI] bundle validate
3+
Recommendation: permissions section should explicitly include the current deployment identity '[USERNAME]' or one of its groups
4+
If it is not included, CAN_MANAGE permissions are only applied if the present identity is used to deploy.
5+
6+
Consider using a adding a top-level permissions section such as the following:
7+
8+
permissions:
9+
- user_name: [USERNAME]
10+
level: CAN_MANAGE
11+
12+
See https://docs.databricks.com/dev-tools/bundles/permissions.html to learn more about permission configuration.
13+
in databricks.yml:8:3
14+
15+
Name: my-bundle
16+
Target: default
17+
Workspace:
18+
User: [USERNAME]
19+
Path: /Workspace/Users/[USERNAME]/.bundle/my-bundle/default
20+
21+
Found 1 recommendation

0 commit comments

Comments
 (0)