Skip to content

Commit 5bda5d4

Browse files
ilyakuz-dbIsaac
andauthored
Avoid syncing cluster policy values into shared bundle config (#7170)
## Changes `config-remote-sync` no longer copies cluster policy values into bundle YAML. After changes are extracted and selected, job and cluster specs whose remote `policy_id` is set are checked against that policy (read lazily, once per sync): - Fields under `fixed` rules, including `*` wildcards, are never synced. The backend rejects any other value for them, so the remote value always comes from the policy. - Fields under other rules are dropped only when they equal the policy `defaultValue` and `apply_policy_default_values` is set; any other value is a user edit and syncs. - Values authored in YAML are kept, and policy values never remove them. - If the policy no longer exists, policy-dependent fields are listed in `skippedChanges` with the new `partial` status, and other edits still sync. Clusters without a policy are not filtered. The blanket `custom_tags`/`cluster_log_conf` skip is removed, so tags added in the UI now sync. ## Why Syncing a UI edit could copy one target's policy values into shared YAML, and deploying that YAML to a target with a different policy then failed validation. ## Tests Unit tests for rule matching, plus acceptance tests for cross-target deploys, missing policies and deploy-and-plan after sync. _This PR was written with assistance from Codex._ This pull request and its description were written by Isaac. --------- Co-authored-by: ilyakuz-db <181845775+ilyakuz-db@users.noreply.github.com> Co-authored-by: Isaac <no-reply@databricks.com>
1 parent a3fd25d commit 5bda5d4

46 files changed

Lines changed: 2360 additions & 116 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
bundle:
2+
name: test-bundle-$UNIQUE_NAME
3+
4+
workspace:
5+
root_path: ~/.bundle/$UNIQUE_NAME/${bundle.target}
6+
7+
variables:
8+
environment:
9+
default: dev
10+
11+
resources:
12+
cluster_policies:
13+
policy:
14+
name: test-policy-$UNIQUE_NAME-${bundle.target}
15+
definition:
16+
custom_tags.workspace:
17+
type: fixed
18+
value: ${var.environment}
19+
spark_env_vars.ENVIRONMENT:
20+
type: fixed
21+
value: ${var.environment}
22+
workload_type.clients.jobs:
23+
type: fixed
24+
value: true
25+
workload_type.clients.notebooks:
26+
type: fixed
27+
value: false
28+
jobs:
29+
policy_sync:
30+
name: test-job-$UNIQUE_NAME-${bundle.target}
31+
description: Before UI edit
32+
job_clusters:
33+
- job_cluster_key: main
34+
new_cluster:
35+
policy_id: ${resources.cluster_policies.policy.id}
36+
spark_version: $DEFAULT_SPARK_VERSION
37+
node_type_id: $NODE_TYPE_ID
38+
num_workers: 1
39+
custom_tags:
40+
owner: bundle-author
41+
tasks:
42+
- task_key: main
43+
job_cluster_key: main
44+
spark_python_task:
45+
python_file: ./hello.py
46+
47+
targets:
48+
dev:
49+
default: true
50+
prod:
51+
variables:
52+
environment: prod
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
print("hello")
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Cloud = true
2+
GOOS.windows = false
3+
EnvMatrix.DMS = ["", "true"]
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
2+
=== Deploy with the dev policy
3+
>>> [CLI] bundle deploy -t dev
4+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/dev/files...
5+
Created cluster_policies.policy
6+
Created jobs.policy_sync
7+
Files: 3 uploaded, 0 deleted
8+
Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
9+
10+
=== Sync the UI edit into shared YAML
11+
>>> [CLI] bundle config-remote-sync -t dev --save
12+
Detected changes in 1 resource(s):
13+
14+
Resource: resources.jobs.policy_sync
15+
description: replace
16+
17+
18+
19+
>>> diff.py databricks.yml.backup databricks.yml
20+
--- databricks.yml.backup
21+
+++ databricks.yml
22+
@@ -29,5 +29,5 @@
23+
policy_sync:
24+
name: test-job-[UNIQUE_NAME]-${bundle.target}
25+
- description: Before UI edit
26+
+ description: Edited in the UI
27+
job_clusters:
28+
- job_cluster_key: main
29+
30+
=== Deploy shared YAML with the prod policy
31+
>>> [CLI] bundle deploy -t prod
32+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/prod/files...
33+
Created cluster_policies.policy
34+
Created jobs.policy_sync
35+
Files: 4 uploaded, 0 deleted
36+
Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
37+
38+
=== Add a user tag beside the policy tag in the UI
39+
>>> [CLI] bundle config-remote-sync -t dev --save
40+
Detected changes in 1 resource(s):
41+
42+
Resource: resources.jobs.policy_sync
43+
job_clusters[job_cluster_key='main'].new_cluster.custom_tags['UserTag']: add
44+
45+
46+
47+
>>> diff.py databricks.yml.backup databricks.yml
48+
--- databricks.yml.backup
49+
+++ databricks.yml
50+
@@ -39,4 +39,5 @@
51+
custom_tags:
52+
owner: bundle-author
53+
+ UserTag: edited-in-ui
54+
tasks:
55+
- task_key: main
56+
57+
=== Deploy the user tag with the prod policy
58+
>>> [CLI] bundle deploy -t prod
59+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/prod/files...
60+
Updated jobs.policy_sync
61+
Files: 2 uploaded, 0 deleted
62+
Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
63+
64+
>>> [CLI] jobs get [POLICY_SYNC_ID_2] -o json
65+
{
66+
"UserTag": "edited-in-ui",
67+
"owner": "bundle-author",
68+
"workspace": "prod"
69+
}
70+
71+
>>> [CLI] bundle destroy -t prod --auto-approve
72+
The following resources will be deleted:
73+
delete resources.cluster_policies.policy
74+
delete resources.jobs.policy_sync
75+
76+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/prod
77+
78+
Destroy: 2 deleted
79+
80+
>>> [CLI] bundle destroy -t dev --auto-approve
81+
The following resources will be deleted:
82+
delete resources.cluster_policies.policy
83+
delete resources.jobs.policy_sync
84+
85+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/dev
86+
87+
Destroy: 2 deleted
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
envsubst < databricks.yml.tmpl > databricks.yml
2+
3+
cleanup() {
4+
trace $CLI bundle destroy -t prod --auto-approve
5+
trace $CLI bundle destroy -t dev --auto-approve
6+
}
7+
trap cleanup EXIT
8+
9+
title "Deploy with the dev policy"
10+
trace $CLI bundle deploy -t dev
11+
job_id=$(read_id.py -t dev policy_sync)
12+
13+
edit_resource.py jobs "$job_id" <<'EOF'
14+
r["description"] = "Edited in the UI"
15+
EOF
16+
17+
title "Sync the UI edit into shared YAML"
18+
cp databricks.yml databricks.yml.backup
19+
trace $CLI bundle config-remote-sync -t dev --save
20+
trace diff.py databricks.yml.backup databricks.yml
21+
22+
title "Deploy shared YAML with the prod policy"
23+
trace $CLI bundle deploy -t prod
24+
25+
title "Add a user tag beside the policy tag in the UI"
26+
edit_resource.py jobs "$job_id" <<'EOF'
27+
r["job_clusters"][0]["new_cluster"]["custom_tags"]["UserTag"] = "edited-in-ui"
28+
EOF
29+
30+
cp databricks.yml databricks.yml.backup
31+
trace $CLI bundle config-remote-sync -t dev --save
32+
trace diff.py databricks.yml.backup databricks.yml
33+
34+
title "Deploy the user tag with the prod policy"
35+
trace $CLI bundle deploy -t prod
36+
prod_job_id=$(read_id.py -t prod policy_sync)
37+
trace $CLI jobs get "$prod_job_id" -o json | jq '.settings.job_clusters[0].new_cluster.custom_tags'
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Cloud = true
2+
RecordRequests = false
3+
Ignore = [".databricks", "databricks.yml", "databricks.yml.backup"]

‎acceptance/bundle/config-remote-sync/cluster_policy_remote_addition/output.txt‎

Lines changed: 52 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,21 +10,72 @@ Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
1010
Detected changes in 1 resource(s):
1111

1212
Resource: resources.jobs.test_job
13+
tasks[task_key='test'].new_cluster.custom_tags: add
1314
tasks[task_key='test'].new_cluster.spark_conf: add
1415

1516

1617

1718
>>> diff.py databricks.yml.backup databricks.yml
1819
--- databricks.yml.backup
1920
+++ databricks.yml
20-
@@ -24,4 +24,6 @@
21+
@@ -24,4 +24,8 @@
2122
node_type_id: [NODE_TYPE_ID]
2223
num_workers: 1
24+
+ custom_tags:
25+
+ UserTag: edited-in-ui
2326
+ spark_conf:
2427
+ spark.sql.shuffle.partitions: "7"
2528
spark_python_task:
2629
python_file: ./hello.py
2730

31+
=== Synced config deploys and converges under the policy
32+
>>> [CLI] bundle deploy
33+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
34+
Updated jobs.test_job
35+
Files: 1 uploaded, 0 deleted
36+
Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
37+
38+
>>> [CLI] bundle plan
39+
Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
40+
41+
=== Add a task with user and policy cluster fields in the UI
42+
>>> [CLI] bundle config-remote-sync --save
43+
Detected changes in 1 resource(s):
44+
45+
Resource: resources.jobs.test_job
46+
tasks[task_key='added_in_ui']: add
47+
48+
49+
50+
>>> diff.py databricks.yml.backup databricks.yml
51+
--- databricks.yml.backup
52+
+++ databricks.yml
53+
@@ -30,2 +30,14 @@
54+
spark_python_task:
55+
python_file: ./hello.py
56+
+ - new_cluster:
57+
+ custom_tags:
58+
+ UserTag: new-task
59+
+ node_type_id: [NODE_TYPE_ID]
60+
+ num_workers: 1
61+
+ policy_id: ${resources.cluster_policies.policy.id}
62+
+ spark_conf:
63+
+ spark.sql.shuffle.partitions: "7"
64+
+ spark_version: [DEFAULT_SPARK_VERSION]
65+
+ spark_python_task:
66+
+ python_file: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files/hello.py
67+
+ task_key: added_in_ui
68+
69+
=== Synced config deploys and converges under the policy
70+
>>> [CLI] bundle deploy
71+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
72+
Updated jobs.test_job
73+
Files: 1 uploaded, 0 deleted
74+
Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
75+
76+
>>> [CLI] bundle plan
77+
Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
78+
2879
>>> [CLI] bundle destroy --auto-approve
2980
The following resources will be deleted:
3081
delete resources.cluster_policies.policy

‎acceptance/bundle/config-remote-sync/cluster_policy_remote_addition/script‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
envsubst < databricks.yml.tmpl > databricks.yml
2+
add_repl "$DEFAULT_SPARK_VERSION" DEFAULT_SPARK_VERSION
23

34
cleanup() {
45
trace $CLI bundle destroy --auto-approve
@@ -10,9 +11,33 @@ job_id=$(read_id.py test_job)
1011

1112
edit_resource.py jobs "$job_id" <<'EOF'
1213
r["tasks"][0]["new_cluster"]["spark_conf"] = {"spark.sql.shuffle.partitions": "7"}
14+
r["tasks"][0]["new_cluster"]["custom_tags"]["UserTag"] = "edited-in-ui"
1315
EOF
1416

1517
cp databricks.yml databricks.yml.backup
1618
trace $CLI bundle config-remote-sync --save
1719
trace diff.py databricks.yml.backup databricks.yml
1820
rm databricks.yml.backup
21+
22+
title "Synced config deploys and converges under the policy"
23+
trace $CLI bundle deploy
24+
trace $CLI bundle plan
25+
26+
title "Add a task with user and policy cluster fields in the UI"
27+
edit_resource.py jobs "$job_id" <<'EOF'
28+
from copy import deepcopy
29+
30+
task = deepcopy(r["tasks"][0])
31+
task["task_key"] = "added_in_ui"
32+
task["new_cluster"]["custom_tags"]["UserTag"] = "new-task"
33+
r["tasks"].append(task)
34+
EOF
35+
36+
cp databricks.yml databricks.yml.backup
37+
trace $CLI bundle config-remote-sync --save
38+
trace diff.py databricks.yml.backup databricks.yml
39+
rm databricks.yml.backup
40+
41+
title "Synced config deploys and converges under the policy"
42+
trace $CLI bundle deploy
43+
trace $CLI bundle plan
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
bundle:
2+
name: test-bundle-$UNIQUE_NAME
3+
4+
workspace:
5+
root_path: ~/.bundle/$UNIQUE_NAME
6+
7+
resources:
8+
jobs:
9+
test_job:
10+
name: test-job-$UNIQUE_NAME
11+
tasks:
12+
- task_key: main
13+
new_cluster:
14+
policy_id: $POLICY_ID
15+
spark_version: $DEFAULT_SPARK_VERSION
16+
node_type_id: $NODE_TYPE_ID
17+
num_workers: 1
18+
init_scripts:
19+
- workspace:
20+
destination: /Workspace/init-a.sh
21+
- workspace:
22+
destination: /Workspace/init-b.sh
23+
spark_python_task:
24+
python_file: ./hello.py
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
print("hello")

0 commit comments

Comments
 (0)