Skip to content

Commit 8186b25

Browse files
denikIsaac
andcommitted
bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations
Bundle configuration is the typed struct plus structvar (pure ${...} references in non-string fields and a tree of source locations). YAML and JSON decode straight into the structs; mutators, validators, variable resolution, configsync and generate work on structvar views and structpath paths. libs/dyn is deleted. Co-authored-by: Isaac <no-reply@databricks.com>
1 parent 3afe070 commit 8186b25

346 files changed

Lines changed: 8538 additions & 20220 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/rules/style-guide-go.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ return fieldPaths
113113

114114
### Encoding
115115

116-
**RULE: When mutating an API response by round-tripping JSON, decode with `json.Decoder` + `UseNumber()`.** The naive `json.Marshal` → `map[string]any` → `Marshal` path corrupts any int64 larger than 2^53 (it degrades to a float64 mantissa — e.g. a real `spark_context_id`) and alphabetizes object keys. `libs/dyn/jsonloader` preserves key order but also lacks `UseNumber`, so it shares the int64 hazard.
116+
**RULE: When mutating an API response by round-tripping JSON, decode with `json.Decoder` + `UseNumber()`.** The naive `json.Marshal` → `map[string]any` → `Marshal` path corrupts any int64 larger than 2^53 (it degrades to a float64 mantissa — e.g. a real `spark_context_id`) and alphabetizes object keys.
117117

118118
**RULE: Be careful with `encoding/csv` `Writer.UseCRLF = true`.** It rewrites both record terminators AND embedded newlines inside quoted fields to `\r\n`, so tests for quoted multiline fields must expect `\r\n`, not just the line endings between rows.
119119

@@ -191,7 +191,7 @@ func NewService(deps ServiceDeps) *Service { ... }
191191

192192
### Configuration patterns
193193

194-
- Bundle config uses `dyn.Value` for dynamic typing
194+
- Bundle config is typed structs; `libs/structs/structvar` carries `${...}` references and source locations
195195
- Config loading supports includes, variable interpolation, and target overrides
196196
- Schema generation is automated from Go struct tags
197197

‎.github/workflows/python_push.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ jobs:
3939
acceptance/bundle/python/ \
4040
go.mod \
4141
go.sum \
42-
libs/dyn/dynvar/testdata/reference_vectors.json \
42+
libs/structs/structvar/testdata/reference_vectors.json \
4343
python/ \
4444
tools/task/ \
4545
)

‎.nextchanges/bundles/6954.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* Speed up bundle commands on large bundles: configuration is kept as typed structs instead of being converted to and from a dynamic tree (`bundle validate` and `bundle plan` with 1000 jobs are about 10x faster and use about 3.5x less memory). ([#6954](https://github.com/databricks/cli/pull/6954))

‎AGENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ This is the Databricks CLI, a command-line interface for interacting with Databr
1414

1515
**RULE: Keep each PR focused on one change.** If you notice an unrelated cleanup, bug fix, or refactor while making your primary change, leave it alone or put it in a separate PR. Reviewers consistently ask to split mixed PRs, especially when a dependency bump or schema diff rides along with a feature change.
1616

17-
**RULE: Before adding a new helper, search the codebase for an existing one.** Common homes: `libs/` (shared utilities), `libs/databrickscfg/` (config), `libs/git/`, `libs/filer/`, `libs/cmdio/` (CLI I/O, spinners, prompts), `libs/env/` (env vars), `libs/testserver/`, `libs/structpath/` and `libs/dyn/` (path / dynamic values), `acceptance/bin/` (acceptance test helpers), `internal/mocks/` (generated mocks). A function that duplicates an existing name and signature in the same package is a compile error waiting to happen; grep before you name.
17+
**RULE: Before adding a new helper, search the codebase for an existing one.** Common homes: `libs/` (shared utilities), `libs/databrickscfg/` (config), `libs/git/`, `libs/filer/`, `libs/cmdio/` (CLI I/O, spinners, prompts), `libs/env/` (env vars), `libs/testserver/`, `libs/structs/structpath/` and `libs/structs/structvar/` (paths / typed config values), `acceptance/bin/` (acceptance test helpers), `internal/mocks/` (generated mocks). A function that duplicates an existing name and signature in the same package is a compile error waiting to happen; grep before you name.
1818

1919
# Development Commands
2020

@@ -73,7 +73,7 @@ GIT_EDITOR=true GIT_SEQUENCE_EDITOR=true VISUAL=true GIT_PAGER=cat git rebase or
7373
- `bundle/phases/` - High-level deployment phases
7474

7575
**libs/** - Shared libraries and utilities
76-
- `libs/dyn/` - Dynamic configuration value manipulation
76+
- `libs/structs/structvar/` - Typed configuration values with references and locations
7777
- `libs/filer/` - File system abstraction (local, DBFS, workspace)
7878
- `libs/auth/` - Databricks authentication handling
7979
- `libs/sync/` - File synchronization between local and remote
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
bundle:
2+
name: test-bundle
3+
4+
resources:
5+
jobs:
6+
bar:
7+
name: job bar
8+
foo:
9+
name: job foo
10+
tasks:
11+
- task_key: job_task
12+
run_job_task:
13+
job_id: ${resources.jobs.bar.id}
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Cloud = false
2+
EnvMatrix.DMS = ["", "true"]
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
2+
>>> [CLI] bundle deploy
3+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
4+
Created jobs.bar
5+
Created jobs.foo
6+
Files: 1 uploaded, 0 deleted
7+
Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
8+
9+
=== run resolves the referenced job id into the int job_id field
10+
>>> [CLI] bundle run foo
11+
Run URL: [DATABRICKS_URL]/jobs/[NUMID]/runs/[NUMID]?o=[NUMID]
12+
13+
[TIMESTAMP] "job foo" RUNNING
14+
[TIMESTAMP] "job foo" TERMINATED SUCCESS
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
trace $CLI bundle deploy
2+
3+
title "run resolves the referenced job id into the int job_id field"
4+
trace $CLI bundle run foo

‎acceptance/invariant_test.go‎

Lines changed: 11 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,9 @@ import (
88
"testing"
99

1010
"github.com/databricks/cli/bundle/config"
11-
"github.com/databricks/cli/libs/dyn"
12-
"github.com/databricks/cli/libs/dyn/yamlloader"
1311
"github.com/stretchr/testify/assert"
1412
"github.com/stretchr/testify/require"
13+
"go.yaml.in/yaml/v3"
1514
)
1615

1716
const invariantConfigsDir = "bundle/invariant/configs"
@@ -88,31 +87,22 @@ func scanInvariantConfigs(t *testing.T) (present, withPermissions, withGrants ma
8887
contents, err := os.ReadFile(path)
8988
require.NoError(t, err)
9089

91-
v, err := yamlloader.LoadYAML(path, strings.NewReader(string(contents)))
92-
require.NoError(t, err, "failed to parse %s", path)
93-
94-
resources := v.Get("resources")
95-
if resources.Kind() != dyn.KindMap {
96-
// Some configs (e.g. PyDABs) declare resources outside of YAML.
97-
continue
90+
var doc struct {
91+
Resources map[string]any `yaml:"resources"`
9892
}
93+
require.NoError(t, yaml.Unmarshal(contents, &doc), "failed to parse %s", path)
9994

100-
for _, group := range resources.MustMap().Pairs() {
101-
groupName := group.Key.MustString()
95+
// Some configs (e.g. PyDABs) declare resources outside of YAML.
96+
for groupName, group := range doc.Resources {
10297
present[groupName] = true
10398

104-
if group.Value.Kind() != dyn.KindMap {
105-
continue
106-
}
107-
for _, resource := range group.Value.MustMap().Pairs() {
108-
cfg := resource.Value
109-
if cfg.Kind() != dyn.KindMap {
110-
continue
111-
}
112-
if cfg.Get("permissions").Kind() != dyn.KindInvalid {
99+
resources, _ := group.(map[string]any)
100+
for _, resource := range resources {
101+
cfg, _ := resource.(map[string]any)
102+
if _, ok := cfg["permissions"]; ok {
113103
withPermissions[groupName] = true
114104
}
115-
if cfg.Get("grants").Kind() != dyn.KindInvalid {
105+
if _, ok := cfg["grants"]; ok {
116106
withGrants[groupName] = true
117107
}
118108
}

‎bundle/apps/validate.go‎

Lines changed: 19 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@ import (
88
"github.com/databricks/cli/bundle"
99
"github.com/databricks/cli/bundle/config/resources"
1010
"github.com/databricks/cli/libs/diag"
11-
"github.com/databricks/cli/libs/dyn"
1211
"github.com/databricks/cli/libs/structs/structpath"
12+
"github.com/databricks/cli/libs/structs/structvar"
1313
"github.com/databricks/databricks-sdk-go/service/apps"
1414
)
1515

@@ -94,32 +94,26 @@ type appResourceReference struct {
9494

9595
// hasPermissions checks if a bundle resource at the given dyn path has a non-empty permissions list.
9696
func hasPermissions(b *bundle.Bundle, resourcePath string) bool {
97-
pv, err := dyn.Get(b.Config.Value(), resourcePath+".permissions")
97+
pv, err := valueAt(b, resourcePath+".permissions")
9898
if err != nil {
9999
return false
100100
}
101-
s, ok := pv.AsSequence()
102-
return ok && len(s) > 0
101+
for range pv.Sequence() {
102+
return true
103+
}
104+
return false
103105
}
104106

105107
// hasAppSPInPermissions checks if any permission entry for the given resource
106108
// references the app's service principal via variable interpolation.
107109
func hasAppSPInPermissions(b *bundle.Bundle, resourcePath, appKey string) bool {
108110
appSPRef := fmt.Sprintf("${resources.apps.%s.service_principal_client_id}", appKey)
109-
pv, err := dyn.Get(b.Config.Value(), resourcePath+".permissions")
111+
pv, err := valueAt(b, resourcePath+".permissions")
110112
if err != nil {
111113
return false
112114
}
113-
s, ok := pv.AsSequence()
114-
if !ok {
115-
return false
116-
}
117-
for _, entry := range s {
118-
spn, err := dyn.Get(entry, "service_principal_name")
119-
if err != nil {
120-
continue
121-
}
122-
if str, ok := spn.AsString(); ok && str == appSPRef {
115+
for _, entry := range pv.Sequence() {
116+
if str, ok := entry.Get("service_principal_name").AsString(); ok && str == appSPRef {
123117
return true
124118
}
125119
}
@@ -173,7 +167,7 @@ func warnForAppResourcePermissions(b *bundle.Bundle, appKey string, app *resourc
173167
ref.permission,
174168
appKey,
175169
),
176-
Paths: structpath.NewStringKeysSlice("resources", "apps", appKey),
170+
Paths: []*structpath.PathNode{structpath.NewStringKeys(nil, "resources", "apps", appKey)},
177171
Locations: b.Config.GetLocations(appPath),
178172
})
179173
}
@@ -188,3 +182,12 @@ func (v *validate) Name() string {
188182
func Validate() bundle.Mutator {
189183
return &validate{}
190184
}
185+
186+
// valueAt returns the view of the configuration value at path.
187+
func valueAt(b *bundle.Bundle, path string) (structvar.View, error) {
188+
p, err := structpath.ParsePath(path)
189+
if err != nil {
190+
return structvar.View{}, err
191+
}
192+
return b.Config.View().Lookup(p), nil
193+
}

0 commit comments

Comments
 (0)