Repository navigation
bundle generate: add subcommands for every deployment-engine resource type - #7365
Draft
janniklasrose wants to merge 7 commits into
Draft
janniklasrose wants to merge 7 commits into
janniklasrose wants to merge 7 commits into
Conversation
ConvertToMapValue sorts keys alphabetically, but without an Order the entries were stored with zero-valued locations and the encoder, which orders by Location.Line, fell back to Go map iteration order. Number the entries so the sorted order survives. Existing callers all pass an Order and are unaffected. Co-authored-by: Isaac <no-reply@databricks.com>
Adds a table-driven generate subcommand for resource types that are read with one API call and have no files to download. A resource[R, C] entry names the lookup flag and a typed fetch; the command narrows the response to the bundle resource type via a JSON round-trip, drops CLI-managed and skipped fields, derives the key from the resource name, writes <config-dir>/<key>.<type>.yml under the bundle root, warns if the file is not included, and supports the hidden --bind flag like the existing generators. The table starts empty; resource families are added in follow-up PRs. A parity test in cmd/bundle fails when a group from config.SupportedResources is neither generated nor listed as unsupported, so new resource types get an explicit decision. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 66f1069
Top 5 slowest tests (at least 2 minutes):
|
…ne adapters Replaces the hand-written spec table with one subcommand per resource type in dresources.SupportedResources. Each command reads the resource through its adapter (DoRead), converts it with RemapState to the create-request shape the bundle config embeds, and maps that onto the bundle resource type via JSON. The mapping is fed back through PrepareState and must reproduce the state; otherwise the command fails with "not supported yet" rather than writing YAML that would not deploy (secret scopes today). The adapters already handle the awkward cases, such as unwrapping the experiment response and converting serving endpoint Output to Input config, so generate inherits them and stays drift-free with deploy by construction. Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
changed the title
Contributor
|
Commit: Regression test reportSelected 5 changed tests (limit=20, 0 not selected) 10 tests: regression=10 unreleased=0 coverage=0 failing=0
TestAccept/bundle/generate/catalog/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/generate/catalog/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-dashboard/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-dashboard/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): TestAccept/bundle/help/bundle-generate-job/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-job/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-pipeline/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-pipeline/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): |
…urce types generate.NewCommands now owns the list: every type in dresources.SupportedResources gets the generic command unless a hand-written generator is registered for it in the generators map, which exists for the six types that also download files or have a watch mode. Folding those into the generic path means deleting their map entry. Co-authored-by: Isaac <no-reply@databricks.com>
…every command
The hand-written job, pipeline, and app commands keep their own lookup flag but
also parse --existing-id, via a pflag normalize alias, so every generate command
takes the same identifier. Each command additionally answers to its resource
group ("jobs", "catalogs"), so a caller holding a key from deployment state can
address it without a plural-to-singular mapping.
Co-authored-by: Isaac <no-reply@databricks.com>
…generate-machinery
…main Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
databricks bundle generategains a subcommand for every bundle resource type.generate.NewCommands()derives the list fromdresources.SupportedResources: each type gets the generic command unless a hand-written generator is registered for it in thegeneratorsmap. That map holds the six existing commands (job, pipeline, dashboard, alert, app, genie-space), which also download files or have a watch mode. Folding them into the generic path is follow-up work; each one removed deletes a map entry. Job runs and the internal snapshot type are the only types without a command.The generic command has no per-resource code. It runs the type's direct-engine adapter:
Adapter.DoRead(id)fetches the remote object.--existing-idtakes the same identifierbundle deployment binduses for the type.Adapter.RemapStateconverts it to the engine's state type, which is the create-request shape the bundle config embeds. The adapters already handle the awkward cases (serving endpoint Output to Input config, unwrappingGetExperimentResponse), so generate inherits them.config.Resources) via a JSON round-trip and fed back throughAdapter.PrepareState. If that does not reproduce the state, the type's state is shaped differently from its configuration and the command fails with "not supported yet" instead of writing YAML that would not deploy. Secret scopes hit this today (scopevsname); the fix belongs in the adapter.GetName()of the bundle resource;--keyoverrides it. The file is written to<config-dir>/<key>.<type>.ymlunder the bundle root, with the usual include warning and hidden--bind.Because the YAML is derived from the engine's own remote-to-state mapping, a generate followed by a deploy is drift-free by construction.
Other changes:
libs/dyn/yamlsaver:ConvertToMapValuewithout an explicitOrdernow numbers entries so the alphabetical sort survives encoding. Previously the encoder fell back to Go map iteration order. The three existing callers (job and pipeline converters, schema annotations writer) all pass anOrderand take the unchanged branch.cmd/bundle/generate_test.go: parity test that fails when a group inconfig.SupportedResources()has neither a generate subcommand nor an entry ingenerateUnsupported.Not in this PR: dropping fields the engine classifies as
output_onlyfrom the generated YAML. They round-trip through deploy without drift, so this is cosmetic until the per-type acceptance tests show how noisy they are.Why
bundle generatecovered six resource types, each a hand-written command. The direct engine already knows how to read every resource type and how its remote shape maps onto bundle configuration, so a generic command on top of the adapters gets the remaining types at once and stays correct as adapters evolve.Tests
cmd/bundle/generate/generic_test.goruns the generic path against the mocked workspace client for catalogs (happy path,--key,--config-dir, overwrite protection, read errors), experiments (wrapper response unwrapped by the adapter), and secret scopes (round-trip check rejects the type), and checks the assembled command list.acceptance/bundle/generate/catalogcreates a catalog on the fake server, generates it, and validates the bundle.bundle/generate/resource_test.goandlibs/dyn/yamlsaver/utils_test.gocover serialization order.bundle generateregenerated.Stacked follow-ups: an acceptance test per resource type, adapter fixes for types that fail the round-trip check, and folding the hand-written generators into the generic path.
This pull request and its description were written by Isaac.