[andreamatt]

Changes

Resource keys ending in an underscore (e.g. my_job_) did not match the reference regex, so bundle plan/deploy on the direct engine failed with cannot parse "/jobs/${resources.jobs.my_job_.id}" when planning permissions. User-written ${resources.jobs.my_job_.id} references were left unresolved.

This is the trailing counterpart of #6422 (leading underscores) and #2203 (double underscores).

Fix: append _* to BaseVarDef in Go and Python, kept in sync through the shared reference_vectors.json parity test. Only underscores are allowed at the end. The existing negative vectors still hold: trailing hyphens (segment-trailing-hyphen, trailing-hyphen-accent) and a trailing -_ (segment-trailing-underscore) still don't match.

Why

Terraform accepts these keys, so bundles deployed with the Terraform engine can already contain them. They show up naturally when keys are generated from job names: a job named [nightly] report (v2) sanitized to [a-zA-Z0-9_-] becomes _nightly__report__v2_.

On such bundles:

• v1.19.x: the automatic migration fails its plan check and silently falls back to Terraform (migration to direct: cannot plan …permissions: cannot parse …, logged only as a warning).
• v1.20.0+: the Terraform engine is removed, so the same failure is fatal (Auto-migration of your bundle failed. Install Databricks CLI v1.19.x …).

The only workaround today is renaming the key. That deletes and recreates the job, which changes its ID and loses its run history.

Backwards compatibility and previous iterations

The "must end with a letter or digit" rule was a readability choice, not a technical requirement. A segment always ends at ., [ or }, so a trailing _ is unambiguous.

• Before Update variable regex to support hyphens #503 (2023), a segment was \w+, so my_job_ was accepted. Update variable regex to support hyphens #503 added hyphen support and narrowed segments to "start with a letter, end with a letter or digit". Its description notes this "does break backwards compatibility", and the review says the reason was keeping names human readable.
• Added support for double underscore variable references #2203 (2025) lifted the double-underscore part of that rule: "Previously we made this restriction stronger with no particular reason, TF provider supports multiple underscores and thus DABs should do as well."
• dynvar: fix references to resource keys with leading underscores #6422 (2026) lifted the leading-underscore part after the same cannot parse failure on the direct engine.

This PR lifts the trailing-underscore part for the same reasons:

• The Terraform engine never checked resource keys against this regex, so existing bundles contain them.
• The direct engine now generates ${resources.<type>.<key>.id} references for permissions and grants itself, so it rejects keys the rest of the CLI accepts.

The change only widens what matches, so every reference that resolved before still resolves the same way. The one behaviour change: a string like ${foo_} that used to be left as literal text is now treated as a reference, like ${_foo} after #6422. $${foo_} still escapes it. Trailing hyphens stay rejected, as the existing vectors require.

Tests

• New vectors in libs/dyn/dynvar/testdata/reference_vectors.json: trailing-underscore, resource-key-trailing-underscore, resource-key-leading-and-trailing-underscores. go test ./libs/dyn/... and the Python parity test (databricks_tests/core) pass.
• New acceptance test acceptance/bundle/resource_deps/trailing_underscore_key_reference, modelled on underscore_key_reference. On main it fails with the cannot parse error; with this change it deploys job_ with permissions, _job__, and a job referencing both.
• bundle/schema/jsonschema.json regenerated (task generate-schema).
• go test ./libs/dyn/... ./bundle/config/... ./bundle/schema/... ./bundle/direct/... pass.
• Checked on a real bundle: bundle plan -t dev on a bundle with 3 such keys fails on main and succeeds with this change.

[github-actions]

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 6965
• Commit SHA: a56a1b01107e8c60ac6c8b4bda3cb4545ff8dfb3

Checks will be approved automatically on success.

[andrewnester]

Nit: in reference_vectors.json, the pre-existing negative vector segment-trailing-underscore (${hello_world-_.world_world}) now sits next to the new positive trailing-underscore and reads as a contradiction; rename it to segment-trailing-hyphen-underscore?

[andrewnester]

libs/structs/structpath/path.go has a third copy of this grammar (baseVarDef, added in #6980) that still lacks the trailing _*: structpath.PureReferenceToPath("${resources.jobs.job_.id}") returns false while dynvar.IsPureVariableReference returns true. TestPureReferenceMatchesDynvar doesn't catch it because none of its inputs ends a segment with _. Can you update that copy, add a ${var.foo_} case to that test, and list structpath in the keep-in-sync comment above?

[andrewnester]

With requests not recorded, the output only shows that the deploy didn't fail. If _job__ stopped matching, jobs.ref would still be created, with the literal ${resources.jobs._job__.id} in its name, and this test would pass. Can you record requests and print the jobs/create payloads (as escaped_ref does, with --nostamp so both DMS variants agree) so the resolved name is in output.txt? Or is Created jobs.job_.permissions considered enough here, as in underscore_key_reference?