Skip to content

Commit 93ee68b

Browse files
pavloKozlovIsaac
andcommitted
Use .workspace in the dbt-sql template
The bundle host comes from .workspace.URL like in the other built-in templates. Deployed dbt jobs get their host from Databricks, so they don't read it. The HTTP path suggested for dbt init gets ?o=<id> when the workspace URL has the ID, because dbt selects the workspace on a host shared by several workspaces with ?o= in the HTTP path. The host suggested for dbt init is .workspace.Hostname. Co-authored-by: Isaac <no-reply@databricks.com>
1 parent 077dc42 commit 93ee68b

9 files changed

Lines changed: 71 additions & 6 deletions

File tree

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
* Built-in templates (except `dbt-sql`) add the workspace ID to `workspace.host` (`?w=<id>`) when the profile or an environment variable sets it, so generated bundles select the right workspace on hosts shared by several workspaces. Welcome messages of all built-in templates show the workspace URL. ([#7350](https://github.com/databricks/cli/pull/7350))
1+
* Built-in templates add the workspace ID to `workspace.host` (`?w=<id>`) when the profile or an environment variable sets it, so generated bundles select the right workspace on hosts shared by several workspaces. Welcome messages show the same URL, and `dbt-sql` also adds `?o=<id>` to the HTTP path it suggests for `dbt init`. ([#7350](https://github.com/databricks/cli/pull/7350))
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"project_name": "my_dbt_sql",
3+
"http_path": "/sql/2.0/warehouses/f00dcafe",
4+
"default_catalog": "main",
5+
"personal_schemas": "yes"
6+
}
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Cloud = false
2+
EnvMatrix.DMS = [""]
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
2+
=== Profile no_workspace_id
3+
4+
>>> grep host: my_dbt_sql/databricks.yml
5+
host: [DATABRICKS_URL]
6+
host: [DATABRICKS_URL]
7+
8+
>>> grep -E ^ default: [^{] my_dbt_sql/profile_template.yml
9+
default: 127.0.0.1
10+
default: /sql/2.0/warehouses/f00dcafe
11+
default: main
12+
default: 4
13+
14+
=== Profile custom_host
15+
16+
>>> grep host: my_dbt_sql/databricks.yml
17+
host: [CUSTOM_HOST]?w=[WORKSPACE_ID]
18+
host: [CUSTOM_HOST]?w=[WORKSPACE_ID]
19+
20+
>>> grep -E ^ default: [^{] my_dbt_sql/profile_template.yml
21+
default: localhost
22+
default: /sql/2.0/warehouses/f00dcafe?o=[WORKSPACE_ID]
23+
default: main
24+
default: 4
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# The test server is reachable as both 127.0.0.1 and localhost. The second
2+
# stands in for a host shared by several workspaces.
3+
CUSTOM_HOST=${DATABRICKS_HOST/127.0.0.1/localhost}
4+
add_repl "$CUSTOM_HOST" CUSTOM_HOST
5+
WORKSPACE_ID=$(curl -s "$DATABRICKS_HOST/.well-known/databricks-config" | jq -r .workspace_id)
6+
7+
cat > .databrickscfg <<EOT
8+
[no_workspace_id]
9+
host = $DATABRICKS_HOST
10+
token = $DATABRICKS_TOKEN
11+
12+
[custom_host]
13+
host = $CUSTOM_HOST
14+
workspace_id = $WORKSPACE_ID
15+
token = $DATABRICKS_TOKEN
16+
EOT
17+
export DATABRICKS_CONFIG_FILE=$PWD/.databrickscfg
18+
unset DATABRICKS_HOST
19+
unset DATABRICKS_TOKEN
20+
21+
# dbt reads the workspace ID from ?o= in the HTTP path. The template adds it
22+
# when the profile sets the ID, like ?w= in the bundle host.
23+
for profile in no_workspace_id custom_host; do
24+
title "Profile $profile\n"
25+
$CLI bundle init dbt-sql --config-file ./input.json --profile $profile > /dev/null 2>&1
26+
trace grep 'host:' my_dbt_sql/databricks.yml
27+
trace grep -E "^ default: [^{]" my_dbt_sql/profile_template.yml
28+
rm -r my_dbt_sql
29+
done
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Ignore = [".databrickscfg"]

‎acceptance/bundle/templates/dbt-sql/output/my_dbt_sql/profile_template.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ fixed:
55
type: databricks
66
prompts:
77
host:
8-
default: [DATABRICKS_HOST]
8+
default: 127.0.0.1
99
token:
1010
hint: 'personal access token to use, dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
1111
hide_input: true

‎libs/template/templates/dbt-sql/template/{{.project_name}}/databricks.yml.tmpl‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@ targets:
2020
mode: development
2121
default: true
2222
workspace:
23-
host: {{workspace_host}}
23+
host: {{.workspace.URL}}
2424
2525
prod:
2626
mode: production
2727
workspace:
28-
host: {{workspace_host}}
28+
host: {{.workspace.URL}}
2929
# We explicitly deploy to /Workspace/Users/{{user_name}} to make sure we only have a single copy.
3030
root_path: /Workspace/Users/{{user_name}}/.bundle/${bundle.name}/${bundle.target}
3131
permissions:

‎libs/template/templates/dbt-sql/template/{{.project_name}}/profile_template.yml.tmpl‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,16 @@ fixed:
55
type: databricks
66
prompts:
77
host:
8-
default: {{(regexp "^https?://").ReplaceAllString workspace_host ""}}
8+
default: {{.workspace.Hostname}}
99
token:
1010
hint: 'personal access token to use, dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
1111
hide_input: true
1212
http_path:
1313
hint: 'HTTP path of SQL warehouse to use'
14-
default: {{.http_path}}
14+
{{- /* dbt selects the workspace on a host shared by several workspaces with ?o=<id> in the
15+
HTTP path. Use the ID from the workspace URL, which has it when the profile sets it. */}}
16+
{{- $id := (regexp `[?&]w=([^&#]+)`).FindStringSubmatch .workspace.URL}}
17+
default: {{.http_path}}{{if $id}}?o={{index $id 1}}{{end}}
1518
{{- if eq .default_catalog ""}}
1619
_choose_unity_catalog:
1720
'use the default workspace catalog (or do not use Unity Catalog)':

0 commit comments

Comments
 (0)