Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/cli/7372.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* Add `databricks sandbox fs` commands (`cp`, `ls`, `cat`, `mkdir`, `rm`) to transfer files between your local filesystem and sandboxes using `<sandbox-id>:/path`. ([#7372](https://github.com/databricks/cli/pull/7372))
4 changes: 4 additions & 0 deletions acceptance/acceptance_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ func setReplsForTestEnvVars(t *testing.T, repls *testdiff.ReplacementsContext) {
"TEST_DEFAULT_WAREHOUSE_ID",
"TEST_DEFAULT_CLUSTER_ID",
"TEST_INSTANCE_POOL_ID",
"TEST_SANDBOX_ID",
}
for _, envVar := range envVars {
if value := os.Getenv(envVar); value != "" {
Expand Down Expand Up @@ -449,6 +450,9 @@ func testAccept(t *testing.T, inprocessMode bool, selectedTests []string, skipTo
if os.Getenv("TEST_INSTANCE_POOL_ID") == "" {
t.Setenv("TEST_INSTANCE_POOL_ID", testserver.TestDefaultInstancePoolId)
}
if os.Getenv("TEST_SANDBOX_ID") == "" {
t.Setenv("TEST_SANDBOX_ID", testserver.TestDefaultSandboxId)
}
}

const sharedProxyHint = "; re-run with -debugsandbox to see which test caused this"
Expand Down
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/cat/hello.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
hello world!
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/cat/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = true
37 changes: 37 additions & 0 deletions acceptance/cmd/sandbox/fs/cat/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@

=== Help

>>> [CLI] sandbox fs cat --help
Show file content. Use <sandbox-id>:/absolute/path for sandbox paths.

Usage:
databricks sandbox fs cat FILE_PATH [flags]

Examples:
databricks sandbox fs cat happy-panda-1234:/tmp/file

Flags:
-h, --help[=true|false] help for cat

Global Flags:
--debug[=true|false] enable debug logging
-o, --output type output type: text or json (default text)
-p, --profile string ~/.databrickscfg profile
-t, --target string bundle target to use (if applicable)

=== cat a file

>>> [CLI] sandbox fs cat [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/hello.txt
hello world!

=== cat a directory, then a missing path

>>> [CLI] sandbox fs cat [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]
Error: not a file: /tmp/sandbox-fs-[UNIQUE_NAME]

Exit code: 1

>>> [CLI] sandbox fs cat [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/missing
Error: file does not exist: /tmp/sandbox-fs-[UNIQUE_NAME]/missing

Exit code: 1
11 changes: 11 additions & 0 deletions acceptance/cmd/sandbox/fs/cat/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
title "Help"
trace $CLI sandbox fs cat --help

$CLI sandbox fs cp hello.txt $SANDBOX_ROOT/hello.txt > /dev/null

title "cat a file"
trace $CLI sandbox fs cat $SANDBOX_ROOT/hello.txt

title "cat a directory, then a missing path"
errcode trace $CLI sandbox fs cat $SANDBOX_ROOT
errcode trace $CLI sandbox fs cat $SANDBOX_ROOT/missing
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/cp/local.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
hello world!
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/cp/localdir/file1.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
file1 content
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/cp/localdir/file2.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
file2 content
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/cp/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = true
83 changes: 83 additions & 0 deletions acceptance/cmd/sandbox/fs/cp/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@

=== Help

>>> [CLI] sandbox fs cp --help
Copy files and directories between your local filesystem and sandboxes.

Use <sandbox-id>:/absolute/path for sandbox paths.

Recursively copying a directory will copy all files inside directory
at SOURCE_PATH to the directory at TARGET_PATH.

When copying a file, if TARGET_PATH is a directory, the file will be created
inside the directory, otherwise the file is created at TARGET_PATH.

Files that already exist at the target are skipped unless --overwrite is set.

Usage:
databricks sandbox fs cp SOURCE_PATH TARGET_PATH [flags]

Examples:
databricks sandbox fs cp ./file happy-panda-1234:/tmp/file
databricks sandbox fs cp --overwrite ./file happy-panda-1234:/tmp/file
databricks sandbox fs cp -r happy-panda-1234:/tmp/dir ./local/dir

Flags:
--concurrency int number of parallel copy operations (default 8)
-h, --help[=true|false] help for cp
--overwrite[=true|false] overwrite existing files
-r, --recursive[=true|false] recursively copy files from directory

Global Flags:
--debug[=true|false] enable debug logging
-o, --output type output type: text or json (default text)
-p, --profile string ~/.databrickscfg profile
-t, --target string bundle target to use (if applicable)

=== Upload a file

>>> [CLI] sandbox fs cp local.txt [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/file.txt
local.txt -> [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/file.txt

>>> [CLI] sandbox fs cat [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/file.txt
hello world!

=== Writing over an existing file needs --overwrite

>>> [CLI] sandbox fs cp local.txt [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/file.txt
local.txt -> [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/file.txt (skipped; already exists)

>>> [CLI] sandbox fs cp --overwrite updated.txt [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/file.txt
updated.txt -> [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/file.txt

>>> [CLI] sandbox fs cat [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/file.txt
hello again!

=== Copying a file to a directory puts it inside

>>> [CLI] sandbox fs cp local.txt [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/target
local.txt -> [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/target/local.txt

>>> [CLI] sandbox fs ls [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/target
local.txt

=== Writing into a directory that does not exist fails

>>> [CLI] sandbox fs cp local.txt [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/missing/file.txt
Error: no such directory: /tmp/sandbox-fs-[UNIQUE_NAME]/missing

Exit code: 1

=== Copy a directory recursively, and back

>>> [CLI] sandbox fs cp -r localdir [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir
localdir/file1.txt -> [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir/file1.txt
localdir/file2.txt -> [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir/file2.txt

>>> [CLI] sandbox fs cp -r [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir downloaded
[TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir/file1.txt -> downloaded/file1.txt
[TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir/file2.txt -> downloaded/file2.txt

>>> cat downloaded/file1.txt downloaded/file2.txt
file1 content
file2 content
25 changes: 25 additions & 0 deletions acceptance/cmd/sandbox/fs/cp/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
title "Help"
trace $CLI sandbox fs cp --help

title "Upload a file"
trace $CLI sandbox fs cp local.txt $SANDBOX_ROOT/file.txt
trace $CLI sandbox fs cat $SANDBOX_ROOT/file.txt

title "Writing over an existing file needs --overwrite"
trace $CLI sandbox fs cp local.txt $SANDBOX_ROOT/file.txt
trace $CLI sandbox fs cp --overwrite updated.txt $SANDBOX_ROOT/file.txt
trace $CLI sandbox fs cat $SANDBOX_ROOT/file.txt

title "Copying a file to a directory puts it inside"
$CLI sandbox fs mkdir $SANDBOX_ROOT/target
trace $CLI sandbox fs cp local.txt $SANDBOX_ROOT/target
trace $CLI sandbox fs ls $SANDBOX_ROOT/target

title "Writing into a directory that does not exist fails"
errcode trace $CLI sandbox fs cp local.txt $SANDBOX_ROOT/missing/file.txt

title "Copy a directory recursively, and back"
trace $CLI sandbox fs cp -r localdir $SANDBOX_ROOT/dir 2>&1 | sort
trace $CLI sandbox fs cp -r $SANDBOX_ROOT/dir downloaded 2>&1 | sort
trace cat downloaded/file1.txt downloaded/file2.txt
rm -r downloaded
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/cp/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Ignore = ["downloaded"]
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/cp/updated.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
hello again!
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/ls/hello.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
hello world!
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/ls/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = true
62 changes: 62 additions & 0 deletions acceptance/cmd/sandbox/fs/ls/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@

=== Help

>>> [CLI] sandbox fs ls --help
Lists files. Use <sandbox-id>:/absolute/path for sandbox paths.

Usage:
databricks sandbox fs ls DIR_PATH [flags]

Examples:
databricks sandbox fs ls happy-panda-1234:/tmp

Flags:
--absolute[=true|false] Displays absolute paths.
-h, --help[=true|false] help for ls
-l, --long[=true|false] Displays full information including size, file type and modification time since Epoch in milliseconds.

Global Flags:
--debug[=true|false] enable debug logging
-o, --output type output type: text or json (default text)
-p, --profile string ~/.databrickscfg profile
-t, --target string bundle target to use (if applicable)

=== List a directory

>>> [CLI] sandbox fs ls [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]
dir
hello.txt

>>> [CLI] sandbox fs ls --absolute [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]
[TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir
[TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/hello.txt

=== List a file, then a missing path

>>> [CLI] sandbox fs ls [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/hello.txt
Error: not a directory: /tmp/sandbox-fs-[UNIQUE_NAME]/hello.txt

Exit code: 1

>>> [CLI] sandbox fs ls [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/missing
Error: no such directory: /tmp/sandbox-fs-[UNIQUE_NAME]/missing

Exit code: 1

=== Complete the entries of a directory, then a partial name

>>> [CLI] __complete sandbox fs ls [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/
[TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir/
:2
Completion ended with directive: ShellCompDirectiveNoSpace

>>> [CLI] __complete sandbox fs ls [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/d
[TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir/
:2
Completion ended with directive: ShellCompDirectiveNoSpace

=== List a directory given with a trailing slash

>>> [CLI] sandbox fs ls [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/
dir
hello.txt
20 changes: 20 additions & 0 deletions acceptance/cmd/sandbox/fs/ls/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
title "Help"
trace $CLI sandbox fs ls --help

$CLI sandbox fs cp hello.txt $SANDBOX_ROOT/hello.txt > /dev/null
$CLI sandbox fs mkdir $SANDBOX_ROOT/dir

title "List a directory"
trace $CLI sandbox fs ls $SANDBOX_ROOT
trace $CLI sandbox fs ls --absolute $SANDBOX_ROOT

title "List a file, then a missing path"
errcode trace $CLI sandbox fs ls $SANDBOX_ROOT/hello.txt
errcode trace $CLI sandbox fs ls $SANDBOX_ROOT/missing

title "Complete the entries of a directory, then a partial name"
trace $CLI __complete sandbox fs ls $SANDBOX_ROOT/
trace $CLI __complete sandbox fs ls $SANDBOX_ROOT/d

title "List a directory given with a trailing slash"
trace $CLI sandbox fs ls $SANDBOX_ROOT/
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/mkdir/hello.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
hello world!
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/mkdir/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = true
41 changes: 41 additions & 0 deletions acceptance/cmd/sandbox/fs/mkdir/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@

=== Help

>>> [CLI] sandbox fs mkdir --help
Make directories. Use <sandbox-id>:/absolute/path for sandbox paths.

Usage:
databricks sandbox fs mkdir DIR_PATH [flags]

Aliases:
mkdir, mkdirs

Examples:
databricks sandbox fs mkdir happy-panda-1234:/tmp/a/b

Flags:
-h, --help[=true|false] help for mkdir

Global Flags:
--debug[=true|false] enable debug logging
-o, --output type output type: text or json (default text)
-p, --profile string ~/.databrickscfg profile
-t, --target string bundle target to use (if applicable)

=== Create nested directories

>>> [CLI] sandbox fs mkdir [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/a/b/c

>>> [CLI] sandbox fs ls [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/a/b
c

=== Creating an existing directory succeeds

>>> [CLI] sandbox fs mkdir [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/a/b/c

=== Creating a directory over a file fails

>>> [CLI] sandbox fs mkdir [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/hello.txt
Error: file already exists: /tmp/sandbox-fs-[UNIQUE_NAME]/hello.txt

Exit code: 1
13 changes: 13 additions & 0 deletions acceptance/cmd/sandbox/fs/mkdir/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
title "Help"
trace $CLI sandbox fs mkdir --help

title "Create nested directories"
trace $CLI sandbox fs mkdir $SANDBOX_ROOT/a/b/c
trace $CLI sandbox fs ls $SANDBOX_ROOT/a/b

title "Creating an existing directory succeeds"
trace $CLI sandbox fs mkdir $SANDBOX_ROOT/a/b/c

title "Creating a directory over a file fails"
$CLI sandbox fs cp hello.txt $SANDBOX_ROOT/hello.txt > /dev/null
errcode trace $CLI sandbox fs mkdir $SANDBOX_ROOT/hello.txt
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/rm/hello.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
hello world!
1 change: 1 addition & 0 deletions acceptance/cmd/sandbox/fs/rm/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = true
51 changes: 51 additions & 0 deletions acceptance/cmd/sandbox/fs/rm/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@

=== Help

>>> [CLI] sandbox fs rm --help
Remove files and directories. Use <sandbox-id>:/absolute/path for sandbox paths.

Usage:
databricks sandbox fs rm PATH [flags]

Examples:
databricks sandbox fs rm -r happy-panda-1234:/tmp/dir

Flags:
-h, --help[=true|false] help for rm
-r, --recursive[=true|false] Recursively delete a non-empty directory.

Global Flags:
--debug[=true|false] enable debug logging
-o, --output type output type: text or json (default text)
-p, --profile string ~/.databrickscfg profile
-t, --target string bundle target to use (if applicable)

=== Remove a file

>>> [CLI] sandbox fs rm [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/hello.txt

>>> [CLI] sandbox fs cat [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/hello.txt
Error: file does not exist: /tmp/sandbox-fs-[UNIQUE_NAME]/hello.txt

Exit code: 1

=== Removing a non-empty directory needs -r

>>> [CLI] sandbox fs rm [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir
Error: directory not empty: /tmp/sandbox-fs-[UNIQUE_NAME]/dir

Exit code: 1

>>> [CLI] sandbox fs rm -r [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir

>>> [CLI] sandbox fs ls [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/dir
Error: no such directory: /tmp/sandbox-fs-[UNIQUE_NAME]/dir

Exit code: 1

=== Removing a missing path fails

>>> [CLI] sandbox fs rm [TEST_SANDBOX_ID]:/tmp/sandbox-fs-[UNIQUE_NAME]/missing
Error: file does not exist: /tmp/sandbox-fs-[UNIQUE_NAME]/missing

Exit code: 1
Loading
Loading