[WIP] Add the testing property of pipelines to bundles

.nextchanges/bundles/editable-project-folder.md

@@ -0,0 +1 @@
+* Fix `-e ./folder` in environment dependencies, which failed before deploy. A local project folder after `-e`, with optional extras such as `-e ./project[test]`, now maps to its deployed folder. ([#7300](https://github.com/databricks/cli/pull/7300))

.nextchanges/bundles/pipelines-testing-plan.md

@@ -0,0 +1 @@
+* `bundle plan` now shows a change to a pipeline's `testing` that was made outside the bundle, and `bundle deploy` undoes it. ([#7300](https://github.com/databricks/cli/pull/7300))

.nextchanges/bundles/pipelines-testing.md

@@ -0,0 +1 @@
+* Add the `testing` property of pipelines, which sets the test sources and test-only dependencies that `databricks pipelines test` runs with. ([#7300](https://github.com/databricks/cli/pull/7300))

acceptance/bundle/environments/editable-folder/databricks.yml

@@ -0,0 +1,33 @@
+bundle:
+  name: editable-folder
+
+resources:
+  jobs:
+    my_job:
+      name: my job
+      tasks:
+        - task_key: main
+          python_wheel_task:
+            package_name: orders
+            entry_point: main
+          environment_key: default
+      environments:
+        - environment_key: default
+          spec:
+            client: "1"
+            dependencies:
+              - -e ./project
+
+  pipelines:
+    my_pipeline:
+      name: my pipeline
+      environment:
+        dependencies:
+          - -e ./project
+      testing:
+        sources:
+          - glob:
+              include: /Workspace/Shared/orders/tests/**
+        environment:
+          dependencies:
+            - -e ./project[test]

acceptance/bundle/environments/editable-folder/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/environments/editable-folder/output.txt

@@ -0,0 +1,13 @@
+
+>>> [CLI] bundle validate -o json
+{
+  "job": [
+    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project"
+  ],
+  "pipeline": [
+    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project"
+  ],
+  "testing": [
+    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project[test]"
+  ]
+}

acceptance/bundle/environments/editable-folder/project/pyproject.toml

@@ -0,0 +1,6 @@
+[project]
+name = "orders"
+version = "0.1.0"
+
+[project.optional-dependencies]
+test = ["faker"]

acceptance/bundle/environments/editable-folder/script

@@ -0,0 +1 @@
+trace $CLI bundle validate -o json | jq '{job: .resources.jobs.my_job.environments[0].spec.dependencies, pipeline: .resources.pipelines.my_pipeline.environment.dependencies, testing: .resources.pipelines.my_pipeline.testing.environment.dependencies}'

acceptance/bundle/environments/editable-folder/test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+Env.DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING = "true"

acceptance/bundle/refschema/output/pipelines.txt

@@ -867,6 +867,14 @@ resources.pipelines.*.storage	string	ALL
 resources.pipelines.*.tags	map[string]string	ALL
 resources.pipelines.*.tags.*	string	ALL
 resources.pipelines.*.target	string	ALL
+resources.pipelines.*.testing	*resources.PipelineTesting	ALL
+resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	ALL
+resources.pipelines.*.testing.environment.dependencies	[]string	ALL
+resources.pipelines.*.testing.environment.dependencies[*]	string	ALL
+resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	ALL
+resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	ALL
+resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	ALL
+resources.pipelines.*.testing.sources[*].glob.include	string	ALL
 resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
 resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
 resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL

acceptance/bundle/resources/pipelines/testing-dependencies/databricks.yml

@@ -0,0 +1,23 @@
+bundle:
+  name: pipeline-testing-dependencies
+
+resources:
+  pipelines:
+    orders:
+      name: orders
+      testing:
+        sources:
+          - glob:
+              include: /Workspace/Shared/orders/tests/**
+        environment:
+          dependencies:
+            # Accepted, as by the API.
+            - --editable /Workspace/Shared/orders[test]
+            - --requirement=/Workspace/Shared/orders/requirements.txt
+            - -r ./orders/requirements.txt
+            # Rejected, as by the API.
+            - faker==33.0
+            - git+https://github.com/example/helpers
+            - /Workspace/Shared/orders/requirements.txt
+            - --requirement ./orders/requirements.txt
+            - /Workspace/Shared/orders -i https://example.com/simple

acceptance/bundle/resources/pipelines/testing-dependencies/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/resources/pipelines/testing-dependencies/output.txt

@@ -0,0 +1,39 @@
+
+>>> musterr [CLI] bundle validate
+Error: testing.environment.dependencies must point at files, not "faker==33.0"
+  at resources.pipelines.orders.testing.environment.dependencies[3]
+  in databricks.yml:19:15
+
+A test dependency is a project folder, a wheel, or a requirements file after -r, in the bundle, under /Workspace/, or under /Volumes/. Declare test packages in a project file, such as the test extra of a pyproject.toml, and reference its folder, for example -e ./project[test]. For a path in the bundle, use -e or -r followed by a space. URLs, other pip options, and .. are not supported.
+
+Error: testing.environment.dependencies must point at files, not "git+https://github.com/example/helpers"
+  at resources.pipelines.orders.testing.environment.dependencies[4]
+  in databricks.yml:20:15
+
+A test dependency is a project folder, a wheel, or a requirements file after -r, in the bundle, under /Workspace/, or under /Volumes/. Declare test packages in a project file, such as the test extra of a pyproject.toml, and reference its folder, for example -e ./project[test]. For a path in the bundle, use -e or -r followed by a space. URLs, other pip options, and .. are not supported.
+
+Error: testing.environment.dependencies needs -r before the requirements file "/Workspace/Shared/orders/requirements.txt"
+  at resources.pipelines.orders.testing.environment.dependencies[5]
+  in databricks.yml:21:15
+
+Without -r, pip treats the file as a package to install. For example: -r ./tests/requirements.txt.
+
+Error: testing.environment.dependencies must point at files, not "--requirement ./orders/requirements.txt"
+  at resources.pipelines.orders.testing.environment.dependencies[6]
+  in databricks.yml:22:15
+
+A test dependency is a project folder, a wheel, or a requirements file after -r, in the bundle, under /Workspace/, or under /Volumes/. Declare test packages in a project file, such as the test extra of a pyproject.toml, and reference its folder, for example -e ./project[test]. For a path in the bundle, use -e or -r followed by a space. URLs, other pip options, and .. are not supported.
+
+Error: testing.environment.dependencies must point at files, not "/Workspace/Shared/orders -i https://example.com/simple"
+  at resources.pipelines.orders.testing.environment.dependencies[7]
+  in databricks.yml:23:15
+
+A test dependency is a project folder, a wheel, or a requirements file after -r, in the bundle, under /Workspace/, or under /Volumes/. Declare test packages in a project file, such as the test extra of a pyproject.toml, and reference its folder, for example -e ./project[test]. For a path in the bundle, use -e or -r followed by a space. URLs, other pip options, and .. are not supported.
+
+Name: pipeline-testing-dependencies
+Target: default
+Workspace:
+  User: [USERNAME]
+  Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-dependencies/default
+
+Found 5 errors

acceptance/bundle/resources/pipelines/testing-dependencies/script

@@ -0,0 +1 @@
+trace musterr $CLI bundle validate

acceptance/bundle/resources/pipelines/testing-dependencies/test.toml

@@ -0,0 +1,3 @@
+# Runs only against the local test server.
+Cloud = false
+Env.DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING = "true"

acceptance/bundle/resources/pipelines/testing-disabled/databricks.yml

@@ -0,0 +1,11 @@
+bundle:
+  name: pipeline-testing-disabled
+
+resources:
+  pipelines:
+    orders:
+      name: orders
+      testing:
+        sources:
+          - glob:
+              include: ./tests/**

acceptance/bundle/resources/pipelines/testing-disabled/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/resources/pipelines/testing-disabled/output.txt

@@ -0,0 +1,10 @@
+
+>>> musterr [CLI] bundle validate
+Error: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it
+  at resources.pipelines.orders.testing
+  in databricks.yml:9:9
+
+Name: pipeline-testing-disabled
+Target: default
+
+Found 1 error

acceptance/bundle/resources/pipelines/testing-disabled/script

@@ -0,0 +1 @@
+trace musterr $CLI bundle validate

acceptance/bundle/resources/pipelines/testing-disabled/test.toml

@@ -0,0 +1,2 @@
+# Runs only against the local test server. The experimental gate is off.
+Cloud = false

acceptance/bundle/resources/pipelines/testing-disabled/tests/test_orders.py

@@ -0,0 +1,2 @@
+def test_orders():
+    pass

acceptance/bundle/resources/pipelines/testing-same-name-libraries/databricks.yml

@@ -0,0 +1,21 @@
+bundle:
+  name: pipeline-testing-same-name-libraries
+
+resources:
+  pipelines:
+    orders:
+      name: orders
+      libraries:
+        - file:
+            path: ./transformations/orders.py
+      environment:
+        dependencies:
+          - ./pipeline/test_helpers-0.1-py3-none-any.whl
+      testing:
+        sources:
+          - glob:
+              include: ./tests/**
+        environment:
+          dependencies:
+            # A different wheel with the same file name would be uploaded to the same path.
+            - ./tests/test_helpers-0.1-py3-none-any.whl

acceptance/bundle/resources/pipelines/testing-same-name-libraries/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/resources/pipelines/testing-same-name-libraries/output.txt

@@ -0,0 +1,12 @@
+
+>>> errcode [CLI] bundle deploy
+Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
+  at resources.pipelines.orders.environment.dependencies[0]
+     resources.pipelines.orders.testing.environment.dependencies[0]
+  in databricks.yml:13:13
+     databricks.yml:21:15
+
+Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl
+
+
+Exit code: 1

acceptance/bundle/resources/pipelines/testing-same-name-libraries/script

@@ -0,0 +1 @@
+trace errcode $CLI bundle deploy

acceptance/bundle/resources/pipelines/testing-same-name-libraries/test.toml

@@ -0,0 +1,7 @@
+# Runs only against the local test server.
+Cloud = false
+Env.DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING = "true"
+
+[[Repls]]
+Old = '\\'
+New = '/'

acceptance/bundle/resources/pipelines/testing-same-name-libraries/tests/test_orders.py

@@ -0,0 +1,2 @@
+def test_orders():
+    pass

acceptance/bundle/resources/pipelines/testing-same-name-libraries/transformations/orders.py

@@ -0,0 +1 @@
+x = 1

acceptance/bundle/resources/pipelines/testing/databricks.yml

@@ -0,0 +1,19 @@
+bundle:
+  name: pipeline-testing
+
+resources:
+  pipelines:
+    orders:
+      name: orders
+      libraries:
+        - glob:
+            include: ./transformations/**
+      testing:
+        sources:
+          - glob:
+              include: ./tests/**
+        environment:
+          dependencies:
+            - --editable ${workspace.file_path}[test]
+            - -r ./tests/requirements.txt
+            - ./dist/*.whl

acceptance/bundle/resources/pipelines/testing/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]