Repository navigation
[WIP] Add the testing property of pipelines to bundles - #7274
Draft
tharunbhuk-db wants to merge 40 commits into
Draft
tharunbhuk-db wants to merge 40 commits into
tharunbhuk-db wants to merge 40 commits into
Conversation
Submits a test-only pipeline update with optional test selections and pytest arguments. Relative selections resolve to deployed workspace paths through the bundle sync root, or against the pipeline's root_path when the pipeline is identified with --pipeline-id. Only -k and -m are accepted in --pytest-arg. Co-authored-by: Isaac <no-reply@databricks.com>
Cover each rejected pytest option with a unit test, and add an acceptance case for `--pytest-arg VALUE`, which keeps commas and spaces. Co-authored-by: Isaac <no-reply@databricks.com>
The service names a test case relative to pytest's rootdir: the deepest folder that holds the pipeline's root_path and every test source, or, without a root_path, the folder of every test source. A test ID copied from the results now runs that test again in any project layout. A relative selection that exists locally still maps to its deployed path. Any other relative selection resolves against that folder, which the command reads from the deployed pipeline. With --pipeline-id, all relative selections resolve against it. Co-authored-by: Isaac <no-reply@databricks.com>
…nd preview A local test path mapped to workspace.file_path in every sync mode. A source-linked deployment runs the source files, so a local path now maps under the sync root, as TranslatePaths maps resource paths. An immutable deployment runs a snapshot whose path the command does not know, so a local path is an error there that points to test IDs and workspace paths, which still work. The folder of test IDs is read once, and only when a selection is a relative path. Paths in errors use forward slashes. The Short text starts with *Preview*, like other preview commands. Tests: the option list covers each rejected form once, and a folder test source without root_path has its parent as the folder of test IDs, as the service computes it. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Oct 30, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Thanks for the contribution! This pull request was opened from a fork. If you're a member of the Databricks organization, please request access to the If you're an external contributor, no action is needed; a maintainer will review your pull request. |
The test command now polls the update, reads its test events filtered by update_id with pagination, and reads every page once more after the update ends. Case results print as they finish, followed by failure details and the summary, or RUN INCOMPLETE when the events do not form a complete result. Adds --no-wait, --timeout, and --resume; a local timeout or Ctrl+C never stops the update. The test server now simulates test-only updates: it collects tests from deployed files, emits test events, advances the update state per Get Update call, rejects a second update while one is active, and cancels it on Stop. Co-authored-by: Isaac <no-reply@databricks.com>
The command printed to read the results later now includes --profile when one was passed, so it targets the same workspace. The summary and received-results lines say "1 error" instead of "1 errors". Co-authored-by: Isaac <no-reply@databricks.com>
Live polls now filter on `update_id = '<id>' AND timestamp >= '<last seen>'`, copying the last event's timestamp string as the server returned it, instead of an event ID cursor. The run already drops events it has seen, so the boundary events that `>=` returns again are ignored. The request uses the SDK's ListPipelineEventsRequest for its query shape. The SDK iterator itself is not used: PipelineEvent has no details or origin.test_case_id, and the iterator repeats filter and order_by with page_token, which the API does not allow. The test server applies the timestamp condition. Co-authored-by: Isaac <no-reply@databricks.com>
The service names a test case relative to pytest's rootdir. When the event carries an absolute workspace path instead, print it, and name JUnit entries, relative to that same folder, so every printed ID can be passed back as a selection. IDs outside that folder stay absolute. The test server emits absolute test case IDs, so this path is covered. Co-authored-by: Isaac <no-reply@databricks.com>
Add the case where every test passes but the update fails afterwards: the command prints the cases and the summary, then the update error, and exits 1. The test server fails the update after the summary for a test named "breaks_update". Add a run with 200 cases. Its 601 events span several pages, and the final read overlaps the live polls, so the summary only matches if every page is read and duplicates are dropped. Co-authored-by: Isaac <no-reply@databricks.com>
A test update that ends FAILED because tests failed is explained by the printed failures. When the update also reports a fatal error of its own, or is CANCELED, the command now prints the update state and that error as well. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
When tests fail, the service ends the update with an error such as "[PIPELINE_TEST_FAILED] 1 of the pipeline's tests failed (1 passed, 0 skipped).". The command treated it as an error of the update's own and printed it after the failures it repeats. It is now explained by the printed failures, like a failed update without an error. Any other update error is still printed. The test server ends a run whose tests failed with the same error. Co-authored-by: Isaac <no-reply@databricks.com>
The service names a test case relative to pytest's rootdir, such as tests/test_orders.py::test_backfill. The command no longer rewrites absolute IDs, which the service does not send, and prints and reports each ID as it arrives. The test server now sends IDs relative to the rootdir that the service uses, so the acceptance tests run the same path as a real run. Co-authored-by: Isaac <no-reply@databricks.com>
A run passed when a collected case had no final outcome, as long as the summary left that case out. Each case the events name must now have one final outcome, or the run is incomplete. The update error that the command prints is the first one the update reported other than the test verdict. A verdict that came after the update's own error replaced it, so the real error was lost. Fixtures use neutral exception names, and unit cases that the acceptance tests already cover are removed. Co-authored-by: Isaac <no-reply@databricks.com>
The test server took an update that ends on its own as inactive from the start, so it accepted a second update that the service rejects with 409. An update is now active until a Get Update call shows a terminal state, and Stop cancels a finite update too, at once. The acceptance scripts that start several updates in a row stop each one first, and command output they don't compare goes to LOG files. Event types in the test server are named constants. Co-authored-by: Isaac <no-reply@databricks.com>
The command printed to read the results later did not quote its values, so a profile such as "CI West" became two arguments. Each value is now shell-quoted, as other commands' hints are. Acceptance runs that must fail use musterr, so a run that passes by mistake fails the test. The events reader says why it does not reuse fetchAllPipelineEvents. Co-authored-by: Isaac <no-reply@databricks.com>
Each PR adds its own changelog file, so this one no longer edits the entry for the command. Co-authored-by: Isaac <no-reply@databricks.com>
In the test server, an update that was never polled is active only if its first state is not terminal. A plain update completes at once, so a second Start Update no longer gets 409, and Stop leaves it COMPLETED. A stopped update now hides the events planned for states it never reached, so a stop before RUNNING shows no case results or summary. The test of a finite update polls a fixed number of times instead of looping until COMPLETED, and the rootdir rule notes that the CLI uses the same rule, so only a run on a real workspace checks it. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
--junit-xml writes one testcase per test with a final outcome and its duration when known. A run without tests, an incomplete run, an update failure after the tests, or a local timeout adds a failing entry naming the cause. --resume replaces an earlier report; --no-wait rejects the flag because no results are available. Co-authored-by: Isaac <no-reply@databricks.com>
The update that fails after its tests passed, and the one that fails with an error of its own after a test failed, now also write a JUnit report. Each report has the failing entry for the update. Co-authored-by: Isaac <no-reply@databricks.com>
The report has one entry per collected case. A case that was collected but has no final outcome, because the run stopped or local waiting ended, is now an error entry with the message "no result received" instead of being left out. It is not a skipped entry, because many CI tools treat skipped as passing. The failing entry for the update stays. Co-authored-by: Isaac <no-reply@databricks.com>
With --junit-xml, an error before the results could be read, such as a rejected Start Update, returned without writing a report, so an older passing report stayed on disk. Every exit after the flags are checked now writes the report; an error without results gives one failing entry that names it. A report that can't be written no longer hides the run's own error, such as the hint to resume. The resume hint carries --junit-xml, so reading the results again replaces the partial report. A parameter ID that contains "::", such as test_cast[a::int], stays with the test name. The printed report path uses forward slashes. Co-authored-by: Isaac <no-reply@databricks.com>
…errors A run outside a project without --pipeline-id now replaces an older JUnit report too, like every other exit after flag validation. When the project has errors, the command's error is the sentinel for an error that was already printed. The report now names the first printed error instead of the sentinel's text, and when the report cannot be written either, the write error is returned on its own so that it is shown. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Add `resources.pipelines.<key>.testing`, the pipeline's test sources and test-only dependencies. The Go SDK has no type for it yet, so the CLI defines PipelineTesting with the API's field names and tags it in the schema with the API's launch stage. The direct engine sends `testing` with Create and Edit. Only a pipeline that sets it takes the new path: its body is the SDK's JSON for the request plus `testing`. Other pipelines use the SDK calls unchanged. Remove the CLI types and the extra request path when the SDK has the field. Co-authored-by: Isaac <no-reply@databricks.com>
The two patterns for environment.dependencies differ only in their translation mode and skip rule. Build them in a loop over the dependency patterns, so that another list of dependencies can reuse them. No behavior change. Co-authored-by: Isaac <no-reply@databricks.com>
Treat `testing.sources[].glob.include` like `libraries[].glob.include`, and `testing.environment.dependencies` like `environment.dependencies`: relative paths are translated to their deployed workspace paths, globs are expanded, and local wheels are uploaded. Without this, relative test paths reached the API unresolved. Co-authored-by: Isaac <no-reply@databricks.com>
Test dependencies must point at files: a project folder, a requirements file, or a wheel, in the workspace, a volume, or the bundle. Package names such as `faker==33.0` and URLs fail on deploy. Report them during validation instead, with their location and how to declare the packages in a project file. Co-authored-by: Isaac <no-reply@databricks.com>
The fake now records `testing.sources` from Create and Edit, like the service: a test-only update requires them, runs every test under them when there is no selection, and rejects a selection outside them. The pipelines test acceptance tests configure `testing` and cover the rejection. Co-authored-by: Isaac <no-reply@databricks.com>
The service stores PipelineSpec.testing and returns it from Get Pipeline, but the Go SDK type drops it. The direct engine now reads pipelines with the same GET request, made raw, and keeps spec.testing in the remote state. bundle plan and bundle deploy then show and fix a remote change to testing, for example an edit by another client that removes it. The test server returns testing from Get Pipeline like the service. Co-authored-by: Isaac <no-reply@databricks.com>
It deploys a bundle with testing.sources, runs databricks pipelines test with a JUnit report and destroys the bundle. Locally it runs against the test server. With CLOUD_ENV set it runs on the workspace only when TEST_PIPELINES_TESTING_ENABLED is set, for a workspace with pipeline testing enabled; otherwise it is skipped. Co-authored-by: Isaac <no-reply@databricks.com>
With testing.sources, the folder that test IDs are relative to can sit above root_path or hold test sources in two folders. The pipeline-id and test-ids acceptance tests now cover these layouts, including a failing test ID printed in a subfolder and passed back from there. Co-authored-by: Isaac <no-reply@databricks.com>
…ranslation The validation now runs after TranslatePaths, so it checks the paths that deploy sends. It catches the usual mistakes: package names, URLs, other pip options such as -i, "..", paths outside /Workspace and /Volumes, and a requirements file without -r. The API checks the rest, so a change to its rule can't make validate reject a dependency that the API accepts. A changelog entry says that bundle plan now shows a change to testing made outside the bundle. Co-authored-by: Isaac <no-reply@databricks.com>
Each request that the Go SDK cannot express yet built a new API client, with its own rate limiter. The pipeline resource now builds it once. A test checks that those requests carry the workspace ID header, and that a pipeline without testing still sends the SDK's request. Unit tests that repeat the testing acceptance test are removed. Co-authored-by: Isaac <no-reply@databricks.com>
The testing test reads the pipeline ID with read_id.py, sends the command output it does not compare to a LOG file, and sets MSYS_NO_PATHCONV for its api paths, which start with "/". The live test sends the run's output to a LOG file and drops an exit code line that could only print 0, because the script stops when the command fails. Co-authored-by: Isaac <no-reply@databricks.com>
A local path after -e in environment dependencies was translated as a file, so a project folder such as -e ./project failed before deploy with "is a directory". This applies to jobs, pipelines, and the pipeline testing field. After -e, a folder in the bundle now maps to its deployed folder, and optional extras stay after it, so -e ./project[test] becomes -e <file_path>/project[test]. A file after -e, and every path after -r, are translated as before. The hint for test dependencies now shows the -e ./project[test] form. Co-authored-by: Isaac <no-reply@databricks.com>
The test-ids acceptance test expects both runs of the split pipeline to fail, so it uses musterr instead of errcode. The raw requests test appends inline instead of through a closure that only names the append. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
tharunbhuk-db
force-pushed
the
tharun/pipelines-test-pr4-bundles
branch
from
October 30, 2026 19:02
cae44ca to
a0679e5
Compare
Contributor
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
tharunbhuk-db
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 4/4: #7271 command · #7272 output · #7273 JUnit · #7274 bundles. Builds on #7273.
This PR's changes: tharunbhuk-db/cli@tharun/pipelines-test-pr3-junit...tharun/pipelines-test-pr4-bundles
Design doc: https://docs.google.com/document/d/1atW-JG_hQu-G-YX-m1M7OodNZdH7LMc9H7CBlQAv5gw/edit
Changes
resources.pipelines.<key>.testing, with the test sources and test-only dependencies thatdatabricks pipelines testruns with.testingare translated likelibraries.bundle validatecatches common mistakes in test dependencies.bundle planshows changes totestingmade outside the bundle, andbundle deployundoes them.-e, such as-e ./project[test], now maps to its deployed folder instead of failing.Why
A bundle needs to declare where a pipeline's tests and their dependencies live. The CLI uses its own types and raw calls for
testing, because the Go SDK does not have the field yet. Depends on API support fortestingon Create and Edit. Keep as draft until then.Tests
acceptance/bundle/resources/pipelines/testing*,acceptance/bundle/environments/editable-folderandacceptance/pipelines/test.This pull request and its description were written by Isaac.