[tharunbhuk-db]

🥞 Stacked PR

• stack/pipelines-test-pr1-command [Files changed]
  • stack/pipelines-test-pr2-output [Files changed]
    • stack/pipelines-test-pr3-junit [Files changed] ← this PR
      • stack/pipelines-test-pr4-bundles [Files changed]

---

Changes

• Add --junit-xml <path>, which writes one testcase per collected test case, with its outcome and duration. Cases are named relative to the project folder, as pytest names them, so the names do not depend on the working directory. With workspace.sync_mode: immutable, the names keep the deployed snapshot path, which changes with each deployment; the help says so.
• A case without a final outcome is an error. A run without tests, an incomplete run, a timeout or a late update failure adds a failing entry that names the cause.
• Every exit writes the report, including invalid flags and a --resume of an update that is not a test update, so an older report is never left behind. --junit-xml cannot be used with --no-wait, and the --resume command printed after a timeout keeps --junit-xml so that it replaces the partial report.

Why

CI systems read JUnit XML to show test results and to gate merges.

Tests

• Unit tests for the case names, the outcome mapping, the failing entries and write errors.
• Acceptance tests that compare the written reports (out.*.xml) for a passing run, each failing outcome, invalid flags, a timeout and resume, and errors before the update is read.

This pull request and its description were written by Isaac.

[github-actions]

Commit: 7c61f68c6 | Workflow run

Regression test report

Selected 9 changed tests (limit=20, 0 not selected)

9 tests: regression=9 unreleased=0 coverage=0 failing=0

test	branch	main (aed6702)	latest (v1.23.0)
TestAccept/pipelines/databricks-cli-help	✅	❌	❌
TestAccept/pipelines/test/arguments	✅	❌	❌
TestAccept/pipelines/test/immutable	✅	❌	❌
TestAccept/pipelines/test/many-cases	✅	❌	❌
TestAccept/pipelines/test/pipeline-id	✅	❌	❌
TestAccept/pipelines/test/results	✅	❌	❌
TestAccept/pipelines/test/run-control	✅	❌	❌
TestAccept/pipelines/test/selection	✅	❌	❌
TestAccept/pipelines/test/submit-error	✅	❌	❌

TestAccept/pipelines/databricks-cli-help: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/databricks-cli-help/output.txt
        +++ /tmp/TestAcceptpipelinesdatabricks-cli-help188765129/002/output.txt
        @@ -30,7 +30,6 @@
           run                   Run a pipeline
           show                  Preview a table's columns and sample rows
           stop                  Stop a pipeline
        -  test                  *Preview* Run the tests of a pipeline
         
         Management Commands
           apply-environment     *Public Preview* Apply the latest environment to the pipeline.

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/arguments: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments1335989516/002/output.txt
        @@ -1,100 +1,747 @@
         
         >>> [CLI] pipelines test --help
        -Run the tests of a pipeline in a test-only update.
        +The Lakeflow Spark Declarative Pipelines API allows you to create, edit,
        +  delete, start, and view details about pipelines.
         
        -This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
        +  Spark Declarative Pipelines is a framework for building reliable,
        +  maintainable, and testable data processing pipelines. You define the
        +  transformations to perform on your data, and Spark Declarative Pipelines
        +  manages task orchestration, cluster management, monitoring, data quality, and
        +  error handling.
         
        -PIPELINE_KEY is the unique name of the pipeline, as defined in its YAML file.
        -It is required when test selections are given. If there is only one pipeline
        -in the project and no arguments are given, it is selected automatically.
        -Outside a project, identify the pipeline with --pipeline-id; all positional
        -arguments are then test selections.
        +  Instead of defining your data pipelines using a series of separate Apache
        +  Spark tasks, Spark Declarative Pipelines manages how your data is transformed
        +  based on a target schema you define for each processing step. You can also
        +  enforce data quality with Spark Declarative Pipelines expectations.
        +  Expectations allow you to define expected data quality and specify how to
        +  handle records that fail those expectations.
         
        -TEST_SELECTION is a test directory, file, or pytest test ID, for example
        -tests/test_orders.py::test_backfill. In a project, a relative path is a local
        -path, relative to the working directory, and is converted to its deployed
        -workspace path. With --pipeline-id, a relative path is relative to the
        -pipeline's root_path. Paths that start with / are workspace paths. Without a
        -selection, all of the pipeline's tests run.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Each --pytest-arg passes one argument to pytest, in the order given. Only -k
        -and -m are supported, each followed by its value in the next --pytest-arg, for
        -example:
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -  databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -The command waits for the results and succeeds only if at least one test was
        -collected and none failed or errored. --no-wait returns after the update starts.
        ---timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
        -to read the results later. To stop the update, run 'pipelines stop'.
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        ---junit-xml writes one entry per test with its outcome and duration. An
        -incomplete run, a run without tests, or a timeout adds a failing entry that
        -names the cause. With workspace.sync_mode immutable, test names keep the path
        -of the deployed snapshot, which changes with each deployment.
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +=== The command is experimental and off by default
        +>>> musterr [CLI] pipelines test --pipeline-id 1234
        +Error: unknown flag: --pipeline-id
        +
         Usage:
        -  databricks pipelines test [PIPELINE_KEY] [TEST_SELECTION...] [flags] [--pytest-arg=ARG ...]
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline Test Flags:
        -      --junit-xml string         Write the results as a JUnit XML report to this file, replacing it.
        -      --no-wait                  Return after the test update starts, without waiting for results.
        -      --pipeline-id string       ID of the pipeline to test, instead of a PIPELINE_KEY.
        -      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        -      --resume string            Read the results of an existing test update instead of starting one.
        -      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
         Flags:
        -  -h, --help   help for test
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
         Global Flags:
               --debug            enable debug logging
           -o, --output type      output type: text or json (default text)
           -p, --profile string   ~/.databrickscfg profile
           -t, --target string    bundle target to use (if applicable)
        -      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
        -=== The command is experimental and off by default
        ->>> musterr [CLI] pipelines test --pipeline-id 1234
        -Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        +
         === pytest arguments must be passed with --pytest-arg
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -k orders
        -Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
... (629 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.23.0):

acceptance_test.go:1341: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments4158609215/002/output.txt
        @@ -1,100 +1,747 @@
         
         >>> [CLI] pipelines test --help
        -Run the tests of a pipeline in a test-only update.
        +The Lakeflow Spark Declarative Pipelines API allows you to create, edit,
        +  delete, start, and view details about pipelines.
         
        -This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
        +  Spark Declarative Pipelines is a framework for building reliable,
        +  maintainable, and testable data processing pipelines. You define the
        +  transformations to perform on your data, and Spark Declarative Pipelines
        +  manages task orchestration, cluster management, monitoring, data quality, and
        +  error handling.
         
        -PIPELINE_KEY is the unique name of the pipeline, as defined in its YAML file.
        -It is required when test selections are given. If there is only one pipeline
        -in the project and no arguments are given, it is selected automatically.
        -Outside a project, identify the pipeline with --pipeline-id; all positional
        -arguments are then test selections.
        +  Instead of defining your data pipelines using a series of separate Apache
        +  Spark tasks, Spark Declarative Pipelines manages how your data is transformed
        +  based on a target schema you define for each processing step. You can also
        +  enforce data quality with Spark Declarative Pipelines expectations.
        +  Expectations allow you to define expected data quality and specify how to
        +  handle records that fail those expectations.
         
        -TEST_SELECTION is a test directory, file, or pytest test ID, for example
        -tests/test_orders.py::test_backfill. In a project, a relative path is a local
        -path, relative to the working directory, and is converted to its deployed
        -workspace path. With --pipeline-id, a relative path is relative to the
        -pipeline's root_path. Paths that start with / are workspace paths. Without a
        -selection, all of the pipeline's tests run.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Each --pytest-arg passes one argument to pytest, in the order given. Only -k
        -and -m are supported, each followed by its value in the next --pytest-arg, for
        -example:
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -  databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -The command waits for the results and succeeds only if at least one test was
        -collected and none failed or errored. --no-wait returns after the update starts.
        ---timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
        -to read the results later. To stop the update, run 'pipelines stop'.
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        ---junit-xml writes one entry per test with its outcome and duration. An
        -incomplete run, a run without tests, or a timeout adds a failing entry that
        -names the cause. With workspace.sync_mode immutable, test names keep the path
        -of the deployed snapshot, which changes with each deployment.
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +=== The command is experimental and off by default
        +>>> musterr [CLI] pipelines test --pipeline-id 1234
        +Error: unknown flag: --pipeline-id
        +
         Usage:
        -  databricks pipelines test [PIPELINE_KEY] [TEST_SELECTION...] [flags] [--pytest-arg=ARG ...]
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline Test Flags:
        -      --junit-xml string         Write the results as a JUnit XML report to this file, replacing it.
        -      --no-wait                  Return after the test update starts, without waiting for results.
        -      --pipeline-id string       ID of the pipeline to test, instead of a PIPELINE_KEY.
        -      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        -      --resume string            Read the results of an existing test update instead of starting one.
        -      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
         Flags:
        -  -h, --help   help for test
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
         Global Flags:
               --debug            enable debug logging
           -o, --output type      output type: text or json (default text)
           -p, --profile string   ~/.databrickscfg profile
           -t, --target string    bundle target to use (if applicable)
        -      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
        -=== The command is experimental and off by default
        ->>> musterr [CLI] pipelines test --pipeline-id 1234
        -Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        +
         === pytest arguments must be passed with --pytest-arg
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -k orders
        -Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
... (629 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

TestAccept/pipelines/test/immutable: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/immutable/output.txt
        +++ /tmp/TestAcceptpipelinestestimmutable69113373/002/output.txt
        @@ -8,46 +8,108 @@
         
         === Local test paths are not supported with workspace.sync_mode immutable
         >>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown command "test" for "databricks pipelines"
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Error: cannot resolve test selection "tests/test_orders.py::test_backfill": local test paths are not supported with workspace.sync_mode immutable; pass an absolute workspace path
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
         === Absolute workspace paths and running all tests still work
         >>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        ->>> [CLI] pipelines test my_pipeline --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
        -      ]
        -    }
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true
        -  }
        -}
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
        +Exit code: 1

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/many-cases: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1341
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1082
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:641
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] pipelines deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-many-cases/default/files...\nCreated pipelines.my_pipeline\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\nView your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]\n\n=== Every case is counted once across pages, polls, and the final read\n>>> tail -n 1 LOG.many\n200 tests: 200 passed, 0 failed, 0 skipped, 0 errors\nEvent pages read: 8\nPages read with page_token: 4\n"
        	            	actual  : "\n>>> [CLI] pipelines deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-many-cases/default/files...\nCreated pipelines.my_pipeline\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\nView your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]\n\n=== Every case is counted once across pages, polls, and the final read\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -9,6 +9,3 @@
        	            	 === Every case is counted once across pages, polls, and the final read
        	            	->>> tail -n 1 LOG.many
        	            	-200 tests: 200 passed, 0 failed, 0 skipped, 0 errors
        	            	-Event pages read: 8
        	            	-Pages read with page_token: 4
        	            	+Exit code: 1
        	            	 
        	Test:       	TestAccept/pipelines/test/many-cases
        	Messages:   	pipelines/test/many-cases/output.txt vs /tmp/TestAcceptpipelinestestmany-cases3768341867/002/output.txt
    acceptance_test.go:1119: 
        LOG.many: Error: unknown command "test" for "databricks pipelines"
        LOG.many: 
        LOG.many: Usage:
        LOG.many:   databricks pipelines [flags]
        LOG.many:   databricks pipelines [command]
        LOG.many: 
        LOG.many: Main Commands
        LOG.many:   deploy                Deploy pipelines
        LOG.many:   destroy               Destroy a pipelines project
        LOG.many:   dry-run               Validate correctness of the pipeline's graph
        LOG.many:   generate              Generate pipeline configuration
        LOG.many:   history               Retrieve past runs for a pipeline
        LOG.many:   init                  Initialize a new pipelines project
        LOG.many:   logs                  Retrieve events for a pipeline
        LOG.many:   open                  Open a pipeline in the browser
        LOG.many:   run                   Run a pipeline
        LOG.many:   show                  Preview a table's columns and sample rows
        LOG.many:   stop                  Stop a pipeline
        LOG.many: 
        LOG.many: Management Commands
        LOG.many:   apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        LOG.many:   clone                 Clone a pipeline.
        LOG.many:   create                Create a pipeline.
        LOG.many:   delete                Delete a pipeline.
        LOG.many:   get                   Get a pipeline.
        LOG.many:   get-update            Get a pipeline update.
        LOG.many:   list-pipeline-events  List pipeline events.
        LOG.many:   list-pipelines        List pipelines.
        LOG.many:   list-updates          List pipeline updates.
        LOG.many:   start-update          Start a pipeline.
        LOG.many:   update                Edit a pipeline.
        LOG.many: 
        LOG.many: Permission Commands
        LOG.many:   get-permission-levels Get pipeline permission levels.
        LOG.many:   get-permissions       Get pipeline permissions.
        LOG.many:   set-permissions       Set pipeline permissions.
        LOG.many:   update-permissions    Update pipeline permissions.
        LOG.many: 
        LOG.many: Flags:
        LOG.many:   -h, --help          help for pipelines
        LOG.many:       --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        LOG.many: 
        LOG.many: Global Flags:
        LOG.many:       --debug            enable debug logging
        LOG.many:   -o, --output type      output type: text or json (default text)
        LOG.many:   -p, --profile string   ~/.databrickscfg profile
        LOG.many:   -t, --target string    bundle target to use (if applicable)
        LOG.many: 
        LOG.many: Use "databricks pipelines [command] --help" for more information about a command.

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/pipeline-id: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/pipeline-id/output.txt
        +++ /tmp/TestAcceptpipelinestestpipeline-id2993692678/002/output.txt
        @@ -1,67 +1,55 @@
         
         === Relative test selections resolve against the pipeline root_path
         >>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID] tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests --no-wait
        -Pipeline: [UUID]
        -Update: [UUID]
        +Error: unknown flag: --pipeline-id
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill",
        -        "/Workspace/Users/someone@example.com/other/tests"
        -      ]
        -    }
        -  }
        -}
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -=== Test case IDs are shown relative to the pipeline root_path
        ->>> [CLI] pipelines test --pipeline-id [UUID] tests
        -Pipeline: [UUID]
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_orders.py::test_valid_order  PASSED (1.00s)
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -=== Without a root_path, test selections must be workspace paths
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID] tests
        -Error: cannot resolve test selection "tests": the pipeline has no root_path; use an absolute workspace path such as /Workspace/Users/someone@example.com/tests
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -=== With --no-wait and only workspace paths, the pipeline is not read
        ->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID] /Workspace/Users/someone@example.com/tests/test_other.py::test_x --no-wait
        -Pipeline: [UUID]
        -Update: [UUID]
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        ->>> print_requests.py --get //pipelines
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_other.py::test_x"
        -      ]
        -    }
        -  }
        -}
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        -=== Without a root_path, test case IDs are absolute workspace paths
        ->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID] /Workspace/Users/someone@example.com/orders/tests/test_orders.py
        -Pipeline: [UUID]
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order  PASSED (1.00s)
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
         
        -=== Outside a project, --pipeline-id is required
        ->>> musterr [CLI] pipelines test tests
        -Error: outside a project, identify the pipeline with --pipeline-id
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.import: Importing files from orders
        LOG.import: tests/test_orders.py -> /Workspace/Users/someone@example.com/orders/tests/test_orders.py
        LOG.import: Import complete

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/results: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/results/output.txt
        +++ /tmp/TestAcceptpipelinestestresults4101088177/002/output.txt
        @@ -8,145 +8,55 @@
         
         === All selected tests pass
         >>> [CLI] pipelines test my_pipeline tests/test_orders.py::test_valid_order --junit-xml out.passed.xml
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --junit-xml
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_orders.py::test_valid_order  PASSED (1.00s)
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
        -JUnit report: out.passed.xml
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        ->>> print_requests.py --get //events
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID]/events",
        -  "q": {
        -    "filter": "update_id='[UUID]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID]/events",
        -  "q": {
        -    "filter": "update_id='[UUID]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID]/events",
        -  "q": {
        -    "filter": "update_id='[UUID]' AND timestamp >= '[TIMESTAMP]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID]/events",
        -  "q": {
        -    "filter": "update_id='[UUID]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -=== Failed, errored, and skipped tests
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py --junit-xml out.failures.xml
        -Recommendation: This command runs the last deployed version of the code
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_orders.py::test_valid_order  PASSED (1.00s)
        -tests/test_orders.py::test_backfill_fails  FAILED (1.00s)
        -tests/test_orders.py::test_legacy_format_skipped  SKIPPED (1.00s)
        -tests/test_orders.py::test_fixture_error  ERROR (1.00s)
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        -FAILURES
        -tests/test_orders.py::test_backfill_fails
        -  AssertionError: assert 100 == 150
        -tests/test_orders.py::test_fixture_error
        -  RuntimeError: fixture 'orders' setup failed
         
        -4 tests: 1 passed, 1 failed, 1 skipped, 1 error
        -JUnit report: out.failures.xml
        -
        -=== No tests collected
        ->>> musterr [CLI] pipelines test my_pipeline tests/helpers.py --junit-xml out.no-tests.xml
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -
        -No tests were collected.
        -JUnit report: out.no-tests.xml
        -
        -=== The update fails after all tests passed
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_delivery.py --junit-xml out.update-failed.xml
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_delivery.py::test_breaks_update_after_summary  PASSED (1.00s)
        -
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
        -JUnit report: out.update-failed.xml
        -Error: update [UUID] is FAILED: ResultDeliveryException: Test results could not be verified.
        -
        -=== The update fails with an error of its own after a test failed
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_late_failure.py --junit-xml out.late-failure.xml
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_late_failure.py::test_backfill_fails  FAILED (1.00s)
        -tests/test_late_failure.py::test_breaks_update_after_summary  PASSED (1.00s)
        -
        -FAILURES
        -tests/test_late_failure.py::test_backfill_fails
        -  AssertionError: assert 100 == 150
        -
        -2 tests: 1 passed, 1 failed, 0 skipped, 0 errors
        -JUnit report: out.late-failure.xml
        -Error: update [UUID] is FAILED: ResultDeliveryException: Test results could not be verified.
        -
        -=== Compute terminates before all results are received
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_compute.py --junit-xml out.incomplete.xml
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_compute.py::test_small_order  PASSED (1.00s)
        -
        -RUN INCOMPLETE: the update failed before all test results were received.
        -Results received: 1 passed.
        -JUnit report: out.incomplete.xml
        -Error: update [UUID] is FAILED: ComputeTerminatedException: Compute terminated unexpectedly.
        +Exit code: 1
        
    acceptance_test.go:1312: Missing output file: out.failures.xml
    acceptance_test.go:1312: Missing output file: out.incomplete.xml
    acceptance_test.go:1312: Missing output file: out.late-failure.xml
    acceptance_test.go:1312: Missing output file: out.no-tests.xml
    acceptance_test.go:1312: Missing output file: out.passed.xml
    acceptance_test.go:1312: Missing output file: out.update-failed.xml

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/run-control: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/run-control/output.txt
        +++ /tmp/TestAcceptpipelinestestrun-control365003030/002/output.txt
        @@ -8,86 +8,55 @@
         
         === --no-wait returns after the update starts
         >>> [CLI] pipelines test my_pipeline tests/test_orders.py --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -=== --resume reads the results of an existing update
        ->>> [CLI] pipelines test my_pipeline --resume [UUID]
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Running tests...
        -tests/test_orders.py::test_valid_order  PASSED (1.00s)
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -=== --timeout stops waiting but not the update
        ->>> [CLI] pipelines test my_pipeline tests/test_slow.py --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        ->>> musterr [CLI] pipelines test my_pipeline --resume [UUID] --timeout 3s --junit-xml out.resumed.xml
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Running tests...
        -JUnit report: out.resumed.xml
        -Error: stopped waiting for update [UUID] after 3s; the update keeps running, to get its results run: databricks pipelines test my_pipeline --junit-xml out.resumed.xml --resume [UUID]
         
        ->>> cat out.resumed.xml
        -<?xml version="1.0" encoding="UTF-8"?>
        -<testsuites>
        -  <testsuite name="my_pipeline" tests="2" failures="0" errors="2" skipped="0">
        -    <testcase classname="tests.test_slow" name="test_hang_on_lock">
        -      <error message="no result received"></error>
        -    </testcase>
        -    <testcase name="update [UUID]">
        -      <error message="stopped waiting for update [UUID] after 3s; the update keeps running, to get its results run: databricks pipelines test my_pipeline --junit-xml out.resumed.xml --resume [UUID]">stopped waiting for update [UUID] after 3s; the update keeps running, to get its results run: databricks pipelines test my_pipeline --junit-xml out.resumed.xml --resume [UUID]</error>
        -    </testcase>
        -  </testsuite>
        -</testsuites>
        -
        -=== The command to read the results keeps the profile
        ->>> musterr [CLI] pipelines test my_pipeline --resume [UUID] --timeout 3s -p run-control
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Running tests...
        -Error: stopped waiting for update [UUID] after 3s; the update keeps running, to get its results run: databricks pipelines test my_pipeline --profile run-control --resume [UUID]
        -
        -=== Stopping the pipeline cancels the test update
        ->>> [CLI] pipelines stop my_pipeline
        -Stopping my_pipeline...
        -my_pipeline has been stopped.
        -
        ->>> musterr [CLI] pipelines test my_pipeline --resume [UUID] --junit-xml out.resumed.xml
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -
        -RUN INCOMPLETE: the update was canceled before all test results were received.
        -Results received: none.
        -JUnit report: out.resumed.xml
        -Error: update [UUID] is CANCELED
        -
        -=== --resume rejects an update that is not a test update, and an older report is replaced
        ->>> musterr [CLI] pipelines test my_pipeline --resume [UUID] --junit-xml out.not-test-update.xml
        -Error: update [UUID] is not a test update; pass the update ID that pipelines test printed
        -
        -=== Outside a project, an older passing report is replaced
        -Error: outside a project, identify the pipeline with --pipeline-id
        -
        -=== A project with errors names the first error in the report
        ->>> musterr [CLI] pipelines test my_pipeline --junit-xml ../out.invalid.xml
        -Error: reference does not exist: ${var.missing}
        -
        -
        -=== When the report cannot be written either, the write error is shown
        ->>> musterr [CLI] pipelines test my_pipeline --junit-xml missing/report.xml
        -Error: reference does not exist: ${var.missing}
        -
        -Error: failed to write JUnit report: open missing/report.xml: no such file or directory
        +Exit code: 1
        
    acceptance_test.go:1312: Missing output file: out.invalid.xml
    acceptance_test.go:1312: Missing output file: out.not-test-update.xml
    acceptance_test.go:1312: Missing output file: out.outside.xml
    acceptance_test.go:1312: Missing output file: out.resumed.xml
    acceptance_test.go:1131: Test produced unexpected files:
        submit.txt

latest (v1.23.0): same output as main (aed6702)

2 more failure details omitted, see the full report in the run.

[eng-dev-ecosystem-bot]

Integration test report

Commit: 7c61f68

Run: 41686110931

	Env	🔄​flaky	✅​pass	🙈​skip	Time
🔄​	aws linux-2core-8gb	1	280	24	12:57
🔄​	aws-windows-latest-4core-16gb	1	282	22	9:25
🔄​	azure linux-2core-8gb	2	278	24	12:33
🔄​	azure-windows-latest-4core-16gb	1	281	22	10:41
✅​	gcp linux-2core-8gb		281	24	13:12
✅​	gcp-windows-latest-4core-16gb		283	22	7:45

	Test Name	aws linux-2core-8gb	aws-windows-latest-4core-16gb	azure linux-2core-8gb	azure-windows-latest-4core-16gb
🔄​	TestFsCpDir	✅​p	✅​p	🔄​f	✅​p
🔄​	TestFsCpDir/uc-volumes_to_dbfs	✅​p	✅​p	🔄​f	✅​p
🔄​	TestSyncFullFileSync	✅​p	✅​p	✅​p	🔄​f
🔄​	TestSyncIncrementalFileSync	✅​p	🔄​f	✅​p	✅​p
🔄​	TestSyncIncrementalSyncPythonNotebookToFile	🔄​f	✅​p	✅​p	✅​p

Top 20 slowest tests (at least 2 minutes):

duration	env	testname
7:19	aws-windows-latest-4core-16gb	TestAccept
7:07	gcp-windows-latest-4core-16gb	TestAccept
6:55	azure-windows-latest-4core-16gb	TestAccept
4:20	aws linux-2core-8gb	TestAccept
4:13	gcp linux-2core-8gb	TestAccept
4:10	azure linux-2core-8gb	TestAccept
3:23	azure-windows-latest-4core-16gb	TestFilerWorkspaceFilesExtensionsReadDir
3:10	azure-windows-latest-4core-16gb	TestFilerWorkspaceFilesExtensionsDelete
2:47	azure linux-2core-8gb	TestFilerWorkspaceFilesExtensionsReadDir
2:32	azure linux-2core-8gb	TestExportDir
2:19	gcp-windows-latest-4core-16gb	TestFilerWorkspaceFilesExtensionsDelete
2:15	gcp linux-2core-8gb	TestExportDirDoesNotOverwrite
2:15	aws linux-2core-8gb	TestFilerWorkspaceFilesExtensionsReadDir
2:12	gcp linux-2core-8gb	TestFilerWorkspaceNotebook/rJupyterNb.ipynb
2:10	gcp-windows-latest-4core-16gb	TestFilerWorkspaceFilesExtensionsStat
2:07	azure-windows-latest-4core-16gb	TestFilerWorkspaceFilesExtensionsRead
2:07	aws linux-2core-8gb	TestSyncFullFileSync
2:06	gcp linux-2core-8gb	TestExport
2:06	gcp linux-2core-8gb	TestFilerWorkspaceFilesExtensionsRead
2:04	aws-windows-latest-4core-16gb	TestFilerWorkspaceFilesExtensionsReadDir

[pranshupand-db]

os.WriteFile truncates first, so a crash mid-write leaves a half-written file that CI reads as broken. Could we use atomicfile.Write from libs/atomicfile? bundle/deploy/state_pull.go already uses it.

[pranshupand-db]

strings.Index(testID, "[") searches the whole ID, so tests/[weird]/test_a.py::test_x gets classname "" and name tests.[weird].test_a.py::test_x. Searching only after the first :: fixes it. Worth adding that case to TestJUnitCaseName.

[pranshupand-db]

This entry is named update <id>, while pre-update errors are named pipelines test (line 167). On the timeout path, the update is still running and the message doesn't start with RUN INCOMPLETE:. Could we use one naming scheme, with a clear message for the local-timeout case?

[pranshupand-db]

Outside a project, positional args are selections, so pipelines test tests/test_x.py --junit-xml r.xml names the suite tests/test_x.py, and with no args the suite name is empty. Could we use the pipeline name or ID once it resolves?

[pranshupand-db]

Non-blocking: thanks for documenting this. Classnames that change with each deploy break CI test-history views (flaky detection, trends). A follow-up could name cases relative to the snapshot root.

[pranshupand-db]

Non-blocking: the "an older report is replaced" check now appears in five scripts: arguments, submit-error, run-control, outside, and an invalid project. Each shows the same thing, that a failure before the update still rewrites the report. One case before the request and one at submit would cover it, and the other goldens could go.