Skip to content

air: don't announce retries before a run fails - #7270

Merged
maggiewang-db merged 2 commits into
mainfrom
maggiewang/air-drop-retry-notice
Oct 30, 2026
Merged

maggiewang-db merged 2 commits into
mainfrom
maggiewang/air-drop-retry-notice

Conversation

@maggiewang-db

@maggiewang-db maggiewang-db commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Remove the "Failed attempts will be retried up to N times." line that databricks air run --watch prints between "Monitoring run and streaming logs..." and the logs divider. A retry is still reported when it actually happens (Retrying (1 of 3)...), so no information is lost.

Why

max_retries defaults to 3, so every watched run printed this line before any logs streamed. On a healthy run it reads as if an attempt had already failed, which users reported as a papercut.

Tests

  • TestRunWatchStreamsLogs now asserts that a successful watched run never mentions retries. The existing logstream_test.go coverage of Retrying (1 of 3)... on a real retry is unchanged.
  • All cmd/air unit tests and the TestAccept/air/ acceptance tests pass locally; golangci-lint on cmd/air reports no issues.

This pull request and its description were written by Isaac.

maggiewang-db and others added 2 commits October 30, 2026 18:35
`air run --watch` printed "Failed attempts will be retried up to 3 times."
on every run, because max_retries defaults to 3. On a healthy run this reads
as if something already failed. The log stream already prints
"Retrying (n of m)..." when a retry actually happens.

Tracking: CNXT-2481.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 30, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: eebc79e

Run: 41109167411

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 15 5:29
✅​ aws-windows-latest-4core-16gb 283 13 3:42
✅​ azure linux-2core-8gb 280 15 5:15
✅​ azure-windows-latest-4core-16gb 282 13 3:34
✅​ gcp linux-2core-8gb 281 15 5:30
✅​ gcp-windows-latest-4core-16gb 283 13 3:29
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:59 aws linux-2core-8gb TestAccept
3:45 azure linux-2core-8gb TestAccept
3:43 gcp linux-2core-8gb TestAccept
3:35 aws-windows-latest-4core-16gb TestAccept
3:28 azure-windows-latest-4core-16gb TestAccept
3:23 gcp-windows-latest-4core-16gb TestAccept

@maggiewang-db
maggiewang-db marked this pull request as ready for review October 30, 2026 19:27
@maggiewang-db
maggiewang-db requested review from a team as code owners October 30, 2026 19:27
@maggiewang-db
maggiewang-db added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit eb72905 Oct 30, 2026
28 checks passed
@maggiewang-db
maggiewang-db deleted the maggiewang/air-drop-retry-notice branch October 30, 2026 19:49
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: eb72905

Run: 41118487922

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 1504 1226 134:42
🔄​ aws-windows-latest-4core-16gb 3 1426 1251 147:31
🔄​ azure linux-2core-8gb 3 1348 1278 103:57
🔄​ azure-windows-latest-4core-16gb 4 1272 1303 127:12
✅​ gcp linux-2core-8gb 1338 1283 110:02
✅​ gcp-windows-latest-4core-16gb 1263 1308 134:03
9 interesting tests: 9 flaky
Test Name aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb
🔄​ TestAccept 🔄​f 🔄​f ✅​p
🔄​ TestAccept/bundle/invariant/continue_293 ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/invariant/continue_293/DMS=/INPUT_CONFIG=app.yml.tmpl ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=job_table_update_trigger.yml.tmpl/READPLAN=1 ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN= ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=1 ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=volume_uppercase_name.yml.tmpl/READPLAN=1 ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/postgres_roles/recreate 🔄​f 🙈​s 🙈​s
🔄​ TestAccept/bundle/resources/postgres_roles/recreate/DMS=true 🔄​f
Top 50 slowest tests (at least 2 minutes):
duration env testname
10:46 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:35 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:54 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
9:30 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:29 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
9:23 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:11 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:33 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:32 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:31 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:30 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:29 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:28 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:27 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:26 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:24 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:38 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:37 gcp-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
7:33 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
7:27 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
7:25 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:19 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
7:17 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:17 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
7:16 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:10 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:07 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:05 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:54 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:51 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
6:48 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:48 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:39 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:36 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:31 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:31 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
6:29 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:24 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:11 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:10 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
6:09 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
6:00 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:51 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:31 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:28 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
5:20 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:14 gcp-windows-latest-4core-16gb TestAccept
5:13 azure linux-2core-8gb TestAccept/bundle/resources/job_runs/failed_run/DMS=
5:10 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
5:08 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants