Skip to content

Open a tracking issue when a workflow fails on main - #7773

Draft
janniklasrose wants to merge 1 commit into
mainfrom
janniklasrose/report-main-failure
Draft

janniklasrose wants to merge 1 commit into
mainfrom
janniklasrose/report-main-failure

Conversation

@janniklasrose

Copy link
Copy Markdown
Member

Changes

Add a Report main failure workflow that opens a tracking issue when one of the workflows that run on main fails, and comments on that issue for every further failure until someone closes it. It covers build, check, python build, Changelog Preview, conftest and release-build, triggered via workflow_run on push and the nightly schedule.

The issue logic is the existing report-failure composite action used by the dependency bumpers. It gains two optional inputs, workflow and run-url, so a workflow_run handler can report on behalf of the failed run. The bumpers keep the current-run defaults. Since several main workflows now share one action, the repeat-failure comment names the workflow and the issue body no longer says "scheduled".

Integration tests are deliberately not covered: they run in another repository and are not expected to be green at all times.

Why

A failed run on main is only visible in the Actions tab, so nobody is notified. The merge queue catches most breakages before a commit lands, but not all: the Windows and macOS test cells only run on push, the changelog preview is not a merge-queue check, and release-build snapshots only run on main. We have had main go red in these ways in the past.

A separate workflow_run handler, rather than a failure job in each affected workflow, keeps the issue logic in one place and avoids granting issues: write to the test workflows.

Design notes:

  • The branches: [main] filter on workflow_run matches the head branch of the triggering run, so a fork PR from a branch named main would match too. The job additionally checks the triggering event is push or schedule.
  • A concurrency group serializes the reporter so one push that fails build and check at the same time opens a single issue instead of two.
  • Assignees are the same list as the bumpers, since a GitHub team mention posted by github-actions does not notify the team's members.

Tests

  • Both files pass the repo's yamlfmt -lint and the whitespace validator.
  • The only external action is actions/checkout, pinned to the SHA already used in push.yml, so the conftest pinning policy holds.
  • workflow_run handlers only fire from the default branch, so the behavior cannot be exercised on this PR. It takes effect once merged. The next failure on main is the first real test. I will close the issue it opens if it turns out to be a false positive.

No changelog fragment: this is CI-only.

This pull request and its description were written by Isaac.

A failed run on main is only visible in the Actions tab, so breakages that
the merge queue does not catch (Windows- or macOS-only test failures, the
changelog preview, snapshot builds) can go unnoticed. Add a workflow_run
handler that reuses the report-failure action from the dependency bumpers
to open one issue per breakage and comment on it for each further failure.
Integration tests are not covered: they run in another repository and are
not expected to always be green.

The action gains optional "workflow" and "run-url" inputs so a workflow_run
handler can report on behalf of the failed run; the bumpers keep the
current-run defaults.

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: f89ccae

Run: 44309994275

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 19:18
💚​ aws-windows-latest-4core-16gb 1 274 17 13:06
💚​ azure linux-2core-8gb 1 271 19 15:33
💚​ azure-windows-latest-4core-16gb 1 273 17 10:56
💚​ gcp linux-2core-8gb 1 272 19 14:55
💚​ gcp-windows-latest-4core-16gb 1 274 17 10:34
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 34 slowest tests (at least 2 minutes):
duration env testname
6:09 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
5:47 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files
4:20 azure linux-2core-8gb TestTelemetryEndpoint
4:17 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
4:12 gcp linux-2core-8gb TestFilerRecursiveDelete/workspace_files
3:48 azure-windows-latest-4core-16gb TestAccept
3:40 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:30 gcp-windows-latest-4core-16gb TestAccept
3:27 aws-windows-latest-4core-16gb TestAccept
3:25 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
3:21 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:20 aws-windows-latest-4core-16gb TestFilerReadWrite/workspace_files_extensions
3:14 aws-windows-latest-4core-16gb TestExportDir
3:10 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
3:04 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
3:04 gcp-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:59 azure-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:50 aws-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:36 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:30 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
2:29 gcp-windows-latest-4core-16gb TestLock
2:28 gcp-windows-latest-4core-16gb TestFilerWorkspaceNotebook/pythonJupyterNb.ipynb
2:28 aws linux-2core-8gb TestSyncNestedFolderDoesntFailOnNonEmptyDirectory
2:28 aws-windows-latest-4core-16gb TestFilerWorkspaceNotebook/rJupyterNb.ipynb
2:27 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:19 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:17 gcp linux-2core-8gb TestFilerReadWrite/workspace_files_extensions
2:14 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
2:12 gcp linux-2core-8gb TestImportDir
2:11 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:09 gcp-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:08 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:05 aws linux-2core-8gb TestWorkspaceFilesExtensions_ExportFormatIsPreserved/source_sql
2:01 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants