Skip to content

Collect per-test coverage in the Linux test cells for the CI dashboard - #7694

Merged
denik merged 7 commits into
mainfrom
denik/ci-coverage
Nov 20, 2026
Merged

denik merged 7 commits into
mainfrom
denik/ci-coverage

Conversation

@denik

@denik denik commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Collect coverage in the existing Linux cells of task test and upload it as a small artifact, coverage-linux-<cell>. It is for the CI dashboard, which will show coverage by package and file (unit, acceptance and combined) and which acceptance tests cover each line.

  • When: on every push to main, and on a PR whose head commit message contains with-coverage (for testing changes to this). Windows, macOS and the merge queue do not collect it.
  • Unit tests: when CLI_GOCOVERDIR_UNIT is set, test-unit-root runs with -cover -coverpkg=./... into that coverage directory. It is a directory rather than -coverprofile because --rerun-fails runs go test again with the same flags, which would overwrite the profile with the coverage of the rerun alone.
  • Acceptance tests: each per-test coverage directory gets a name file with t.Name(). The directory name ends in a variant index that does not say which variant ran.
  • tools/coverpack (new, Go) packs both into one zstd file. It reads Go's binary covmeta/covcounters files directly instead of running go tool covdata per test: each distinct meta-data file is parsed once, and a test costs one pass over its counter files. The format is documented at the top of format.go. In short: the block layout of each file once, then the distinct per-file bitmaps (one bit per Go block) once, then per test, which bitmap it has for each file it covered. go run ./coverpack -dump FILE prints it back as Go cover profile text.
  • The metadata records the commit that was checked out. For a PR that is GitHub's merge commit, which is what the block positions belong to.

Why

Each per-test coverage directory holds its own ~900 KB copy of the CLI's coverage meta-data, so the raw directories come to several GB per cell. Most tests cover a file the same way as many other tests (mostly the CLI's startup): 2,259 tests have about 3M (test, file) pairs but only 12k distinct bitmaps.

direct cell without coverage (mean of 5 main runs) this PR
Run tests 6m19s 8m56s
Pack coverage – 1s (0.7s for 10,846 counter files)
Whole job 7m20s 10m24s
Artifact – 339 KB (2.9 MB uncompressed)

The directdms cell gives 307 KB. The overhead is running the tests with -cover; packing is negligible.

Tests

  • tools/coverpack/main_test.go builds a small program with -cover, runs it as two "tests" and one unit run, and checks that the packed and dumped profiles equal go tool covdata textfmt, including a line with two blocks and a function that never runs. A second test checks that every truncation of a file is rejected.
  • On 358 local acceptance tests, the dump matches the per-test covdata textfmt output in all 360 sections.
  • This PR's own runs (its commit messages contain with-coverage) exercise the CI path end to end; see the coverage-linux-* artifacts of its latest run.

This pull request and its description were written by Isaac.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 889eb44

Run: 44740237122

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 3:48
💚​ aws-windows-latest-4core-16gb 1 274 17 5:03
💚​ azure linux-2core-8gb 1 271 19 3:32
💚​ azure-windows-latest-4core-16gb 1 273 17 4:49
💚​ gcp linux-2core-8gb 1 272 19 3:26
💚​ gcp-windows-latest-4core-16gb 1 274 17 4:02
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:26 aws-windows-latest-4core-16gb TestAccept
4:01 gcp-windows-latest-4core-16gb TestAccept
3:20 azure-windows-latest-4core-16gb TestAccept

@github-actions github-actions Bot added the Dependencies Pull requests that update a dependency file label Nov 17, 2026
@denik denik changed the title Collect coverage in the Linux test cells for the CI dashboard Collect per-test coverage in the Linux test cells for the CI dashboard Nov 17, 2026
@denik
denik marked this pull request as ready for review November 20, 2026 12:51
@denik
denik requested a review from a team as a code owner November 20, 2026 12:51
denik and others added 7 commits November 20, 2026 12:55
On push to main, and on a PR whose head commit message has "with-coverage",
the Linux cells of `task test` record unit coverage (-coverpkg=./...) and
per-test acceptance coverage, and upload it as coverage-linux-<cell>: one
zstd --long file with a section per acceptance test (tools/coverpack.py).

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
Binary format instead of text: 3.9k tests have 3M (test, file) pairs but
only 13k distinct bitmaps. Same content, checked with --dump.

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
tools/coverpack (Go) replaces tools/coverpack.py: it decodes covmeta and
covcounters files itself instead of running `go tool covdata textfmt` per
test, so each distinct meta-data file is parsed once and a test costs one
pass over its counter files. Same output format; 0.27s instead of 10s on
358 local tests. A test compares it with covdata on a small program.

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
For a PR, CI tests GitHub's merge of it into main, not its head, so the
block positions belong to that commit.

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
Push-to-main runs on Linux now collect coverage and build with -cover, so they
no longer save the Go cache; the nightly run does. Packing and uploading
coverage can fail without failing the tests. coverpack rejects a few more
malformed inputs before allocating.

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik force-pushed the denik/ci-coverage branch from 99de2b0 to 889eb44 Compare November 20, 2026 12:56
@denik
denik added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 36fdeb1 Nov 20, 2026
38 checks passed
@denik
denik deleted the denik/ci-coverage branch November 20, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants