ci: shard the coverage job, and name a budget overrun as one - #648
Draft
bburda wants to merge 6 commits into
Draft
ci: shard the coverage job, and name a budget overrun as one#648bburda wants to merge 6 commits into
bburda wants to merge 6 commits into
Conversation
A test step killed by its cap is marked failed by the runner rather than cancelled, and the test that was in flight leaves no result file. `colcon test-result` then reports one erroring test named `<test>.xunit.missing_result` - a test that never failed - among thousands that never ran, while every package queued behind it disappears without trace. Read on its own that is indistinguishable from a single flake, so the budget gets re-run instead of raised. Measured over one month: 17 of 25 killed jobs died on a step cap, and every timeout in this file has been raised at least once. The step's own log is not readable from inside the job, so the overrun is detected by the clock: the start is recorded before the test step and compared against that step's cap afterwards. The same check warns at 80% of the cap, which is the number nobody had when the coverage job reached twelve seconds of margin. It deliberately does not count registered tests against result files - `test_dds_domain_allocation` is registered in every package and writes no xunit, so that comparison reports missing results on a run where everything passed. Container jobs now ask for 1 GB of /dev/shm instead of Docker's default 64 MB. Fast DDS puts a 512 KB segment plus port files there per participant, measured at about 0.65 MB, and a participant killed rather than shut down never gets them back: 15 live participants take 9.6 MB, and 125 killed ones fill 63 MB of 64. Locally, scripts/test.sh now reclaims that memory before a run through `fastdds shm clean`, the vendor's own tool, which decides a segment is stale by taking an exclusive non-blocking flock on its lock file - a lock the kernel drops when a process dies, SIGKILL included. Measured: eight killed participants stranded 5.24 MB across 66 files, all of it reclaimed.
The coverage job finished 89m48s against its 90 minute cap on 2026-08-27 - twelve seconds - and four of its last five runs on main sat between 86 and 90 minutes. Its test step grew from 19m13s to 51m07s in a month, 166%, while every other test-bearing job stayed flat. The cause is one line: build-and-test and jazzy-test both skip ros2_medkit_graph_watchdog, whose suite is 24 minutes on its own, and coverage skips only ros2_medkit_opcua. It alone kept running the suite the others carved out on 2026-08-24, which is the date its step time steps up. Measured test time per package on one run: integration_tests 1050s, graph_watchdog about 1440s, gateway plus fault_manager 389s, and everything else together 85s. The four shards follow those numbers, and their selections partition the workspace - checked against src/, 19 packages, none uncovered and no name that does not exist. Each shard builds the whole workspace because every package needs its dependencies, and tests only its own selection. Minutes are free on a public repository and wall clock is not, so paying for the build four times to quarter the wall clock is the trade worth making. fail-fast is off: with it, one failing shard cancels the others, and a cancelled job cannot be told apart from one killed by its cap. The completeness gate, the HTML report and the Codecov upload move to a merge job, because a shard's tracefile is legitimately partial and no single shard can answer whether the report still describes the whole workspace. lcov -a sums execution counts for matching files, so a line covered by only one shard is covered in the merged report - verified on two tracefiles where a line uncovered in one and covered in the other merges to covered, and the totals sum. The cap stays at 90 for now. The worst shard should land near 45, but the first run starts from a cold per-shard ccache, so the number to lower it by is one to read from the margin the test step now records, not one to guess.
… proven before it reaches main
…nch name in github.ref
The codecov action's dependency check fails the step outright when curl is absent, rather than skipping the upload, so the merge job died at the last step with every tracefile already merged and the completeness gate already green.
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment Thanks for integrating Codecov - We've got you covered ☂️ |
The two commits above widened the condition so this branch could execute the sharded coverage job at all - it only runs on push to main, so the change was otherwise unverifiable until it had already landed there. It has now run: four shards green, four tracefiles merged, the completeness gate green on the merged report, and the same line universe as the unsharded run. Restoring the condition rather than reverting the commits, so the run that proves it stays reachable from this branch's history.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
Three CI changes, each from a measurement rather than a hunch.
The coverage job is twelve seconds from failing. It finished 89m48s against its
90 minute cap on 2026-08-27, and four of its last five runs on main sat between 86 and
90 minutes. Its test step grew from 19m13s to 51m07s in a month while every other
test-bearing job stayed flat. The cause is one line:
build-and-testandjazzy-testboth skip
ros2_medkit_graph_watchdog, whose suite is 24 minutes on its own, andcoverageskips onlyros2_medkit_opcua. It alone kept running the suite the otherscarved out.
It is now sharded four ways by measured test time per package, with a merge job carrying
the completeness gate, the HTML report and the Codecov upload. A shard's tracefile is
legitimately partial, so no single shard can answer whether the report still describes the
whole workspace.
fail-fastis off deliberately: with it, one failing shard cancels theothers, and a cancelled job cannot be told apart from one killed by its cap.
A step timeout reads as a test failure. A step killed by its cap is marked failed by
the runner, not cancelled, and the in-flight test leaves no result file;
colcon test-resultthen reports one erroring test named<test>.xunit.missing_resultamongthousands that never ran. Over one month, 17 of 25 killed jobs died on a step cap, and
every timeout in this file has been raised at least once. A new step now says so plainly
and warns at 80% of the cap. It detects the overrun by the clock, because a job cannot read
its own step log.
Shared memory was unmeasured and stranded. Container jobs got Docker's default 64 MB of
/dev/shm. Fast DDS puts a 512 KB segment plus port files there per participant, measuredat ~0.65 MB, and a participant killed rather than shut down never gets them back: 15 live
participants take 9.6 MB, and 125 killed ones fill 63 MB of 64. Jobs now ask for 1 GB, and
scripts/test.shreclaims stranded segments locally throughfastdds shm clean.Issue
No issue filed for this one - it came out of investigating a CI failure rather than from a
report. Happy to open one if the history is worth having separately.
Type
Testing
The sharded coverage job only runs on push to main, so it could not execute before landing
there. Two temporary commits widened that condition, the job ran on this branch, and the
condition is restored - the net diff against main carries no temporary condition. Results
from run 33426859082, every job green:
--shm-sizeis accepted and takes effectshm 1.0G 2.1M 1022M 1% /dev/shm, 44 segments at the end of the suite - the first time this has been measured at allcoverage (integration/graph-watchdog/gateway/rest), including the folded-YAML catch-all shardmerging 4 tracefilesOK: all 15 checked C++ package(s) include ROS2MedkitCoverage and reached coverage.infotest step: 1573s elapsed of a 2700s cap (58%); results: 285 written, 0 empty, 0 missing_resultoutcome=successSharding loses no coverage. The last unsharded run on main measured 86.9% at 34537 of
39753 lines; this one measures 86.9% at 34539 of 39753. Identical denominator, so the same
instrumented line universe, and the small difference is ordinary run-to-run variance.
Wall clock. Worst shard
graph-watchdog3046s (51 min), thenintegration2663s,gateway2041s,rest1462s. Against 86-90 minutes serial and a 90 minute cap, the margingoes from twelve seconds to about 39 minutes. The cap stays at 90 in this PR: these are
cold-cache numbers, and the figure to lower it by should come from a few runs of the margin
line rather than from this one.
Also verified outside CI: the sweep reclaimed 5.23 MB of the 5.24 MB stranded by eight
SIGKILLed participants in a lyrical container; the overrun guard was exercised at 4%, 85%
and 100% of a cap and with no result files at all;
lcov -awas checked on two tracefileswhere a line uncovered in one and covered in the other merges to covered.
The guard's first version was wrong and is worth recording: it compared registered tests
against result files and reported missing results on a run where everything passed, because
test_dds_domain_allocationis registered in every package and writes no xunit. It nowchecks what actually goes missing instead.
Checklist