master red since #4527 (Bazel/LTO build): CI-evidence bisect, aes-mbff PSM-0069, and dead UpdateRules automation #4550

master has been red since 2026-09-17. This is a CI-evidence bisect confirming where it broke, plus two findings that are not just threshold drift. Posting it so nobody has to re-derive it — I expect the cause is already suspected, but as far as I can tell it hasn't been pinned down in writing.

A local git bisect isn't practical here (each step means rebuilding OpenROAD and running 62 designs, and the builds aren't reproducible anyway), so this is bisected from Jenkins build artifacts.

Bisect

Jenkins pins the ORFS SHA per build, so the builds themselves are the bisect steps:

bc5af8cbd..3000fc797 contains nothing but #4527's own commits:

3000fc797 Merge pull request #4527 from vvbandeira/openroad-bazel
f0a810dee Merge branch 'master' into openroad-bazel
84d3ba1eb fix(build): keep backslashes in --openroad-args
e7d41e0d4 build: enable LTO for the Bazel OpenROAD build
af01027ac fix(docker): include OpenROAD version in dev tags
4e30d84a9 build: install Bazel in the dev image, not the builder stage
bba680a85 build: install OpenROAD with Bazel

The single failure at #1601 is asap7/mock-alu, cts__timing__setup__ws -289.509 >= -289 — 0.18% over, ordinary drift, unrelated.

This is an interaction, which is why neither PR's own CI caught it

#4527's own merge build, PR-4527-merge #10, tested the same #4527 content (head f0a810dee, LTO included) but merged onto pre-#4533 master (fbc8fa48d). It showed 8 failing designs — matching the QoR bot's "62 designs checked — 8 with regression(s)" on that PR — and aes-mbff asap7 passed all 27 rules there.

So:

Neither change alone produces the current state. #4527 is the dominant factor, but the aes-mbff hard failure and 7 of the 13 designs only appear with both.

Caveat worth stating: #4533 also carried an OpenROAD submodule bump of 145 commits (90e29809c..60807181d, including the gpl convergence fixes), so "#4533" here means that whole bundle, not only the nonPlaceArea change.

aes-mbff asap7 — a hard error, not a threshold

This one cannot be fixed by regenerating rules. From logs/asap7/aes-mbff/base/6_report.log:

[WARNING PSM-0043] Shorted shape on net VDD to terminal _tray_size4_1108/CLK
    (net clknet_leaf_3_clk) at (29.583um, 35.893um) - (29.589um, 35.927um), layer: M3.
[ERROR PSM-0069] Check connectivity failed on VDD.
Error: final_report.tcl, 7 PSM-0069

A clock-leaf pin on a multi-bit flop tray shorting to VDD on M3. Clean at #1600 and #1601, and clean in PR-4527-merge #10.

The 13 failing designs (master #1605)

Most are sub-1% and are plain rules drift. These four are large enough that I'd want them looked at rather than loosened:

• gt2n/aes — GR/finish TNS -585 → -818 (+40%), WS -75.2 → -83.6
• asap7/cva6 — finish TNS -851 → -1048 (+23%)
• sky130hd/chameleon — finish TNS -5.01 → -8.98, GR antenna diodes 100 → 139
• sky130hd/microwatt — DRT antenna violating nets 81 → 96, diodes 1463 → 1608

If enabling LTO on the Bazel build is what moved these, it's worth knowing whether that's acceptable before it's baked into new goldens.

Blast radius

Every open PR inherits this through pr-merge: #4546, #4547 and #4535 are all red for this reason alone and cannot be landed.

The UpdateRules automation is dead

This is probably why the tree is still red rather than fixed in an afternoon.

.github/workflows/github-actions-update-rules.yml (the set-new-golden repository_dispatch handler) and .github/workflows/github-actions-manual-update-rules.yml both run:

python flow/util/updateRules.py --keyFile ... --apiURL ... --commitSHA ...

flow/util/updateRules.py was deleted on 2025-02-25 in 63cfaa332 "util: remove updateRules, not used". Consequences:

• the dispatch-driven workflow has zero runs, ever;
• the manual one last ran 2026-04-05 and failed;
• nothing in the repo emits the set-new-golden dispatch any more;
• github-actions-cron-update-OR.yml still applies the UpdateRules label to the nightly submodule-bump PR, where it now does nothing.

So applying the UpdateRules label to a PR has no effect. Either the script should come back or those two workflows and the label should go.

Regenerating the rules without a local run

Since the flow isn't reproducible locally, the goldens can be taken straight from CI — genRuleFile.py only reads metadata.json and never runs the flow:

B=https://jenkins.openroad.tools/job/OpenROAD-flow-scripts-Public/job/master/1605/artifact/reports
# note the report dir is nangate45/bp_fe while the design dir is nangate45/bp_fe_top
curl -sSL -o flow/reports/<platform>/<design>/base/metadata.json \
     $B/<platform>/<design>/base/metadata.json
make DESIGN_CONFIG=./designs/<platform>/<design>/config.mk update_rules

update_rules is genRuleFile.py --failing --tighten, so it only moves rules that actually fail and tightens ones that have gone slack.

That fixes the 13 metadata-check failures. It does nothing for aes-mbff, which needs a real fix or a revert.

🤖 Generated with Claude Code