Skip to content

Gate make metadata on the QoR dashboard and retire the rules files - #4552

Merged
vvbandeira merged 7 commits into
The-OpenROAD-Project:masterfrom
vvbandeira:palantir-metadata-check
Sep 23, 2026
Merged

vvbandeira merged 7 commits into
The-OpenROAD-Project:masterfrom
vvbandeira:palantir-metadata-check

Conversation

@vvbandeira

Copy link
Copy Markdown
Member

Summary

make metadata-check now sends metadata.json to the QoR dashboard
(https://dashboard.precisioninno.com) and takes its verdict. The committed
rules-<variant>.json files, checkMetadata.py, genRuleFile.py,
genAllRuleFiles.sh, synDashboard.py, and the RULES_JSON variable are
removed.

Why

The rules files held absolute limits. Someone regenerated them by hand after
every accepted QoR change. The dashboard already holds one rule per metric
and design and compares a run against the latest master build. The
baseline moves with master, so a merged change becomes the new baseline
without a file update. CI has gated on the dashboard check since September
2026. Two rule sets drift apart.

What changed

  • flow/util/checkQorMetrics.py gets a single-run mode (--metadata, -p,
    -d, --variant), a metric filter (--only-prefix), and exit codes. The
    report ends with QoR check: PASS, FAIL, INCONCLUSIVE, or ERROR.
  • FAIL means a dashboard rule failed. ERROR means metadata.json is
    missing, malformed, not UTF-8, or has no numeric metric. Both exit 1.
  • INCONCLUSIVE means no baseline, nothing evaluated, or the dashboard was
    not reached. It exits 0 with [WARN] lines. --strict returns 2, or 3
    when the service was not reached.
  • The sweep mode over reports/ is unchanged.
  • make metadata-check and make metadata-check-synth call that script.
    make metadata-synth sends only the synth__ and constraints__ metrics.
  • update_ok, update_rules, update_rules_force, and their helper targets
    are stubs. They print a deprecation warning and exit 0. Setting RULES_JSON
    prints a make warning.
  • DASHBOARD_API_KEY grants access to private platforms. DASHBOARD_JOB_NAME
    selects the baseline pipeline. The default is OpenROAD-flow-scripts-Public.
  • docs/contrib/Metrics.md is rewritten. flow/README.md,
    flow/designs/README.md, docs/user/FlowVariables.md, and the flow
    tutorial are updated.

Breaking change

bazel-orfs marks a design as CI-tested when rules-base.json exists. Its
test targets run checkMetadata.py, and orfs_update_rules runs
do-update_rules. After it bumps its ORFS pin, every generated test target
becomes manual and the update targets do nothing. It needs a new marker and
the dashboard check first.

Downstream design repositories with their own rules-base.json hit the
stubs. Their files are dead weight.

A log classifier that matched All metadata rules passed must match
QoR check: instead.

Testing

  • checkQorMetrics.py --metadata against the live dashboard with asap7/gcd
    metrics from a master build. Full set PASS. synth__ and constraints__
    subset PASS. Unknown design INCONCLUSIVE, exit 2 with --strict.
  • Relative --metadata path from another directory. Out-of-tree WORK_HOME
    dry run shows an absolute path.
  • Malformed JSON, non-object, no numeric metric, non-UTF-8. ERROR, exit 1.
  • Unreachable dashboard. [WARN] line, exit 0. Exit 3 with --strict.
  • make update_rules and make RULES_JSON=x update_ok print the warnings
    and exit 0.
  • variables.json and FlowVariables.md regenerated with the repo scripts.

The committed rules-<variant>.json files hold absolute limits. Someone
must regenerate them by hand after each accepted QoR change. The QoR
dashboard already holds one rule per metric and design and compares a
run against the latest master build. Jenkins has gated on that check
since 2026-09-14, so the rules files duplicate the work and drift.

`make metadata-check` now runs checkQorMetrics.py on
$(REPORTS_DIR)/metadata.json. The script gets a single-run mode
(--metadata, --only-prefix, --strict) and ends its log with
`QoR check: PASS|FAIL|INCONCLUSIVE`. Only a failed rule returns exit
code 1. An unreachable dashboard or a design without a baseline does
not fail the target unless --strict is set. `make metadata-synth`
sends only the synth__ and constraints__ metrics.

checkMetadata.py, update_rules, and the rules files stay for
bazel-orfs and for `make metadata-check-rules`. Remove them after
bazel-orfs stops reading them.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
The dashboard check in `make metadata` replaced the rules-file check.
Nothing in this repository read the rules files, checkMetadata.py,
genRuleFile.py, genAllRuleFiles.sh, or the RULES_JSON variable any
more. synDashboard.py only parsed checkMetadata.py output. Two rule
sets would drift apart again, so all of them are removed.

The targets update_ok, update_rules, and update_rules_force stay as
stubs. They print a deprecation warning and exit 0, so an old wrapper
script keeps running. Setting RULES_JSON prints a make warning.

BREAKING CHANGE: bazel-orfs marks a design as CI-tested when
rules-base.json exists. Its test targets run checkMetadata.py and
do-update_rules. It needs a new marker and the dashboard check before
it bumps its ORFS pin. The Jenkins update-rules job imports
genRuleFile.py from ORFS master and must be retired.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
Three review findings on the dashboard check.

The script changes to flow/ before it reads anything. The relative
path from $(REPORTS_DIR) did not resolve for an out-of-tree WORK_HOME,
so an existing metadata.json was reported as missing. The make targets
now pass an absolute path. The script resolves --metadata against the
caller's directory.

A malformed metadata.json, or one without a numeric metric, passed as
INCONCLUSIVE with exit 0. It is a local input error. It now has its own
status, prints an [ERROR] line, ends with `QoR check: ERROR`, and exits
1.

An unreachable dashboard printed an [ERROR] line. genReport.py counts
[ERROR] lines in metadata-check.log as a failed design, although the
check exits 0. That line is now a [WARN].

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
A file that is not valid UTF-8 raised UnicodeDecodeError past the
metadata handler. The check reported it as a dashboard error and passed
as INCONCLUSIVE with exit 0. It is a local input error, like malformed
JSON. It now ends with `QoR check: ERROR` and exits 1.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request transitions the Quality of Results (QoR) gating system from a local rules-file comparison (using rules-base.json files and checkMetadata.py) to an online QoR dashboard-based check using checkQorMetrics.py. It removes all local design rules files, the rule generation script (genRuleFile.py), and the local syn-dashboard script. The documentation, Makefiles, and variables are updated to support this new dashboard check, including legacy stubs in utils.mk to maintain backward compatibility for deprecated targets. I have no feedback to provide as there are no review comments.

@oharboe

oharboe commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

TL;DR: the verdict is still one rule per metric per design against a
hand-set tolerance. That shape cannot express a trade, cannot tell a
regression from a bad draw, and measures timing on an axis (WNS) that
manufactures distorted percentages. The dashboard already returns everything
needed to fix all three, and the fix is one file.

What a Pareto front is, and why a per-metric rule cannot see one. A flow
result is a point in a few dimensions: minimum clock period, area, power. A
point dominates another when it is at least as good on every axis and better
on one. The set of undominated points is the Pareto front, and the flow's job
is to move it. A change that shrinks area 8% and gives up 1% of period did not
regress; it moved along the front, from one undominated point to another.
That is a trade, and it is the normal way a flow improves. A rule that looks
at one metric at a time sees only the axis that got worse, fails it, and says
nothing about the axis that paid for it. Read literally it is a regression;
read honestly it is a different operating point. The two are indistinguishable
from a per-metric threshold, whatever its tolerance.

Timing is a percent of the minimum clock period, never a percent of WNS.
WNS is a signed number that lives near zero, so a fractional change in it is
noise dressed as precision: +5.8 ps to +0.9 ps reads as "lost 85%" of nothing,
while −274 ps to −280 ps reads as "lost 2%" of a great deal. The stable
quantity is the minimum clock period, clock − WNS: it is of order the clock,
a fractional change in it means what it says, and a noise band on it is a
fraction of the clock. ORFS already records the register-to-register version
as finish__timing__fmax; it is not the same number as clock − WNS
(asap7/gcd: 329 vs 351 ps, WNS includes the I/O paths) and both belong in the
record.

Timing closure is not an absolute. The SDC period is a number someone
picked, so crossing zero slack is not a change in kind; it is a change of a
percent or so in the achieved period, and it is judged like any other point
on that axis. This matters because the current rules treat zero as special
in the wrong direction: genRuleFile.py takes min(ws, 0) before padding,
and the dashboard's PeriodPadding mode does the same. Checked live against
asap7/ibex (master WNS +28 ps, 1000 ps clock): the threshold is −50 ps, so a
run may lose 78 ps, 8% of its minimum period, and pass. On the period axis
that is a large regression; on a WNS rule anchored at zero it is invisible.
The optimizers also stop at slack zero, so a design with positive WNS sits
where nothing was ever traded for it. All the more reason to read the
period, not the sign.

Judge the point, not the metric. The response this PR parses already
carries baseValue, targetValue and clockPeriod per metric, so the verdict
can be computed client-side with no dashboard change:

  • hard constraints (DRC, placement violations, antenna) fail outright; these
    are not axes you are allowed to spend;
  • otherwise the run fails only if dominated: every axis worse-or-tied and one
    worse beyond its noise band. Axes: minimum clock period, core area, cell
    area, power. A trade passes and is printed as one.

Start where Kahng started. Kahng & Mantik, Measurement of Inherent Noise in
EDA Tools
(ISQED 2002): meaning-preserving perturbations move results by
amounts comparable to claimed improvements. We have the data to measure that
floor. Every master build is a fresh draw and /api/metric-trends/master
serves the series. For asap7/gcd over 150 daily points the median nonzero
day-to-day step in setup WNS is 1.2 ps, 0.4% of the 310 ps clock (MAD sigma
1.7 ps); the raw sd is 24 ps because it contains the accepted changes (steps
of 72 and 35 ps). Detrend, and the residual is the noise band, per design
and metric, no seed sweep purchased. The tolerance stops being a number
someone typed.

Roll up across designs to one PR verdict: better, worse, trade, or within
noise. One design wobbling inside its band should not fail a PR that improved
ten others.

Toy version in bazel-orfs, ready to lift:
check_pareto.py
(axes, domination, hard list, the WNS-versus-period argument in its header),
check_pareto_test.py
(17 cases), and the reasoning in
docs/estimate.md
("A flow result is a draw", "The audit you can afford"). One correction to
carry over: the toy still fails lost closure outright. Drop that; the period
axis already prices it.

Prompt-sized spec for whoever picks it up: in checkQorMetrics.py, (1) add
tie_band(platform, design, metric) = 1.4826 × MAD of nonzero first
differences of metric-trends/master, falling back to the dashboard
threshold when fewer than 20 steps exist; (2) in render_run, synthesise the
minimum clock period from clockPeriod and WNS, classify each axis
improved/tied/regressed as a percent of that period against its band, and set
the run status to hard-fail, dominated, trade, improved or noise; (3) in
verdict, FAIL only on hard-fail or dominated, and print the trades. No
dashboard change needed.

@openroad-ci

openroad-ci commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

🔍 QoR check

Metrics reflect the PR merge build — i.e. what will land on the target branch.

Commit 9182e42 · Jenkins build #3 · Baseline: build · View build on dashboard

59 design(s) checked — 0 with regression(s), 1 without a comparable baseline.

@vvbandeira

Copy link
Copy Markdown
Member Author

@codex review

flow/util/genReportTable.py compared each run's metadata.json against a
committed metadata-<variant>-ok.json to color a QoR table. The QoR
dashboard check in `make metadata` is now the gate, and the golden
files are stale, so drop the table, the golden json files, and
`update_metadata` (now a deprecation stub), and update the docs.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
The OpenROAD bump PR carried the [ci:rules-tighten] tag in its body and
the UpdateRules label. The Jenkins update-rules job polls open PRs for
that label and rewrites designs/<platform>/<design>/rules-<variant>.json
from the dashboard into the PR. The rules files are removed, so that
job would add them back on the next bump. The job itself can be retired
in the CI repository.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
uploadMetadata.py read designs/<platform>/<design>/rules-<variant>.json
and sent it as a `rules` field in every design record. The rules files
are removed, so the field was an empty object on every upload. The
dashboard reads that field only to attach per-value thresholds for its
build pages, and tolerates a missing key. Rule policy lives in the
dashboard rule configs.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
@vvbandeira
vvbandeira merged commit 1266b85 into The-OpenROAD-Project:master Sep 23, 2026
11 of 13 checks passed
@vvbandeira

Copy link
Copy Markdown
Member Author

@oharboe
This seems like a longer discussion. We should start an issue or GitHub Discussion to address the points you raised.

@oharboe

oharboe commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

@oharboe This seems like a longer discussion. We should start an issue or GitHub Discussion to address the points you raised.

Thanks for reading my comments, yes, orthogonalt to the narrower concern of this PR.

I will leave it to you to open an issue if it is congruent with your plans

@vvbandeira
vvbandeira deleted the palantir-metadata-check branch September 25, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants