Repository navigation
Gate make metadata on the QoR dashboard and retire the rules files - #4552
Conversation
The committed rules-<variant>.json files hold absolute limits. Someone must regenerate them by hand after each accepted QoR change. The QoR dashboard already holds one rule per metric and design and compares a run against the latest master build. Jenkins has gated on that check since 2026-09-14, so the rules files duplicate the work and drift. `make metadata-check` now runs checkQorMetrics.py on $(REPORTS_DIR)/metadata.json. The script gets a single-run mode (--metadata, --only-prefix, --strict) and ends its log with `QoR check: PASS|FAIL|INCONCLUSIVE`. Only a failed rule returns exit code 1. An unreachable dashboard or a design without a baseline does not fail the target unless --strict is set. `make metadata-synth` sends only the synth__ and constraints__ metrics. checkMetadata.py, update_rules, and the rules files stay for bazel-orfs and for `make metadata-check-rules`. Remove them after bazel-orfs stops reading them. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
The dashboard check in `make metadata` replaced the rules-file check. Nothing in this repository read the rules files, checkMetadata.py, genRuleFile.py, genAllRuleFiles.sh, or the RULES_JSON variable any more. synDashboard.py only parsed checkMetadata.py output. Two rule sets would drift apart again, so all of them are removed. The targets update_ok, update_rules, and update_rules_force stay as stubs. They print a deprecation warning and exit 0, so an old wrapper script keeps running. Setting RULES_JSON prints a make warning. BREAKING CHANGE: bazel-orfs marks a design as CI-tested when rules-base.json exists. Its test targets run checkMetadata.py and do-update_rules. It needs a new marker and the dashboard check before it bumps its ORFS pin. The Jenkins update-rules job imports genRuleFile.py from ORFS master and must be retired. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
Three review findings on the dashboard check. The script changes to flow/ before it reads anything. The relative path from $(REPORTS_DIR) did not resolve for an out-of-tree WORK_HOME, so an existing metadata.json was reported as missing. The make targets now pass an absolute path. The script resolves --metadata against the caller's directory. A malformed metadata.json, or one without a numeric metric, passed as INCONCLUSIVE with exit 0. It is a local input error. It now has its own status, prints an [ERROR] line, ends with `QoR check: ERROR`, and exits 1. An unreachable dashboard printed an [ERROR] line. genReport.py counts [ERROR] lines in metadata-check.log as a failed design, although the check exits 0. That line is now a [WARN]. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
A file that is not valid UTF-8 raised UnicodeDecodeError past the metadata handler. The check reported it as a dashboard error and passed as INCONCLUSIVE with exit 0. It is a local input error, like malformed JSON. It now ends with `QoR check: ERROR` and exits 1. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
There was a problem hiding this comment.
Code Review
This pull request transitions the Quality of Results (QoR) gating system from a local rules-file comparison (using rules-base.json files and checkMetadata.py) to an online QoR dashboard-based check using checkQorMetrics.py. It removes all local design rules files, the rule generation script (genRuleFile.py), and the local syn-dashboard script. The documentation, Makefiles, and variables are updated to support this new dashboard check, including legacy stubs in utils.mk to maintain backward compatibility for deprecated targets. I have no feedback to provide as there are no review comments.
|
TL;DR: the verdict is still one rule per metric per design against a What a Pareto front is, and why a per-metric rule cannot see one. A flow Timing is a percent of the minimum clock period, never a percent of WNS. Timing closure is not an absolute. The SDC period is a number someone Judge the point, not the metric. The response this PR parses already
Start where Kahng started. Kahng & Mantik, Measurement of Inherent Noise in Roll up across designs to one PR verdict: better, worse, trade, or within Toy version in bazel-orfs, ready to lift: Prompt-sized spec for whoever picks it up: in |
🔍 QoR checkMetrics reflect the PR merge build — i.e. what will land on the target branch. Commit 59 design(s) checked — 0 with regression(s), 1 without a comparable baseline. |
|
@codex review |
flow/util/genReportTable.py compared each run's metadata.json against a committed metadata-<variant>-ok.json to color a QoR table. The QoR dashboard check in `make metadata` is now the gate, and the golden files are stale, so drop the table, the golden json files, and `update_metadata` (now a deprecation stub), and update the docs. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
The OpenROAD bump PR carried the [ci:rules-tighten] tag in its body and the UpdateRules label. The Jenkins update-rules job polls open PRs for that label and rewrites designs/<platform>/<design>/rules-<variant>.json from the dashboard into the PR. The rules files are removed, so that job would add them back on the next bump. The job itself can be retired in the CI repository. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
uploadMetadata.py read designs/<platform>/<design>/rules-<variant>.json and sent it as a `rules` field in every design record. The rules files are removed, so the field was an empty object on every upload. The dashboard reads that field only to attach per-value thresholds for its build pages, and tolerates a missing key. Rule policy lives in the dashboard rule configs. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
1266b85
into
The-OpenROAD-Project:master
|
@oharboe |
Thanks for reading my comments, yes, orthogonalt to the narrower concern of this PR. I will leave it to you to open an issue if it is congruent with your plans |
Summary
make metadata-checknow sendsmetadata.jsonto the QoR dashboard(https://dashboard.precisioninno.com) and takes its verdict. The committed
rules-<variant>.jsonfiles,checkMetadata.py,genRuleFile.py,genAllRuleFiles.sh,synDashboard.py, and theRULES_JSONvariable areremoved.
Why
The rules files held absolute limits. Someone regenerated them by hand after
every accepted QoR change. The dashboard already holds one rule per metric
and design and compares a run against the latest
masterbuild. Thebaseline moves with
master, so a merged change becomes the new baselinewithout a file update. CI has gated on the dashboard check since September
2026. Two rule sets drift apart.
What changed
flow/util/checkQorMetrics.pygets a single-run mode (--metadata,-p,-d,--variant), a metric filter (--only-prefix), and exit codes. Thereport ends with
QoR check: PASS,FAIL,INCONCLUSIVE, orERROR.FAILmeans a dashboard rule failed.ERRORmeansmetadata.jsonismissing, malformed, not UTF-8, or has no numeric metric. Both exit 1.
INCONCLUSIVEmeans no baseline, nothing evaluated, or the dashboard wasnot reached. It exits 0 with
[WARN]lines.--strictreturns 2, or 3when the service was not reached.
reports/is unchanged.make metadata-checkandmake metadata-check-synthcall that script.make metadata-synthsends only thesynth__andconstraints__metrics.update_ok,update_rules,update_rules_force, and their helper targetsare stubs. They print a deprecation warning and exit 0. Setting
RULES_JSONprints a make warning.
DASHBOARD_API_KEYgrants access to private platforms.DASHBOARD_JOB_NAMEselects the baseline pipeline. The default is
OpenROAD-flow-scripts-Public.docs/contrib/Metrics.mdis rewritten.flow/README.md,flow/designs/README.md,docs/user/FlowVariables.md, and the flowtutorial are updated.
Breaking change
bazel-orfs marks a design as CI-tested when
rules-base.jsonexists. Itstest targets run
checkMetadata.py, andorfs_update_rulesrunsdo-update_rules. After it bumps its ORFS pin, every generated test targetbecomes manual and the update targets do nothing. It needs a new marker and
the dashboard check first.
Downstream design repositories with their own
rules-base.jsonhit thestubs. Their files are dead weight.
A log classifier that matched
All metadata rules passedmust matchQoR check:instead.Testing
checkQorMetrics.py --metadataagainst the live dashboard with asap7/gcdmetrics from a
masterbuild. Full set PASS.synth__andconstraints__subset PASS. Unknown design INCONCLUSIVE, exit 2 with
--strict.--metadatapath from another directory. Out-of-treeWORK_HOMEdry run shows an absolute path.
ERROR, exit 1.[WARN]line, exit 0. Exit 3 with--strict.make update_rulesandmake RULES_JSON=x update_okprint the warningsand exit 0.
variables.jsonandFlowVariables.mdregenerated with the repo scripts.