Repository navigation
qor-check: name each failed rule in the report summary - #4631
vvbandeira wants to merge 1 commit into
Conversation
The report summary that CI prints at the end of a build showed only counts for a failed QoR check, such as "2 failed". genReport.py copies only the [ERROR] lines of metadata-check.log into the summary, and the failed rules were in a table without that prefix. To find the values, a reader had to open the log of each design. Now each failed rule has its own [ERROR] line with the current value, the baseline value, the limit, and the delta. The run headline and the "Failing runs" line are now [INFO], so "Found N metrics failures" in the summary is the number of failed rules. The headline stays [ERROR] when the dashboard fails a run but names no metric. The table now shows only with --verbose, because the [ERROR] lines hold the same values. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
There was a problem hiding this comment.
Code Review
This pull request updates the QoR metrics checking flow to write individual [ERROR] lines for each failed rule in metadata-check.log, allowing genReport.py to copy them into the report summary. It also introduces a comprehensive test suite to verify this behavior. The review feedback suggests optimizing the list comprehension for shown in checkQorMetrics.py to only execute when verbose is enabled, preventing unnecessary computation during typical CI runs.
| ) | ||
|
|
||
| if shown: | ||
| shown = [m for m in metrics if m.get("status") != "missing_from_baseline"] |
There was a problem hiding this comment.
To avoid performing an unnecessary list comprehension over all metrics when verbose is False (which is the default in typical CI runs), we can conditionally evaluate the list comprehension only when verbose is True.
| shown = [m for m in metrics if m.get("status") != "missing_from_baseline"] | |
| shown = [m for m in metrics if m.get("status") != "missing_from_baseline"] if verbose else [] |
🔍 QoR checkMetrics reflect the PR merge build — i.e. what will land on the target branch. Commit 62 design(s) checked — 0 with regression(s), 0 without a comparable baseline. |
Problem
When the QoR check fails, the report summary at the end of a CI build shows
only counts:
To see which metrics failed, and by how much, you must open the log of each
design. Also, "Found 2 metrics failures" counts these two lines, not the
5 failed rules.
The cause:
genReport.pycopies only the[ERROR]lines ofmetadata-check.logintoreport-summary.log.checkQorMetrics.pyprintedthe failed rules in a table, and the table rows have no
[ERROR]prefix.Change
checkQorMetrics.pyprints one[ERROR]line for each failed rule, withthe current value, the baseline value, the limit, and the delta.
Failing runsline are now[INFO]. Thus thesummary count is the number of failed rules. If the dashboard fails a run
but names no metric, the headline stays
[ERROR], so the failure stillshows.
--verbose(all metrics). Without--verbose, it repeated the[ERROR]lines.docs/contrib/Metrics.mddescribes the new lines.The summary now reads:
The last line of
metadata-check.log(QoR check: ...) and the exitstatus do not change.
Tests
flow/test/test_checkQorMetrics_report.pychecks the new lines. It also runsgenReport.py -sv(the CI command) on a small flow tree and checks that thesummary shows the failed rules with their values. 4 of its 6 tests fail on
the current
master.