Skip to content

qor-check: name each failed rule in the report summary - #4631

Closed
vvbandeira wants to merge 1 commit into
The-OpenROAD-Project:masterfrom
vvbandeira:qor-check-failing-metric-lines
Closed

vvbandeira wants to merge 1 commit into
The-OpenROAD-Project:masterfrom
vvbandeira:qor-check-failing-metric-lines

Conversation

@vvbandeira

Copy link
Copy Markdown
Member

Problem

When the QoR check fails, the report summary at the end of a CI build shows
only counts:

sky130hs jpeg (base)
  Flow reached last stage.
  Found 2 metrics failures.
      [ERROR] sky130hs/jpeg/base: FAIL — 20 passed, 5 failed, 0 missing (521 metrics sent)
      [ERROR] Failing runs: sky130hs/jpeg/base

To see which metrics failed, and by how much, you must open the log of each
design. Also, "Found 2 metrics failures" counts these two lines, not the
5 failed rules.

The cause: genReport.py copies only the [ERROR] lines of
metadata-check.log into report-summary.log. checkQorMetrics.py printed
the failed rules in a table, and the table rows have no [ERROR] prefix.

Change

  • checkQorMetrics.py prints one [ERROR] line for each failed rule, with
    the current value, the baseline value, the limit, and the delta.
  • The run headline and the Failing runs line are now [INFO]. Thus the
    summary count is the number of failed rules. If the dashboard fails a run
    but names no metric, the headline stays [ERROR], so the failure still
    shows.
  • The table now shows only with --verbose (all metrics). Without
    --verbose, it repeated the [ERROR] lines.
  • docs/contrib/Metrics.md describes the new lines.

The summary now reads:

sky130hd riscv32i (base)
  Flow reached last stage.
  Found 2 metrics failures.
      [ERROR] sky130hd/riscv32i/base: finish__timing__setup__tns: current -8900.7900, baseline -4650.0500, limit -5580.0600, delta +91.41%
      [ERROR] sky130hd/riscv32i/base: globalroute__timing__setup__tns: current -8899.9500, baseline -4617.6600, limit -5541.1920, delta +92.74%

The last line of metadata-check.log (QoR check: ...) and the exit
status do not change.

Tests

flow/test/test_checkQorMetrics_report.py checks the new lines. It also runs
genReport.py -sv (the CI command) on a small flow tree and checks that the
summary shows the failed rules with their values. 4 of its 6 tests fail on
the current master.

The report summary that CI prints at the end of a build showed only
counts for a failed QoR check, such as "2 failed". genReport.py copies
only the [ERROR] lines of metadata-check.log into the summary, and the
failed rules were in a table without that prefix. To find the values, a
reader had to open the log of each design.

Now each failed rule has its own [ERROR] line with the current value,
the baseline value, the limit, and the delta. The run headline and the
"Failing runs" line are now [INFO], so "Found N metrics failures" in the
summary is the number of failed rules. The headline stays [ERROR] when
the dashboard fails a run but names no metric. The table now shows only
with --verbose, because the [ERROR] lines hold the same values.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the QoR metrics checking flow to write individual [ERROR] lines for each failed rule in metadata-check.log, allowing genReport.py to copy them into the report summary. It also introduces a comprehensive test suite to verify this behavior. The review feedback suggests optimizing the list comprehension for shown in checkQorMetrics.py to only execute when verbose is enabled, preventing unnecessary computation during typical CI runs.

)

if shown:
shown = [m for m in metrics if m.get("status") != "missing_from_baseline"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To avoid performing an unnecessary list comprehension over all metrics when verbose is False (which is the default in typical CI runs), we can conditionally evaluate the list comprehension only when verbose is True.

Suggested change
shown = [m for m in metrics if m.get("status") != "missing_from_baseline"]
shown = [m for m in metrics if m.get("status") != "missing_from_baseline"] if verbose else []

@vvbandeira vvbandeira closed this Oct 5, 2026
@openroad-ci

Copy link
Copy Markdown
Member

🔍 QoR check

Metrics reflect the PR merge build — i.e. what will land on the target branch.

Commit 36fa833 · Jenkins build #1 · Baseline: build · View build on dashboard

62 design(s) checked — 0 with regression(s), 0 without a comparable baseline.

@vvbandeira
vvbandeira deleted the qor-check-failing-metric-lines branch October 7, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants