Skip to content

genReport: sort the CI summary by failure severity - #4789

Open
vvbandeira wants to merge 3 commits into
The-OpenROAD-Project:masterfrom
vvbandeira:genreport-summary
Open

vvbandeira wants to merge 3 commits into
The-OpenROAD-Project:masterfrom
vvbandeira:genreport-summary

Conversation

@vvbandeira

Copy link
Copy Markdown
Member

flow/util/genReport.py -sv writes the summary at the end of each CI run.
It printed one block per failing design, in name order.
Each block counted [ERROR] lines instead of regressed metrics, and each block repeated the QoR baseline URL.
On master build 1736, every design showed "Found 1 metrics failures", but the 12 designs had 25 regressed metrics between them.
The report also hid some results: [WARN] lines from the QoR check, the INCONCLUSIVE verdict, and passing designs with DRC violations under the limit, which printed without a design name.

The summary now starts with one row per design, sorted by severity:

Code Meaning
STOP the flow did not reach the last stage
ERR the flow finished with an [ERROR] in a stage log
DRC Calibre DRC failed
QOR QoR regression; designs with more regressed metrics first
WARN the design passes, but a note needs attention

Detail sections per category follow the table, then a "Needs attention" section.
The rules that make a design pass or fail did not change.
The output file names that Jenkins archives did not change.

Example from the build 1736 data (shortened):

============================================================
SUMMARY: 67 designs, 12 failing, 55 passing
  flow stopped 0 | flow error 0 | Calibre DRC 0 | QoR regression 12
  Most regressed: finish__timing__setup__tns (9), globalroute__timing__setup__tns (6), cts__timing__setup__tns (4)
  QoR baseline: https://dashboard.precisioninno.com/builds/35280
============================================================

QOR   sky130hd/chameleon/base                          5 metrics: detailedroute__antenna__violating__nets, ...
QOR   ihp-sg13g2/gcd/base                              3 metrics: cts__timing__setup__tns, ...
...
WARN  asap7/aes-block_aes_rcon/base                    missing metadata-generate.log, metadata-check.log

------------------------------------------------------------
QOR REGRESSION (12)
------------------------------------------------------------
ihp-sg13g2/gcd/base: 3 metrics over the limit
    metric                           current       base          limit
    cts__timing__setup__tns          -5.87544      -4.45276      -5.343312
    finish__timing__setup__tns       -5.66599      -3.87928      -4.655136
    globalroute__timing__setup__tns  -7.59107      -6.19756      -7.437072

Other changes:

  • Each error line shows the log file it came from, and repeated errors print once with a count.
  • A stopped flow with no [ERROR] line shows the end of its last log, correctly indented, and names a kill signal when it finds one.
  • The "Failed to open ..." lines before the banner are replaced by a "missing ..." note on the design.
  • Designs are named platform/design/variant, the same identity that the QoR check uses.
  • The per-design mode (no -s) creates the reports directory when a flow stopped before it existed.

The CI summary listed each failing design as a separate block. It
counted error lines instead of regressed metrics and printed the same
baseline URL for each design. Readers could not see which failures
were most severe, or which metric regressed in many designs.

The summary now starts with one row per design, sorted by severity:
flow stopped, flow error, Calibre DRC, QoR regression, then warnings.
Grouped details follow.

QoR results that did not give a verdict were not shown before. The
same applies to missing metadata logs. These items now show under
"Needs attention" and do not change the pass or fail status.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
At -vv, a design with notes printed twice. Its notes now show only
under "Needs attention".

A "Reason:" line after a PASS or FAIL verdict created a note. It is
now kept only for a check without a verdict.

The metric count changed the order of non-QoR rows. It now orders
only QoR rows.

At -v, errors in metadata-generate.log for a passing design gave only
a count. The summary now shows the error text.

Tests now cover each verbose level, the per-design reports, the local
QoR check format, and different baselines.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
The script dropped the "Failing runs" line before it set the design
status. A design with only that [ERROR] line then passed. The script
now hides that line only in the text, so the old status rule applies.

The CI writes FAIL as "[ERROR] Dashboard QoR check: FAIL". The parser
did not match that prefix, so a later "Reason:" line could join an
earlier INCONCLUSIVE note.

The script printed the metadata-generate errors of a passing design
two times: once as a note and once in its details. They now print once.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the report generation utility genReport.py and its test suite test_genReport.py. The changes introduce structured log parsing, improved error and warning classification, support for multiple verbosity levels, and robust handling of non-UTF-8 log files. The test suite has been updated and expanded to cover these new reporting features and edge cases. There are no review comments, so I have no feedback to provide.

@openroad-ci

Copy link
Copy Markdown
Member

🔍 QoR check

Metrics reflect the PR merge build — i.e. what will land on the target branch.

Commit 5fc5977 · Jenkins build #1 · Baseline: build · View build on dashboard

62 design(s) checked — 0 with regression(s), 0 without a comparable baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants