Repository navigation
genReport: sort the CI summary by failure severity - #4789
Open
vvbandeira wants to merge 3 commits into
Open
vvbandeira wants to merge 3 commits into
vvbandeira wants to merge 3 commits into
Conversation
The CI summary listed each failing design as a separate block. It counted error lines instead of regressed metrics and printed the same baseline URL for each design. Readers could not see which failures were most severe, or which metric regressed in many designs. The summary now starts with one row per design, sorted by severity: flow stopped, flow error, Calibre DRC, QoR regression, then warnings. Grouped details follow. QoR results that did not give a verdict were not shown before. The same applies to missing metadata logs. These items now show under "Needs attention" and do not change the pass or fail status. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
At -vv, a design with notes printed twice. Its notes now show only under "Needs attention". A "Reason:" line after a PASS or FAIL verdict created a note. It is now kept only for a check without a verdict. The metric count changed the order of non-QoR rows. It now orders only QoR rows. At -v, errors in metadata-generate.log for a passing design gave only a count. The summary now shows the error text. Tests now cover each verbose level, the per-design reports, the local QoR check format, and different baselines. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
The script dropped the "Failing runs" line before it set the design status. A design with only that [ERROR] line then passed. The script now hides that line only in the text, so the old status rule applies. The CI writes FAIL as "[ERROR] Dashboard QoR check: FAIL". The parser did not match that prefix, so a later "Reason:" line could join an earlier INCONCLUSIVE note. The script printed the metadata-generate errors of a passing design two times: once as a note and once in its details. They now print once. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors the report generation utility genReport.py and its test suite test_genReport.py. The changes introduce structured log parsing, improved error and warning classification, support for multiple verbosity levels, and robust handling of non-UTF-8 log files. The test suite has been updated and expanded to cover these new reporting features and edge cases. There are no review comments, so I have no feedback to provide.
Member
🔍 QoR checkMetrics reflect the PR merge build — i.e. what will land on the target branch. Commit 62 design(s) checked — 0 with regression(s), 0 without a comparable baseline. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
flow/util/genReport.py -svwrites the summary at the end of each CI run.It printed one block per failing design, in name order.
Each block counted
[ERROR]lines instead of regressed metrics, and each block repeated the QoR baseline URL.On master build 1736, every design showed "Found 1 metrics failures", but the 12 designs had 25 regressed metrics between them.
The report also hid some results:
[WARN]lines from the QoR check, theINCONCLUSIVEverdict, and passing designs with DRC violations under the limit, which printed without a design name.The summary now starts with one row per design, sorted by severity:
STOPERR[ERROR]in a stage logDRCQORWARNDetail sections per category follow the table, then a "Needs attention" section.
The rules that make a design pass or fail did not change.
The output file names that Jenkins archives did not change.
Example from the build 1736 data (shortened):
Other changes:
[ERROR]line shows the end of its last log, correctly indented, and names a kill signal when it finds one.platform/design/variant, the same identity that the QoR check uses.-s) creates the reports directory when a flow stopped before it existed.