Skip to content

Report how each design's flow run ended in the metadata upload - #4819

Merged
maliberty merged 2 commits into
The-OpenROAD-Project:masterfrom
The-OpenROAD-Project-staging:upload-design-outcomes
Nov 11, 2026
Merged

maliberty merged 2 commits into
The-OpenROAD-Project:masterfrom
The-OpenROAD-Project-staging:upload-design-outcomes

Conversation

@maliberty

Copy link
Copy Markdown
Member

Summary

uploadMetadata.py now reports how each design's flow run ended. A design that failed before writing metadata.json is no longer silently dropped.

 for reports/<platform>/<design>/<variant>/
-  no metadata.json → [WARN] skip
+  outcome = read_flow_outcome(flow-status.json, logs/<platform>/<design>/<variant>/)
+  no metadata.json and no outcome → [WARN] skip              # as before
+  record = {platform, design, variant, [metrics], [outcome]}
"outcome": {
  "status": "error",
  "stage": "grt",
  "elapsed_minutes": 41.5,
  "limit_minutes": 240,
  "message": "[ERROR GRT-0116] Global routing failed."
}

How the outcome is derived:

  • status:
    • timeout when timed_out is set.
    • passed on exit code 0.
    • crash when the last stage log shows Signal N received, Segmentation fault or core dumped.
    • error otherwise.
  • stage: the highest-numbered stage log, mapped to synth, floorplan, place, cts, grt, route or finish.
  • message: the first [ERROR XXX-NNNN], ERROR: or Error: line, searching the logs from the last stage back, truncated to 1024 characters.

flow-status.json is written next to each design's reports by the CI job (jenkins-ci orfsRunFlow), with exit_code, timed_out, elapsed_minutes and limit_minutes. Designs without that file are reported exactly as before.

On the per-design fallback for oversized payloads, outcome rides at the message root. An outcome-only design there names itself with run__flow__platform, run__flow__design, run__flow__variant and run__flow__generate_date, the last one from the build time.

Evidence

  • Before: a design that timed out produced [WARN] No data file for nangate45 ibex base. and no record.
    After: new cases in flow/test/test_uploadMetadata.py:
    • test_timed_out_design_without_metrics_reports_its_outcome: the record has no metrics and an outcome of timeout, stage grt, elapsed and limit.
    • test_failed_design_reports_stage_and_first_error_line: error, grt, [ERROR GRT-0116] Global routing failed.
    • test_yosys_error_line_is_reported: synth, ERROR: Module ...
    • test_signal_in_last_stage_log_is_a_crash: crash, place.
    • test_passing_design_keeps_its_metrics_and_reports_passed
    • test_long_error_line_is_truncated
    • test_unreadable_status_without_metrics_skips_the_design: as before, with a warning.
    • test_outcome_only_design_names_itself_on_the_v1_fallback
  • python3 -m unittest test.test_uploadMetadata: 43 tests OK. black 26.10.1: clean.

Merge Danger

Door: two-way

A revert stops sending outcome and outcome-only records.

Blast Radius: metrics upload

The payload changes shape for every pipeline that has the CI change: design records gain outcome, and failed designs appear as outcome-only records. That is the reason for the hold above. genReport.py and genReportTable.py are unaffected: one walks logs/, the other only collects images under reports/.

uploadMetadata.py now reads flow-status.json, which the CI job writes
next to each design's reports with the run's exit code, whether its time
limit stopped it, and the elapsed and limit minutes. Each design record
gains an "outcome" object: passed, error, timeout or crash, with the
stage the run ended in and the first error line found in its stage logs.
A crash is a failed run whose last stage log shows the tool died on a
signal.

A design that failed before writing metadata.json used to be skipped
with a warning. When its run left a status file it is now reported with
only its outcome, so a design that keeps timing out shows up instead of
silently going missing. On the per-design fallback for oversized
payloads, such a record names itself with the run__flow__ keys that
metadata.json would have held. Designs without a status file are
reported exactly as before.

Signed-off-by: Miguel Dalberto Pedro <miguel.pedro@precisioninno.com>
Signed-off-by: Miguel Dalberto Pedro <miguel.pedro@precisioninno.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T23:53:47.149258Z 3ab41e4 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces functionality to report the outcome of flow runs (such as timeouts, crashes, errors, or successful completions) by reading flow-status.json and scanning stage logs, even when the design's metadata file is missing. It also adds comprehensive unit tests to verify these scenarios. The review feedback suggests optimizing the log scanning and stage log retrieval processes, specifically by reducing redundant regex matching in _stage_logs and avoiding unnecessary line-by-line crash scans in older log files to improve performance.

Comment on lines +214 to +225
def _stage_logs(log_dir):
"""The design's stage logs, last step first."""
try:
names = os.listdir(log_dir)
except OSError:
return []

def key(name):
match = STAGE_LOG.match(name)
return (int(match.group(1)), int(match.group(2) or 0), name)

return sorted((n for n in names if STAGE_LOG.match(n)), key=key, reverse=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation of _stage_logs is slightly inefficient because it calls STAGE_LOG.match(n) twice for each matching log name (once in the generator filter and once inside the key function). Additionally, defining the nested key function inside _stage_logs recreates it on every call.\n\nWe can simplify and optimize this by performing a single pass over names, matching once, and sorting the resulting tuples directly.

def _stage_logs(log_dir):\n    \"\"\"The design's stage logs, last step first.\"\"\"\n    try:\n        names = os.listdir(log_dir)\n    except OSError:\n        return []\n\n    matched_logs = []\n    for name in names:\n        match = STAGE_LOG.match(name)\n        if match:\n            matched_logs.append((int(match.group(1)), int(match.group(2) or 0), name))\n\n    matched_logs.sort(reverse=True)\n    return [item[2] for item in matched_logs]

Comment on lines +228 to +245
def _scan_log(path):
"""The first error line of a log and whether it shows a crash."""
error_line = None
crashed = False
try:
with open(path, errors="replace") as f:
for line in f:
if error_line is None:
match = ERROR_LINE.search(line)
if match:
error_line = match.group(0).strip()
if not crashed and CRASH_LINE.search(line):
crashed = True
if error_line is not None and crashed:
break
except OSError:
pass
return error_line, crashed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Scanning the entire log file line-by-line can be very slow for large log files (which can be tens or hundreds of megabytes in VLSI flows). Since we only care about detecting a crash in the last stage log (i == 0), we can avoid scanning for crash lines in older logs (i > 0).\n\nBy adding a scan_for_crash parameter (defaulting to True), we can break out of the loop immediately once the first error line is found when scan_for_crash is False.

def _scan_log(path, scan_for_crash=True):\n    \"\"\"The first error line of a log and whether it shows a crash.\"\"\"\n    error_line = None\n    crashed = False\n    try:\n        with open(path, errors=\"replace\") as f:\n            for line in f:\n                if error_line is None:\n                    match = ERROR_LINE.search(line)\n                    if match:\n                        error_line = match.group(0).strip()\n                if scan_for_crash and not crashed and CRASH_LINE.search(line):\n                    crashed = True\n                if error_line is not None and (not scan_for_crash or crashed):\n                    break\n    except OSError:\n        pass\n    return error_line, crashed

Comment on lines +297 to +303
for i, name in enumerate(logs):
error_line, log_crashed = _scan_log(os.path.join(log_dir, name))
if i == 0:
crashed = log_crashed
if error_line:
message = error_line
break

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Pass scan_for_crash=(i == 0) to _scan_log so that we only scan for crash lines in the last stage log. For older logs (i > 0), this allows _scan_log to stop reading the file immediately once the first error line is found, significantly improving performance.

    for i, name in enumerate(logs):\n        error_line, log_crashed = _scan_log(os.path.join(log_dir, name), scan_for_crash=(i == 0))\n        if i == 0:\n            crashed = log_crashed\n        if error_line:\n            message = error_line\n            break

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ab41e42aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


def key(name):
match = STAGE_LOG.match(name)
return (int(match.group(1)), int(match.group(2) or 0), name)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Respect execution order when selecting the crash-detection log

When final reporting crashes, 6_report.log is ordered behind the successful 6_1_fill.log because its absent substep is converted to zero, even though flow/Makefile:685 makes final reporting depend on fill. read_flow_outcome() only takes the crash flag from logs[0], so a Signal 11 received in 6_report.log is published as error, not crash. The same ordering problem affects 1_synth.log after 1_2_yosys.log. Account for these unnumbered terminal logs when selecting the log that determines the failure type.

Useful? React with 👍 / 👎.

@openroad-ci

Copy link
Copy Markdown
Member

🔍 QoR check

Metrics reflect the PR merge build — i.e. what will land on the target branch.

Commit 34a8b57 · Jenkins build #1 · Baseline: build · View build on dashboard

62 design(s) checked — 0 with regression(s), 0 without a comparable baseline.

oharboe added a commit to oharboe/OpenROAD-flow-scripts that referenced this pull request Nov 11, 2026
uploadMetadata.py and its test came onto this branch with a merge; the
change is unrelated to cts.tcl and is under review on its own as The-OpenROAD-Project#4819.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
@maliberty
maliberty merged commit 05bbba7 into The-OpenROAD-Project:master Nov 11, 2026
13 checks passed
@maliberty
maliberty deleted the upload-design-outcomes branch November 11, 2026 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants