Repository navigation
Conversation
uploadMetadata.py now reads flow-status.json, which the CI job writes next to each design's reports with the run's exit code, whether its time limit stopped it, and the elapsed and limit minutes. Each design record gains an "outcome" object: passed, error, timeout or crash, with the stage the run ended in and the first error line found in its stage logs. A crash is a failed run whose last stage log shows the tool died on a signal. A design that failed before writing metadata.json used to be skipped with a warning. When its run left a status file it is now reported with only its outcome, so a design that keeps timing out shows up instead of silently going missing. On the per-design fallback for oversized payloads, such a record names itself with the run__flow__ keys that metadata.json would have held. Designs without a status file are reported exactly as before. Signed-off-by: Miguel Dalberto Pedro <miguel.pedro@precisioninno.com>
Signed-off-by: Miguel Dalberto Pedro <miguel.pedro@precisioninno.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
Code Review
This pull request introduces functionality to report the outcome of flow runs (such as timeouts, crashes, errors, or successful completions) by reading flow-status.json and scanning stage logs, even when the design's metadata file is missing. It also adds comprehensive unit tests to verify these scenarios. The review feedback suggests optimizing the log scanning and stage log retrieval processes, specifically by reducing redundant regex matching in _stage_logs and avoiding unnecessary line-by-line crash scans in older log files to improve performance.
| def _stage_logs(log_dir): | ||
| """The design's stage logs, last step first.""" | ||
| try: | ||
| names = os.listdir(log_dir) | ||
| except OSError: | ||
| return [] | ||
|
|
||
| def key(name): | ||
| match = STAGE_LOG.match(name) | ||
| return (int(match.group(1)), int(match.group(2) or 0), name) | ||
|
|
||
| return sorted((n for n in names if STAGE_LOG.match(n)), key=key, reverse=True) |
There was a problem hiding this comment.
The current implementation of _stage_logs is slightly inefficient because it calls STAGE_LOG.match(n) twice for each matching log name (once in the generator filter and once inside the key function). Additionally, defining the nested key function inside _stage_logs recreates it on every call.\n\nWe can simplify and optimize this by performing a single pass over names, matching once, and sorting the resulting tuples directly.
def _stage_logs(log_dir):\n \"\"\"The design's stage logs, last step first.\"\"\"\n try:\n names = os.listdir(log_dir)\n except OSError:\n return []\n\n matched_logs = []\n for name in names:\n match = STAGE_LOG.match(name)\n if match:\n matched_logs.append((int(match.group(1)), int(match.group(2) or 0), name))\n\n matched_logs.sort(reverse=True)\n return [item[2] for item in matched_logs]| def _scan_log(path): | ||
| """The first error line of a log and whether it shows a crash.""" | ||
| error_line = None | ||
| crashed = False | ||
| try: | ||
| with open(path, errors="replace") as f: | ||
| for line in f: | ||
| if error_line is None: | ||
| match = ERROR_LINE.search(line) | ||
| if match: | ||
| error_line = match.group(0).strip() | ||
| if not crashed and CRASH_LINE.search(line): | ||
| crashed = True | ||
| if error_line is not None and crashed: | ||
| break | ||
| except OSError: | ||
| pass | ||
| return error_line, crashed |
There was a problem hiding this comment.
Scanning the entire log file line-by-line can be very slow for large log files (which can be tens or hundreds of megabytes in VLSI flows). Since we only care about detecting a crash in the last stage log (i == 0), we can avoid scanning for crash lines in older logs (i > 0).\n\nBy adding a scan_for_crash parameter (defaulting to True), we can break out of the loop immediately once the first error line is found when scan_for_crash is False.
def _scan_log(path, scan_for_crash=True):\n \"\"\"The first error line of a log and whether it shows a crash.\"\"\"\n error_line = None\n crashed = False\n try:\n with open(path, errors=\"replace\") as f:\n for line in f:\n if error_line is None:\n match = ERROR_LINE.search(line)\n if match:\n error_line = match.group(0).strip()\n if scan_for_crash and not crashed and CRASH_LINE.search(line):\n crashed = True\n if error_line is not None and (not scan_for_crash or crashed):\n break\n except OSError:\n pass\n return error_line, crashed| for i, name in enumerate(logs): | ||
| error_line, log_crashed = _scan_log(os.path.join(log_dir, name)) | ||
| if i == 0: | ||
| crashed = log_crashed | ||
| if error_line: | ||
| message = error_line | ||
| break |
There was a problem hiding this comment.
Pass scan_for_crash=(i == 0) to _scan_log so that we only scan for crash lines in the last stage log. For older logs (i > 0), this allows _scan_log to stop reading the file immediately once the first error line is found, significantly improving performance.
for i, name in enumerate(logs):\n error_line, log_crashed = _scan_log(os.path.join(log_dir, name), scan_for_crash=(i == 0))\n if i == 0:\n crashed = log_crashed\n if error_line:\n message = error_line\n breakThere was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3ab41e42aa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| def key(name): | ||
| match = STAGE_LOG.match(name) | ||
| return (int(match.group(1)), int(match.group(2) or 0), name) |
There was a problem hiding this comment.
Respect execution order when selecting the crash-detection log
When final reporting crashes, 6_report.log is ordered behind the successful 6_1_fill.log because its absent substep is converted to zero, even though flow/Makefile:685 makes final reporting depend on fill. read_flow_outcome() only takes the crash flag from logs[0], so a Signal 11 received in 6_report.log is published as error, not crash. The same ordering problem affects 1_synth.log after 1_2_yosys.log. Account for these unnumbered terminal logs when selecting the log that determines the failure type.
Useful? React with 👍 / 👎.
🔍 QoR checkMetrics reflect the PR merge build — i.e. what will land on the target branch. Commit 62 design(s) checked — 0 with regression(s), 0 without a comparable baseline. |
uploadMetadata.py and its test came onto this branch with a merge; the change is unrelated to cts.tcl and is under review on its own as The-OpenROAD-Project#4819. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
Summary
uploadMetadata.pynow reports how each design's flow run ended. A design that failed before writingmetadata.jsonis no longer silently dropped.How the outcome is derived:
status:timeoutwhentimed_outis set.passedon exit code 0.crashwhen the last stage log showsSignal N received,Segmentation faultorcore dumped.errorotherwise.stage: the highest-numbered stage log, mapped to synth, floorplan, place, cts, grt, route or finish.message: the first[ERROR XXX-NNNN],ERROR:orError:line, searching the logs from the last stage back, truncated to 1024 characters.flow-status.jsonis written next to each design's reports by the CI job (jenkins-ciorfsRunFlow), withexit_code,timed_out,elapsed_minutesandlimit_minutes. Designs without that file are reported exactly as before.On the per-design fallback for oversized payloads,
outcomerides at the message root. An outcome-only design there names itself withrun__flow__platform,run__flow__design,run__flow__variantandrun__flow__generate_date, the last one from the build time.Evidence
[WARN] No data file for nangate45 ibex base.and no record.After: new cases in
flow/test/test_uploadMetadata.py:test_timed_out_design_without_metrics_reports_its_outcome: the record has nometricsand an outcome of timeout, stagegrt, elapsed and limit.test_failed_design_reports_stage_and_first_error_line: error,grt,[ERROR GRT-0116] Global routing failed.test_yosys_error_line_is_reported: synth,ERROR: Module ...test_signal_in_last_stage_log_is_a_crash: crash, place.test_passing_design_keeps_its_metrics_and_reports_passedtest_long_error_line_is_truncatedtest_unreadable_status_without_metrics_skips_the_design: as before, with a warning.test_outcome_only_design_names_itself_on_the_v1_fallbackpython3 -m unittest test.test_uploadMetadata: 43 tests OK. black 26.10.1: clean.Merge Danger
Door: two-way
A revert stops sending
outcomeand outcome-only records.Blast Radius: metrics upload
The payload changes shape for every pipeline that has the CI change: design records gain
outcome, and failed designs appear as outcome-only records. That is the reason for the hold above.genReport.pyandgenReportTable.pyare unaffected: one walkslogs/, the other only collects images underreports/.