diff --git a/docs/user/FlowVariables.md b/docs/user/FlowVariables.md index 0df148d71d..0f5899b40d 100644 --- a/docs/user/FlowVariables.md +++ b/docs/user/FlowVariables.md @@ -183,6 +183,7 @@ configuration file. | LEC_CHECK| Perform formal equivalence checks between before and after netlists. This checks CTS repair timing and the initial synthesis netlist against the final netlist. If this fails, report an issue to OpenROAD.| $(if $(wildcard $(KEPLER_FORMAL_EXE)),1,0)| | LIB_FILES| A Liberty file of the standard cell library with PVT characterization, input and output characteristics, timing and power definitions for each cell.| | | LIB_MODEL| Selects between NLDM and CCS timing models for the ASAP7 platform. Valid values: NLDM (default), CCS. Used in flow/platforms/asap7/config.mk to pick the LIB_DIR subdirectory and accumulate the corresponding $(CORNER)_$(LIB_MODEL)_LIB_FILES list, and in flow/scripts/load.tcl to gate CCS-specific Tcl branches.| NLDM| +| LOWEST_VT_ONLY| Synthesize, place and repair timing with the lowest (fastest) VT only: the cells in NON_LOWEST_VT_CELLS are added to the dont use cells. Use REPAIR_POWER to move cells with slack back to the higher VTs after the global route timing repair. No effect on a platform with a single VT.| 0| | MACRO_PLACEMENT_TCL| Specifies the path of a TCL file on how to place macros manually. The user may choose to place just some of the macros in the design. The macro placer will handle the remaining unplaced macros. Note that using the macro placement file generated by the macro placer will not result in the same global placement as actually running the macro placer: the macro placement file does not contain the locations that the macro placer generates for the standard cells, and those locations impact global placement.| | | MACRO_ROWS_HALO_X| Horizontal distance between the edge of the macro and the beginning of the rows created by tapcell. Only available for ASAP7 PDK and GF180/uart-blocks design.| | | MACRO_ROWS_HALO_Y| Vertical distance between the edge of the macro and the beginning of the rows created by tapcell. Only available for ASAP7 PDK and GF180/uart-blocks design.| | @@ -198,6 +199,7 @@ configuration file. | MIN_PLACE_STEP_COEF| Sets the minimum phi coefficient (pcof_min / ยต_k Lower Bound) for global placement optimization. This parameter controls the step size lower bound in the RePlAce Nesterov optimization algorithm. Lower values may improve convergence but can increase runtime. Valid range: 0.95-1.05| 0.95| | MIN_ROUTING_LAYER| The lowest metal layer name to be used in routing.| | | NEG_CLKGATE_AND_PORTS| Specifies the active low enable clock gating cell and port names when used in conjunction with the INFER_CLKGATES variable. Format is "cell_name enable_port:master_clock:gated_clock"| | +| NON_LOWEST_VT_CELLS| Library cell patterns of every VT but the lowest, set by the platform. Used by LOWEST_VT_ONLY. Empty on a platform with a single VT.| | | NUM_CORES| Passed to `openroad -threads $(NUM_CORES)`, defaults to numbers of cores in system as determined by system specific code in Makefile, `nproc` is tried first. OpenROAD does not limit itself to this number of cores across OpenROAD running instances, which can lead to overprovisioning in contexts such as bazel-orfs where there could be many routing, or place jobs running at the same time.| | | OPENROAD_HIERARCHICAL| Feature toggle to enable to run OpenROAD in hierarchical mode, otherwise considered flat. Will eventually be the default and this option will be retired.| 0| | OPT_POST_GRT_WNS| Optimize WNS after global routing by additional repair_timing that uses VT swap and wire rerouting only to minimize placement and routing disturbance| 1| @@ -250,6 +252,8 @@ configuration file. | RECOVER_POWER| Specifies how many percent of paths with positive slacks can be slowed for power savings [0-100].| 0| | REMOVE_CELLS_FOR_LEC| String patterns directly passed to write_verilog -remove_cells <> for lec checks.| | | REPAIR_PDN_VIA_LAYER| Remove power grid vias which generate DRC violations after detailed routing.| | +| REPAIR_POWER| Run repair_power after the global route timing repair: cells with setup slack to spare are swapped to a higher VT of the same footprint to cut power. With LOWEST_VT_ONLY, the power recovery may use every VT.| 0| +| REPAIR_POWER_VT_BUDGET| Passed as -vt_budget to repair_power: pairs of library cell patterns and the largest percent of the logic cells their instances may be, e.g. `*_SL 5 {*_SL *_L} 20`.| | | REPORT_CLOCK_SKEW| Report clock skew as part of reporting metrics, starting at CTS, before which there is no clock skew. This metric can be quite time-consuming, so it can be useful to disable.| 1| | REPORT_ENDPOINT_SLACK| If set to 1, write the setup and hold slack of every timing endpoint at the end of every flow step (e.g. 3_3_place_gp) to $REPORTS_DIR/_endpoint_slack.json. These snapshots can be compared across steps with flow/util/endpointSlack.py. See docs/contrib/EndpointSlackTracking.md.| 0| | ROUTING_LAYER_ADJUSTMENT| Adjusts routing layer capacities to manage congestion and improve detailed routing. High values ease detailed routing but risk excessive detours and long global routing times, while low values reduce global routing failure but can complicate detailed routing. The global routing running time normally reduces dramatically (entirely design specific, but going from hours to minutes has been observed) when the value is low (such as 0.10). Sometimes, global routing will succeed with lower values and fail with higher values. Exploring results with different values can help shed light on the problem. Start with a too low value, such as 0.10, and bisect to value that works by doing multiple global routing runs. As a last resort, `make global_route_issue` and using the tools/OpenROAD/etc/whittle.py can be useful to debug global routing errors. If there is something specific that is impossible to route, such as a clock line over a macro, global routing will terminate with DRC errors routes that could have been routed were it not for the specific impossible routes. whittle.py should weed out the possible routes and leave a minimal failing case that pinpoints the problem.| 0.5| @@ -365,8 +369,10 @@ configuration file. - [LATCH_MAP_FILE](#LATCH_MAP_FILE) - [LEC_AUX_VERILOG_FILES](#LEC_AUX_VERILOG_FILES) - [LEC_CHECK](#LEC_CHECK) +- [LOWEST_VT_ONLY](#LOWEST_VT_ONLY) - [MIN_BUF_CELL_AND_PORTS](#MIN_BUF_CELL_AND_PORTS) - [NEG_CLKGATE_AND_PORTS](#NEG_CLKGATE_AND_PORTS) +- [NON_LOWEST_VT_CELLS](#NON_LOWEST_VT_CELLS) - [POST_SYNTH_TCL](#POST_SYNTH_TCL) - [POS_CLKGATE_AND_PORTS](#POS_CLKGATE_AND_PORTS) - [PRE_SYNTH_TCL](#PRE_SYNTH_TCL) @@ -426,6 +432,7 @@ configuration file. - [GLOBAL_CONNECT_TCL](#GLOBAL_CONNECT_TCL) - [HOLD_SLACK_MARGIN](#HOLD_SLACK_MARGIN) - [IO_CONSTRAINTS](#IO_CONSTRAINTS) +- [LOWEST_VT_ONLY](#LOWEST_VT_ONLY) - [MACRO_PLACEMENT_TCL](#MACRO_PLACEMENT_TCL) - [MACRO_ROWS_HALO_X](#MACRO_ROWS_HALO_X) - [MACRO_ROWS_HALO_Y](#MACRO_ROWS_HALO_Y) @@ -434,6 +441,7 @@ configuration file. - [MATCH_CELL_FOOTPRINT](#MATCH_CELL_FOOTPRINT) - [MAX_ROUTING_LAYER](#MAX_ROUTING_LAYER) - [MIN_ROUTING_LAYER](#MIN_ROUTING_LAYER) +- [NON_LOWEST_VT_CELLS](#NON_LOWEST_VT_CELLS) - [PDN_TCL](#PDN_TCL) - [PLACE_DENSITY](#PLACE_DENSITY) - [PLACE_DENSITY_LB_ADDON](#PLACE_DENSITY_LB_ADDON) @@ -509,11 +517,13 @@ configuration file. - [GPL_TIMING_DRIVEN](#GPL_TIMING_DRIVEN) - [IO_PLACER_H](#IO_PLACER_H) - [IO_PLACER_V](#IO_PLACER_V) +- [LOWEST_VT_ONLY](#LOWEST_VT_ONLY) - [MATCH_CELL_FOOTPRINT](#MATCH_CELL_FOOTPRINT) - [MAX_PLACE_STEP_COEF](#MAX_PLACE_STEP_COEF) - [MAX_ROUTING_LAYER](#MAX_ROUTING_LAYER) - [MIN_PLACE_STEP_COEF](#MIN_PLACE_STEP_COEF) - [MIN_ROUTING_LAYER](#MIN_ROUTING_LAYER) +- [NON_LOWEST_VT_CELLS](#NON_LOWEST_VT_CELLS) - [PLACE_DENSITY](#PLACE_DENSITY) - [PLACE_DENSITY_LB_ADDON](#PLACE_DENSITY_LB_ADDON) - [PLACE_PINS_ARGS](#PLACE_PINS_ARGS) @@ -548,7 +558,9 @@ configuration file. - [HOLD_SLACK_MARGIN](#HOLD_SLACK_MARGIN) - [LEC_AUX_VERILOG_FILES](#LEC_AUX_VERILOG_FILES) - [LEC_CHECK](#LEC_CHECK) +- [LOWEST_VT_ONLY](#LOWEST_VT_ONLY) - [MATCH_CELL_FOOTPRINT](#MATCH_CELL_FOOTPRINT) +- [NON_LOWEST_VT_CELLS](#NON_LOWEST_VT_CELLS) - [POST_CTS_TCL](#POST_CTS_TCL) - [PRE_CTS_TCL](#PRE_CTS_TCL) - [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW) @@ -576,12 +588,16 @@ configuration file. - [GLOBAL_ROUTE_USE_CUGR](#GLOBAL_ROUTE_USE_CUGR) - [GRT_SEED](#GRT_SEED) - [HOLD_SLACK_MARGIN](#HOLD_SLACK_MARGIN) +- [LOWEST_VT_ONLY](#LOWEST_VT_ONLY) - [MAX_REPAIR_ANTENNAS_ITER_GRT](#MAX_REPAIR_ANTENNAS_ITER_GRT) - [MAX_ROUTING_LAYER](#MAX_ROUTING_LAYER) - [MIN_ROUTING_LAYER](#MIN_ROUTING_LAYER) +- [NON_LOWEST_VT_CELLS](#NON_LOWEST_VT_CELLS) - [OPT_POST_GRT_WNS](#OPT_POST_GRT_WNS) - [POST_GLOBAL_ROUTE_TCL](#POST_GLOBAL_ROUTE_TCL) - [PRE_GLOBAL_ROUTE_TCL](#PRE_GLOBAL_ROUTE_TCL) +- [REPAIR_POWER](#REPAIR_POWER) +- [REPAIR_POWER_VT_BUDGET](#REPAIR_POWER_VT_BUDGET) - [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW) - [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK) - [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT) diff --git a/flow/platforms/asap7/config.mk b/flow/platforms/asap7/config.mk index 286310146f..301c06faff 100644 --- a/flow/platforms/asap7/config.mk +++ b/flow/platforms/asap7/config.mk @@ -119,6 +119,10 @@ export VT_LIST = $(if $(strip $(ASAP7_USE_VT)), $(ASAP7_USE_VT), RVT) # # The first VT in the ASAP7_USE_VT list is the primary VT. The others get added to OTHER_VT export PRIMARY_VT = $(word 1, $(VT_LIST)) +# Cells of every loaded VT but the lowest one (SLVT < LVT < RVT), for +# LOWEST_VT_ONLY +.LOWEST_VT = $(firstword $(foreach vt,SLVT LVT RVT,$(filter $(vt),$(VT_LIST)))) +export NON_LOWEST_VT_CELLS ?= $(patsubst %VT,*_ASAP7_75t_%,$(filter-out $(.LOWEST_VT),$(VT_LIST))) export PRIMARY_VT_TAG = $(strip $(patsubst %VT, %, $(PRIMARY_VT))) export OTHER_VT = $(wordlist 2, $(words $(VT_LIST)), $(VT_LIST)) diff --git a/flow/platforms/gt2n/config.mk b/flow/platforms/gt2n/config.mk index 50e24068be..dfd77aa16f 100644 --- a/flow/platforms/gt2n/config.mk +++ b/flow/platforms/gt2n/config.mk @@ -31,6 +31,11 @@ export OTHER_W = $(wordlist 2, $(words $(GT2N_USE_W)), $(GT2N_USE_W)) export PRIMARY_VT = $(word 1, $(GT2N_USE_VT)) export OTHER_VT = $(wordlist 2, $(words $(GT2N_USE_VT)), $(GT2N_USE_VT)) +# Cells of every loaded Vt but the lowest one (elvt < ulvt < lvt < svt < hvt), +# for LOWEST_VT_ONLY +.LOWEST_VT = $(firstword $(foreach vt,elvt ulvt lvt svt hvt,$(filter $(vt),$(GT2N_USE_VT)))) +export NON_LOWEST_VT_CELLS ?= $(addprefix *_,$(filter-out $(.LOWEST_VT),$(GT2N_USE_VT))) + # Cartesian product of (W, Vt) used by every cell-name template below. ALL_WVT = $(foreach w,$(GT2N_USE_W), $(foreach vt,$(GT2N_USE_VT),$(w)_$(vt))) PRIMARY_WVT = $(PRIMARY_W)_$(PRIMARY_VT) diff --git a/flow/scripts/cts.tcl b/flow/scripts/cts.tcl index 7439569d12..5963bb4b40 100644 --- a/flow/scripts/cts.tcl +++ b/flow/scripts/cts.tcl @@ -31,7 +31,7 @@ if { [env_var_exists_and_non_empty CTS_ARGS] } { set cts_args $::env(CTS_ARGS) } -set_dont_use $::env(DONT_USE_CELLS) +set_dont_use [dont_use_cells] log_cmd clock_tree_synthesis {*}$cts_args diff --git a/flow/scripts/floorplan.tcl b/flow/scripts/floorplan.tcl index 3190eeb612..a8274c40e3 100644 --- a/flow/scripts/floorplan.tcl +++ b/flow/scripts/floorplan.tcl @@ -120,7 +120,7 @@ if { [env_var_exists_and_non_empty FASTROUTE_TCL] } { } source_env_var_if_exists FOOTPRINT_TCL -log_cmd set_dont_use $::env(DONT_USE_CELLS) +log_cmd set_dont_use [dont_use_cells] # The transforms below (repair_tie_fanout, replace_arith_modules, # repair_timing_helper) look like synthesis-stage diff --git a/flow/scripts/global_place.tcl b/flow/scripts/global_place.tcl index 4a125954e2..3b0e2265a4 100644 --- a/flow/scripts/global_place.tcl +++ b/flow/scripts/global_place.tcl @@ -4,7 +4,7 @@ erase_non_stage_variables place load_design 3_2_place_iop.odb 2_floorplan.sdc source_step_tcl PRE GLOBAL_PLACE -set_dont_use $::env(DONT_USE_CELLS) +set_dont_use [dont_use_cells] if { $::env(GPL_TIMING_DRIVEN) } { remove_buffers diff --git a/flow/scripts/global_route.tcl b/flow/scripts/global_route.tcl index 01ad91ba30..daaec7c1a7 100644 --- a/flow/scripts/global_route.tcl +++ b/flow/scripts/global_route.tcl @@ -63,9 +63,7 @@ set_placement_padding -global \ set_propagated_clock [all_clocks] log_cmd estimate_parasitics -if { [env_var_exists_and_non_empty DONT_USE_CELLS] } { - set_dont_use $::env(DONT_USE_CELLS) -} +set_dont_use [dont_use_cells] if { !$::env(SKIP_INCREMENTAL_REPAIR) } { if { $::env(DETAILED_METRICS) } { @@ -133,6 +131,15 @@ if { !$::env(OPT_POST_GRT_WNS) } { -congestion_report_file $::env(REPORTS_DIR)/congestion_post_recover_power.rpt } +if { $::env(REPAIR_POWER) } { + log_cmd global_route -start_incremental {*}$allow_congestion + repair_power_helper + # Route the nets of the swapped cells + run_global_route_and_catch_failures -end_incremental {*}$res_aware \ + {*}$allow_congestion \ + -congestion_report_file $::env(REPORTS_DIR)/congestion_post_repair_power.rpt +} + if { !$::env(SKIP_ANTENNA_REPAIR) && [env_var_exists_and_non_empty MAX_REPAIR_ANTENNAS_ITER_GRT] diff --git a/flow/scripts/resize.tcl b/flow/scripts/resize.tcl index fa8a26da3d..0a3e1838ee 100644 --- a/flow/scripts/resize.tcl +++ b/flow/scripts/resize.tcl @@ -9,7 +9,7 @@ log_cmd estimate_parasitics set instance_count_before [sta::network_leaf_instance_count] set pin_count_before [sta::network_leaf_pin_count] -set_dont_use $::env(DONT_USE_CELLS) +set_dont_use [dont_use_cells] if { [env_var_exists_and_non_empty EARLY_SIZING_CAP_RATIO] } { log_cmd set_opt_config -set_early_sizing_cap_ratio $env(EARLY_SIZING_CAP_RATIO) diff --git a/flow/scripts/synth_preamble.tcl b/flow/scripts/synth_preamble.tcl index f024281f80..021eb4b1c3 100644 --- a/flow/scripts/synth_preamble.tcl +++ b/flow/scripts/synth_preamble.tcl @@ -297,10 +297,8 @@ foreach lib $::env(LIB_FILES) { # Exclude dont_use cells. This includes macros that are specified via # LIB_FILES and ADDITIONAL_LIBS that are included in LIB_FILES. set lib_dont_use_args "" -if { [env_var_exists_and_non_empty DONT_USE_CELLS] } { - foreach cell $::env(DONT_USE_CELLS) { - lappend lib_dont_use_args -dont_use $cell - } +foreach cell [dont_use_cells] { + lappend lib_dont_use_args -dont_use $cell } # Never map logic onto a hardened block. A block's timing model gives a # constant output a function, and ABC would take the block for a gate. diff --git a/flow/scripts/synth_syn.tcl b/flow/scripts/synth_syn.tcl index 52617bd5ae..1912677fea 100644 --- a/flow/scripts/synth_syn.tcl +++ b/flow/scripts/synth_syn.tcl @@ -13,9 +13,7 @@ if { [env_var_exists_and_non_empty ADDITIONAL_LEFS] } { log_cmd read_lef $lef } } -if { [env_var_exists_and_non_empty DONT_USE_CELLS] } { - log_cmd set_dont_use $::env(DONT_USE_CELLS) -} +log_cmd set_dont_use [dont_use_cells] # Setup verilog include directories set vIdirsArgs "" diff --git a/flow/scripts/util.tcl b/flow/scripts/util.tcl index 09ab88033a..234ede6fed 100644 --- a/flow/scripts/util.tcl +++ b/flow/scripts/util.tcl @@ -81,6 +81,41 @@ proc recover_power_helper { } { report_power } +# Cells the flow must not use: DONT_USE_CELLS, plus the cells of every VT but +# the lowest in a lowest-VT-only flow +proc dont_use_cells { } { + set cells [env_var_or_empty DONT_USE_CELLS] + if { $::env(LOWEST_VT_ONLY) } { + lappend cells {*}$::env(NON_LOWEST_VT_CELLS) + } + return $cells +} + +proc repair_power_helper { } { + puts "Switching non critical gates to higher VT for power recovery" + report_tns + report_wns + report_power + # The power recovery of a lowest-VT-only flow may use every VT + if { $::env(LOWEST_VT_ONLY) } { + log_cmd reset_dont_use + log_cmd set_dont_use [env_var_or_empty DONT_USE_CELLS] + } + set additional_args "-verbose" + if { $::env(SETUP_SLACK_MARGIN) != 0 } { + lappend additional_args -setup_margin $::env(SETUP_SLACK_MARGIN) + } + append_env_var additional_args REPAIR_POWER_VT_BUDGET -vt_budget 1 + log_cmd repair_power {*}$additional_args + if { $::env(LOWEST_VT_ONLY) } { + log_cmd reset_dont_use + log_cmd set_dont_use [dont_use_cells] + } + report_tns + report_wns + report_power +} + proc extract_stage { input_file } { # Match the stage prefix on the basename, not the full path: an ancestor # dir like ".../4_something/3_place.odb" would otherwise match "4_" -> stage 4. diff --git a/flow/scripts/variables.yaml b/flow/scripts/variables.yaml index ac9d582494..3ab244baef 100644 --- a/flow/scripts/variables.yaml +++ b/flow/scripts/variables.yaml @@ -47,6 +47,21 @@ RECOVER_POWER: Specifies how many percent of paths with positive slacks can be slowed for power savings [0-100]. default: 0 +REPAIR_POWER: + description: > + Run repair_power after the global route timing repair: cells with setup + slack to spare are swapped to a higher VT of the same footprint to cut + power. With LOWEST_VT_ONLY, the power recovery may use every VT. + default: 0 + stages: + - grt +REPAIR_POWER_VT_BUDGET: + description: > + Passed as -vt_budget to repair_power: pairs of library cell patterns and + the largest percent of the logic cells their instances may be, e.g. + `*_SL 5 {*_SL *_L} 20`. + stages: + - grt SKIP_INCREMENTAL_REPAIR: default: 0 description: > @@ -159,6 +174,30 @@ PLATFORM_TCL: DONT_USE_CELLS: description: | Dont use cells eases pin access in detailed routing. +LOWEST_VT_ONLY: + description: > + Synthesize, place and repair timing with the lowest (fastest) VT only: + the cells in NON_LOWEST_VT_CELLS are added to the dont use cells. Use + REPAIR_POWER to move cells with slack back to the higher VTs after the + global route timing repair. No effect on a platform with a single VT. + default: 0 + stages: + - synth + - floorplan + - place + - cts + - grt +NON_LOWEST_VT_CELLS: + description: > + Library cell patterns of every VT but the lowest, set by the platform. + Used by LOWEST_VT_ONLY. Empty on a platform with a single VT. + default: "" + stages: + - synth + - floorplan + - place + - cts + - grt SYNTH_GUT: description: > Load design and remove all internal logic before doing synthesis. This is diff --git a/flow/test/test_uploadMetadata.py b/flow/test/test_uploadMetadata.py index 92555ed5a5..be94f63757 100755 --- a/flow/test/test_uploadMetadata.py +++ b/flow/test/test_uploadMetadata.py @@ -102,6 +102,23 @@ def add_design(self, platform, design, variant="base", metrics=None): with open(os.path.join(report_dir, "metadata.json"), "w") as f: json.dump(metrics or {"finish:area": 1.0}, f) + def add_flow_status(self, platform, design, status, variant="base", logs=None): + """Record how a design's flow run ended, as the CI job does, with stage + logs given as {file name: content}.""" + report_dir = os.path.join(self.flow, "reports", platform, design, variant) + os.makedirs(report_dir, exist_ok=True) + with open(os.path.join(report_dir, "flow-status.json"), "w") as f: + f.write(status if isinstance(status, str) else json.dumps(status)) + log_dir = os.path.join(self.flow, "logs", platform, design, variant) + os.makedirs(log_dir, exist_ok=True) + for name, content in (logs or {}).items(): + with open(os.path.join(log_dir, name), "w") as f: + f.write(content) + + def designs(self, messages): + """The pipeline payload's design records, by design name.""" + return {d["design"]: d for d in messages[0]["data"]["designs"]} + def run_script(self, *extra, fail=False, tz=None, extra_env=None): env = dict(os.environ) for name in REQUESTER_ENV: @@ -537,6 +554,133 @@ def test_no_publisher_publishes_nothing(self): self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertFalse(os.path.exists(self.out)) + def test_timed_out_design_without_metrics_reports_its_outcome(self): + self.add_design("nangate45", "gcd") + self.add_flow_status( + "nangate45", + "ibex", + { + "exit_code": 124, + "timed_out": True, + "elapsed_minutes": 240.2, + "limit_minutes": 240, + }, + logs={"3_3_place_gp.log": "placing\n", "5_1_grt.log": "routing\n"}, + ) + result, messages = self.run_script("--jobName", "ORFS") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + designs = self.designs(messages) + self.assertEqual( + designs["ibex"], + { + "platform": "nangate45", + "design": "ibex", + "variant": "base", + "outcome": { + "status": "timeout", + "stage": "grt", + "elapsed_minutes": 240.2, + "limit_minutes": 240, + }, + }, + ) + self.assertNotIn("outcome", designs["gcd"]) + self.assertIn("reporting its timeout outcome only", result.stdout) + + def test_failed_design_reports_stage_and_first_error_line(self): + self.add_flow_status( + "asap7", + "aes", + {"exit_code": 2, "timed_out": False}, + logs={ + "4_1_cts.log": "[ERROR CTS-0001] not this one\n", + "5_1_grt.log": "info\n[ERROR GRT-0116] Global routing failed.\n" + "[ERROR GRT-0001] later\n", + }, + ) + _, messages = self.run_script() + outcome = self.designs(messages)["aes"]["outcome"] + self.assertEqual(outcome["status"], "error") + self.assertEqual(outcome["stage"], "grt") + self.assertEqual(outcome["message"], "[ERROR GRT-0116] Global routing failed.") + + def test_yosys_error_line_is_reported(self): + self.add_flow_status( + "asap7", + "aes", + {"exit_code": 2}, + logs={"1_2_yosys.log": "ERROR: Module `foo' referenced but not found.\n"}, + ) + _, messages = self.run_script() + outcome = self.designs(messages)["aes"]["outcome"] + self.assertEqual((outcome["status"], outcome["stage"]), ("error", "synth")) + self.assertEqual( + outcome["message"], "ERROR: Module `foo' referenced but not found." + ) + + def test_signal_in_last_stage_log_is_a_crash(self): + self.add_flow_status( + "sky130hd", + "jpeg", + {"exit_code": 2}, + logs={"3_3_place_gp.log": "Signal 11 received\nStack trace:\n"}, + ) + _, messages = self.run_script() + outcome = self.designs(messages)["jpeg"]["outcome"] + self.assertEqual((outcome["status"], outcome["stage"]), ("crash", "place")) + self.assertNotIn("message", outcome) + + def test_passing_design_keeps_its_metrics_and_reports_passed(self): + self.add_design("nangate45", "gcd", metrics={"finish:area": 5.0}) + self.add_flow_status( + "nangate45", "gcd", {"exit_code": 0, "elapsed_minutes": 3.5} + ) + _, messages = self.run_script() + record = self.designs(messages)["gcd"] + self.assertEqual(record["metrics"], {"finish__area": 5.0}) + self.assertEqual( + record["outcome"], {"status": "passed", "elapsed_minutes": 3.5} + ) + + def test_long_error_line_is_truncated(self): + self.add_flow_status( + "asap7", + "aes", + {"exit_code": 2}, + logs={"5_2_route.log": "[ERROR DRT-0001] " + "x" * 2000 + "\n"}, + ) + _, messages = self.run_script() + outcome = self.designs(messages)["aes"]["outcome"] + self.assertEqual(outcome["stage"], "route") + self.assertEqual(len(outcome["message"]), 1024) + + def test_unreadable_status_without_metrics_skips_the_design(self): + self.add_design("nangate45", "gcd") + self.add_flow_status("nangate45", "ibex", "not json") + self.add_flow_status("nangate45", "jpeg", {"timed_out": False}) + result, messages = self.run_script() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(set(self.designs(messages)), {"gcd"}) + self.assertIn("Reporting no outcome", result.stdout) + self.assertIn("[WARN] No data file for nangate45 ibex base.", result.stdout) + + def test_outcome_only_design_names_itself_on_the_v1_fallback(self): + self.add_design("nangate45", "gcd", metrics=OVERSIZED_METRICS) + self.add_flow_status( + "nangate45", + "ibex", + {"exit_code": 124, "timed_out": True, "limit_minutes": 60}, + ) + result, messages = self.run_script("--buildTime", "2026-10-05 12:34") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + by_design = {m["data"].get("run__flow__design"): m["data"] for m in messages} + ibex = by_design["ibex"] + self.assertEqual(ibex["run__flow__platform"], "nangate45") + self.assertEqual(ibex["run__flow__variant"], "base") + self.assertEqual(ibex["run__flow__generate_date"], "2026-10-05 12:34") + self.assertEqual(ibex["outcome"], {"status": "timeout", "limit_minutes": 60}) + self.assertNotIn("outcome", by_design.get(None, {})) + if __name__ == "__main__": unittest.main() diff --git a/flow/util/uploadMetadata.py b/flow/util/uploadMetadata.py index 5aff02839a..4a7419d4a8 100755 --- a/flow/util/uploadMetadata.py +++ b/flow/util/uploadMetadata.py @@ -3,6 +3,7 @@ import json import argparse import os +import re import sys from datetime import datetime, timezone @@ -160,17 +161,154 @@ def _non_negative_int(val): MAX_PUBSUB_BYTES = 8 * 1024 * 1024 -def build_design_record(dataFile, platform, design, variant): - """Return a dict for one design to be included in the pipeline-level payload.""" - with open(dataFile) as f: - data = json.load(f) - metrics = {k.replace(":", "__"): v for k, v in data.items()} +def build_design_record(dataFile, platform, design, variant, outcome=None): + """Return a dict for one design to be included in the pipeline-level payload. + + ``dataFile`` is the design's metadata.json, or ``None`` for a run that ended + before writing one; such a record carries only its ``outcome``. + """ + record = {"platform": platform, "design": design, "variant": variant} + if dataFile: + with open(dataFile) as f: + data = json.load(f) + record["metrics"] = {k.replace(":", "__"): v for k, v in data.items()} + if outcome: + record["outcome"] = outcome + return record + + +# How a flow run ended, as the CI job records it next to the design's reports: +# {"exit_code": int, "timed_out": bool, "elapsed_minutes": float, +# "limit_minutes": int}. Every key is optional. +STATUS_FILENAME = "flow-status.json" +MAX_OUTCOME_MESSAGE = 1024 +# Stage log names start with the step number: 1_2_yosys.log, 5_1_grt.log. +STAGE_LOG = re.compile(r"^(\d+)_(?:(\d+)_)?.*\.log$") +# The first matching line of a failing stage's log is the error reported. The +# OpenROAD form carries the tool's message ID; Yosys prints "ERROR: ". +ERROR_LINE = re.compile( + r"\[(?:ERROR|CRITICAL|FATAL) [A-Z][A-Z0-9]*-\d+\].*|^ERROR: .*|^Error: .*" +) +# What a tool that died on a signal leaves in its log, as opposed to an error it +# reported itself. +CRASH_LINE = re.compile(r"Signal \d+ received|Segmentation fault|core dumped") + + +def _stage_name(log_name): + """The flow stage a stage log belongs to, e.g. 5_1_grt.log -> "grt".""" + match = STAGE_LOG.match(log_name) + if not match: + return None + step, sub = match.group(1), match.group(2) + if step == "5": + return "grt" if sub == "1" else "route" return { - "platform": platform, - "design": design, - "variant": variant, - "metrics": metrics, - } + "1": "synth", + "2": "floorplan", + "3": "place", + "4": "cts", + "6": "finish", + }.get(step) + + +def _stage_logs(log_dir): + """The design's stage logs, last step first.""" + try: + names = os.listdir(log_dir) + except OSError: + return [] + + def key(name): + match = STAGE_LOG.match(name) + return (int(match.group(1)), int(match.group(2) or 0), name) + + return sorted((n for n in names if STAGE_LOG.match(n)), key=key, reverse=True) + + +def _scan_log(path): + """The first error line of a log and whether it shows a crash.""" + error_line = None + crashed = False + try: + with open(path, errors="replace") as f: + for line in f: + if error_line is None: + match = ERROR_LINE.search(line) + if match: + error_line = match.group(0).strip() + if not crashed and CRASH_LINE.search(line): + crashed = True + if error_line is not None and crashed: + break + except OSError: + pass + return error_line, crashed + + +def read_flow_outcome(report_dir, log_dir): + """The design's outcome from its flow-status.json and stage logs, or ``None``. + + ``status`` is ``timeout`` when the time limit stopped the run, ``passed`` on + exit code 0, ``crash`` when the last stage log shows the tool died on a + signal, and ``error`` otherwise. A failed run also names the stage it ended + in and the first error line found in its logs, last stage first. + """ + path = os.path.join(report_dir, STATUS_FILENAME) + if not os.path.exists(path): + return None + try: + with open(path) as f: + status = json.load(f) + except (OSError, ValueError) as e: + print(f"[WARN] Could not read {path}: {e}. Reporting no outcome.") + return None + if not isinstance(status, dict): + print(f"[WARN] {path} does not hold a JSON object. Reporting no outcome.") + return None + + exit_code = status.get("exit_code") + timed_out = status.get("timed_out") is True + if not timed_out and not isinstance(exit_code, int): + print(f"[WARN] {path} has no exit_code. Reporting no outcome.") + return None + + outcome = {} + for key in ("elapsed_minutes", "limit_minutes"): + value = status.get(key) + if ( + isinstance(value, (int, float)) + and not isinstance(value, bool) + and value >= 0 + ): + outcome[key] = value + + if not timed_out and exit_code == 0: + outcome["status"] = "passed" + return outcome + + logs = _stage_logs(log_dir) + if logs: + stage = _stage_name(logs[0]) + if stage: + outcome["stage"] = stage + + message = None + crashed = False + for i, name in enumerate(logs): + error_line, log_crashed = _scan_log(os.path.join(log_dir, name)) + if i == 0: + crashed = log_crashed + if error_line: + message = error_line + break + if message: + outcome["message"] = message[:MAX_OUTCOME_MESSAGE] + + if timed_out: + outcome["status"] = "timeout" + else: + outcome["status"] = "crash" if crashed else "error" + return outcome def _submodule_defect(entry): @@ -426,6 +564,10 @@ def publish_v1_per_design( The requester keys go on after the metrics, so a metric that happened to share their name could not replace who asked for the build. + + A design's outcome rides at the root as ``outcome``. A design that has only + an outcome carries no metrics to name it, so its message names the design + with the same run__flow__ keys metadata.json would hold, dated by the build. """ futures = [] failed = 0 @@ -441,7 +583,19 @@ def publish_v1_per_design( } if provenance: payload.update(provenance) - payload.update(d["metrics"]) + if "metrics" in d: + payload.update(d["metrics"]) + else: + payload.update( + { + "run__flow__platform": d["platform"], + "run__flow__design": d["design"], + "run__flow__variant": d["variant"], + "run__flow__generate_date": resolve_build_time(args), + } + ) + if "outcome" in d: + payload["outcome"] = d["outcome"] if requester: payload.update(requester) @@ -513,16 +667,28 @@ def publish_v1_per_design( platform = dirList[1] design = dirList[2] variant = dirList[3] + if platform == "sky130hd_fakestack" or platform == "src": + print(f"[WARN] Skiping upload {platform} {design} {variant}.") + continue dataFile = os.path.join(reportDir, RUN_FILENAME) if not os.path.exists(dataFile): + dataFile = None + outcome = read_flow_outcome( + reportDir, os.path.join("logs", platform, design, variant) + ) + if dataFile is None and outcome is None: print(f"[WARN] No data file for {platform} {design} {variant}.") continue - if platform == "sky130hd_fakestack" or platform == "src": - print(f"[WARN] Skiping upload {platform} {design} {variant}.") - continue + if dataFile is None: + print( + f"[INFO] No data file for {platform} {design} {variant}; " + f"reporting its {outcome['status']} outcome only." + ) # --- PUBSUB --- if publisher: - design_records.append(build_design_record(dataFile, platform, design, variant)) + design_records.append( + build_design_record(dataFile, platform, design, variant, outcome) + ) # --- END PUBSUB --- # --- PUBSUB ---