diff --git a/docs/contrib/EndpointSlackTracking.md b/docs/contrib/EndpointSlackTracking.md new file mode 100644 index 0000000000..ef9a30a27e --- /dev/null +++ b/docs/contrib/EndpointSlackTracking.md @@ -0,0 +1,162 @@ +# Tracking Endpoint Slack Across Flow Steps + +When working on an optimization (resizing, buffering, CTS, repair timing, +...) it is useful to see how the slack of each timing endpoint moves from +one flow step to the next, not just how WNS/TNS change. ORFS can record the +setup and hold slack of every endpoint at the end of every flow step +(`1_synth`, `2_1_floorplan`, ..., `3_3_place_gp`, ..., `6_final`) and compare +any two steps, including a scatter plot of slack (x: earlier step, y: later +step). + +## Recording + +Set `REPORT_ENDPOINT_SLACK=1`: + +``` shell +cd flow +make DESIGN_CONFIG=designs/nangate45/gcd/config.mk REPORT_ENDPOINT_SLACK=1 +``` + +Every step then writes `$(REPORTS_DIR)/_endpoint_slack.json`, e.g. +`3_3_place_gp_endpoint_slack.json`. The snapshot is taken by +`write_endpoint_slack` in `flow/scripts/util.tcl`, called from +`orfs_write_db` so the step name is the name of the `.odb` the step writes. +It works in the single-process `flow/scripts/flow.tcl` flow as well. Steps +that just copy their input (`orfs_copy_db`, e.g. `5_3_fillcell` without fill +cells) re-time the design if it is loaded, or else copy the previous step's +snapshot. Intermediate `.odb` files written through `orfs_write_db` (e.g. CTS +`save_progress`) also get a snapshot; `*-failed` ones don't. + +So that all steps are timed the same way, parasitics are estimated before +reporting as in `open.tcl`, based on the step number: + +| Steps | Parasitics | +| --- | --- | +| 1, 2 | none (cells are not placed) | +| 3, 4 | `estimate_parasitics -placement` | +| 5, `6_1_fill` | `estimate_parasitics -global_routing` if global routes exist | +| `6_final` | `rcx`: the extracted `6_final.spef`, as in `6_finish.rpt` (`global_routing` on platforms without `RCX_RULES`) | + +The parasitics model is recorded in each snapshot and shown by +`endpointSlack.py`. Slack changes between steps with different models (e.g. +`2_4_floorplan_pdn` to `3_1_place_gp_skip_io`, `4_1_cts` to `5_1_grt`, or +`6_1_fill` to `6_final`) include the change of model. Detailed routing is +only extracted in `6_final`, so `5_2_route` to `6_1_fill` show global routing +estimates. + +Each file contains `design`, `step`, `parasitics`, `time_unit` (seconds per +time unit of the values) and `endpoints`: a map from every endpoint pin +(`sta::endpoints`) to `[setup slack, hold slack]`, `null` where the endpoint +is unconstrained. The slack is the pin's worst slack over all path groups +and clocks (`get_property $pin slack_max|slack_min`). On asap7/swerv_wrapper +(about 30k endpoints) a snapshot is about 1 MB and takes about 10 s, mostly +the timing update. + +## Analysis + +`flow/util/endpointSlack.py` only uses the Python standard library. + +``` shell +R=reports/nangate45/gcd/base + +# Steps in execution order with parasitics, WNS, TNS and violation counts +python3 util/endpointSlack.py list $R + +# Compare consecutive steps across the whole run; --html also writes a +# scatter for every step vs. its successor and an index.html to a directory +python3 util/endpointSlack.py track $R --html track + +# Compare two steps; -o writes an interactive HTML scatter +python3 util/endpointSlack.py compare \ + $R/3_5_place_dp_endpoint_slack.json $R/4_1_cts_endpoint_slack.json \ + --unit ps -o compare.html + +# Flatten all snapshots into one CSV (one row per endpoint per step) +python3 util/endpointSlack.py export $R -o slack.csv +``` + +For setup and hold, `compare` reports: + +- endpoints in both steps, and how many improved, degraded, stayed the + same, became violating (`newly failing`) or stopped violating + (`newly passing`), with statistics of the slack change (B - A). + Improved, degraded and unchanged exclude endpoints with positive slack in + both steps, which are not being optimized; +- endpoints only in A or only in B, i.e. added or removed from the netlist; +- endpoints in both netlists but constrained in only one of them; +- how well the two steps agree on the critical endpoints (see below). + +Unconstrained endpoints, such as the outputs of CTS dummy load cells, are +ignored. + +### Agreement metrics + +Correlation over all endpoints is dominated by endpoints with large positive +slack and is close to 1 even when the critical endpoints change a lot (0.9997 +for setup on asap7/swerv_wrapper from `3_5_place_dp` to `4_1_cts`). The +metrics are therefore measured on a critical set: the worst K endpoints of A +or of B (`--critical K`, default 1000), excluding endpoints with positive +slack in both steps, which are not being optimized. When fewer than K +endpoints violate, the critical set is the endpoints violating in either +step; when nothing violates (often hold) it is empty and the metrics are +reported as `-`. + +| Metric | Meaning | +| --- | --- | +| `spearman` | Spearman rank correlation of A and B slack (ties get average ranks): is the order of the critical endpoints preserved? | +| `top agreement` | Fraction of the critical set that is in the worst K of both steps. | +| `bias`, `MAE`, `RMSE`, `max\|delta\|` | Mean, mean absolute, root mean square and maximum absolute slack change B - A, i.e. distance from the y = x line. | +| `violator recall` | Of the endpoints violating in B, the fraction already violating in A (over all endpoints). | +| `violator precision` | Of the endpoints violating in A, the fraction still violating in B (over all endpoints). | + +They are printed by `compare` and `track`, shown in the HTML stats, and +the `track` index has Spearman, top agreement and MAE columns for every step +transition. + +Times are shown in `--unit` (default ns) with a fixed number of digits after +the decimal point (3 for ns, 1 for ps, or `--digits`), in a fixed-width font +in the HTML. +Slack changes smaller than half the last displayed digit (0.5 ps with the +default ns and 3 digits), or `--tolerance` in `--unit`, count as unchanged +for the improved/degraded counts, the tables of most degraded and improved +endpoints and the `track` row colors. Otherwise numerical noise, e.g. a few +femtoseconds between `5_1_grt` and `5_2_route`, shows up as changes that +display as 0. + +`--worst N` restricts the endpoints in both to the union of the worst N +endpoints of A and of B, to focus on the critical part of the design. + +The HTML is self-contained (no network access needed). It has: + +- WNS, TNS and violation counts of both steps; +- a scatter of endpoint slack with a y = x line, where above the line means + the endpoint improved. It is drawn on a canvas so it handles hundreds of + thousands of endpoints. Drag to zoom, double-click to reset, hover for + the endpoint name; +- a setup/hold toggle; +- tables of the most degraded and most improved endpoints, excluding + endpoints with positive slack in both steps, and of the endpoints only in + one step. + +`track --html DIR` writes the same page for every step and its successor +(`__.html`, with previous/next links) and `DIR/index.html`, +a table of every transition with WNS/TNS and the improved, degraded, newly +failing and newly passing counts for setup and hold. Endpoints with positive +slack in both steps are not considered. Transitions where no considered +endpoint changed (e.g. `2_2_floorplan_macro` to `2_3_floorplan_tapcell`) are +greyed out, and transitions where every considered endpoint improved, for +setup and hold, are green. + +For other analyses, import the module (e.g. from a notebook): +`load_snapshot()`, `load_dir()`, `summary()`, `compare()`, `movement()`, +`metrics()` and `spearman()` +return plain Python dicts and lists. + +## Testing + +`flow/test/test_endpointSlack.py` covers loading and comparison with +synthetic snapshots and runs in the util CI job: + +``` shell +python3 flow/test/test_endpointSlack.py +``` diff --git a/docs/toc.yml b/docs/toc.yml index 18baefd3fd..92b208b6b0 100644 --- a/docs/toc.yml +++ b/docs/toc.yml @@ -45,6 +45,8 @@ entries: title: Guide to Integrate a New Platform into the OpenROAD Flow - file: contrib/CI.md title: CI Guide + - file: contrib/EndpointSlackTracking.md + title: Tracking Endpoint Slack Across Flow Steps - file: contrib/CODE_OF_CONDUCT.md title: Code of Conduct diff --git a/docs/user/FlowVariables.md b/docs/user/FlowVariables.md index c575653135..dfb6ab40f2 100644 --- a/docs/user/FlowVariables.md +++ b/docs/user/FlowVariables.md @@ -247,6 +247,7 @@ configuration file. | REMOVE_CELLS_FOR_LEC| String patterns directly passed to write_verilog -remove_cells <> for lec checks.| | | REPAIR_PDN_VIA_LAYER| Remove power grid vias which generate DRC violations after detailed routing.| | | REPORT_CLOCK_SKEW| Report clock skew as part of reporting metrics, starting at CTS, before which there is no clock skew. This metric can be quite time-consuming, so it can be useful to disable.| 1| +| REPORT_ENDPOINT_SLACK| If set to 1, write the setup and hold slack of every timing endpoint at the end of every flow step (e.g. 3_3_place_gp) to $REPORTS_DIR/_endpoint_slack.json. These snapshots can be compared across steps with flow/util/endpointSlack.py. See docs/contrib/EndpointSlackTracking.md.| 0| | ROUTING_LAYER_ADJUSTMENT| Adjusts routing layer capacities to manage congestion and improve detailed routing. High values ease detailed routing but risk excessive detours and long global routing times, while low values reduce global routing failure but can complicate detailed routing. The global routing running time normally reduces dramatically (entirely design specific, but going from hours to minutes has been observed) when the value is low (such as 0.10). Sometimes, global routing will succeed with lower values and fail with higher values. Exploring results with different values can help shed light on the problem. Start with a too low value, such as 0.10, and bisect to value that works by doing multiple global routing runs. As a last resort, `make global_route_issue` and using the tools/OpenROAD/etc/whittle.py can be useful to debug global routing errors. If there is something specific that is impossible to route, such as a clock line over a macro, global routing will terminate with DRC errors routes that could have been routed were it not for the specific impossible routes. whittle.py should weed out the possible routes and leave a minimal failing case that pinpoints the problem.| 0.5| | RTLMP_AREA_WT| Weight for the area of the current floorplan.| 0.1| | RTLMP_ARGS| Overrides all other RTL macro placer arguments.| | @@ -354,6 +355,7 @@ configuration file. - [POST_SYNTH_TCL](#POST_SYNTH_TCL) - [POS_CLKGATE_AND_PORTS](#POS_CLKGATE_AND_PORTS) - [PRE_SYNTH_TCL](#PRE_SYNTH_TCL) +- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK) - [SDC_FILE](#SDC_FILE) - [SEC_CHECK](#SEC_CHECK) - [SKIP_REPORT_METRICS](#SKIP_REPORT_METRICS) @@ -425,6 +427,7 @@ configuration file. - [PRE_PDN_TCL](#PRE_PDN_TCL) - [PRE_TAPCELL_TCL](#PRE_TAPCELL_TCL) - [REMOVE_ABC_BUFFERS](#REMOVE_ABC_BUFFERS) +- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK) - [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT) - [RTLMP_AREA_WT](#RTLMP_AREA_WT) - [RTLMP_ARGS](#RTLMP_ARGS) @@ -502,6 +505,7 @@ configuration file. - [PRE_GLOBAL_PLACE_TCL](#PRE_GLOBAL_PLACE_TCL) - [PRE_IO_PLACEMENT_TCL](#PRE_IO_PLACEMENT_TCL) - [PRE_RESIZE_TCL](#PRE_RESIZE_TCL) +- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK) - [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT) - [SKIP_REPORT_METRICS](#SKIP_REPORT_METRICS) - [TNS_END_PERCENT](#TNS_END_PERCENT) @@ -525,6 +529,7 @@ configuration file. - [POST_CTS_TCL](#POST_CTS_TCL) - [PRE_CTS_TCL](#PRE_CTS_TCL) - [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW) +- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK) - [SETUP_MOVE_SEQUENCE](#SETUP_MOVE_SEQUENCE) - [SETUP_SLACK_MARGIN](#SETUP_SLACK_MARGIN) - [SKIP_BUFFER_REMOVAL](#SKIP_BUFFER_REMOVAL) @@ -554,6 +559,7 @@ configuration file. - [POST_GLOBAL_ROUTE_TCL](#POST_GLOBAL_ROUTE_TCL) - [PRE_GLOBAL_ROUTE_TCL](#PRE_GLOBAL_ROUTE_TCL) - [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW) +- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK) - [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT) - [SETUP_MOVE_SEQUENCE](#SETUP_MOVE_SEQUENCE) - [SETUP_SLACK_MARGIN](#SETUP_SLACK_MARGIN) @@ -587,6 +593,7 @@ configuration file. - [PRE_DETAIL_ROUTE_TCL](#PRE_DETAIL_ROUTE_TCL) - [PRE_FILLCELL_TCL](#PRE_FILLCELL_TCL) - [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW) +- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK) - [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT) - [SKIP_ANTENNA_REPAIR_POST_DRT](#SKIP_ANTENNA_REPAIR_POST_DRT) - [SKIP_DETAILED_ROUTE](#SKIP_DETAILED_ROUTE) @@ -610,6 +617,7 @@ configuration file. - [PRE_FINAL_REPORT_TCL](#PRE_FINAL_REPORT_TCL) - [PWR_NETS_VOLTAGES](#PWR_NETS_VOLTAGES) - [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW) +- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK) - [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT) - [SEC_CHECK](#SEC_CHECK) - [SKIP_DETAILED_ROUTE](#SKIP_DETAILED_ROUTE) diff --git a/flow/scripts/final_outputs.tcl b/flow/scripts/final_outputs.tcl index 2cb34a0039..520b9d0374 100644 --- a/flow/scripts/final_outputs.tcl +++ b/flow/scripts/final_outputs.tcl @@ -35,6 +35,7 @@ if { # Read Spef for OpenSTA read_spef $::env(RESULTS_DIR)/6_final.spef + set final_parasitics rcx # Static IR drop analysis if { [env_var_exists_and_non_empty PWR_NETS_VOLTAGES] } { @@ -59,7 +60,9 @@ if { puts "OpenRCX is not enabled for this platform." puts "Falling back to global route-based estimates." log_cmd estimate_parasitics -global_routing + set final_parasitics global_routing } +write_endpoint_slack 6_final $final_parasitics report_cell_usage diff --git a/flow/scripts/final_report.tcl b/flow/scripts/final_report.tcl index 279dfdd891..2916f2f473 100644 --- a/flow/scripts/final_report.tcl +++ b/flow/scripts/final_report.tcl @@ -2,6 +2,7 @@ # (scripts/flow.tcl) can source the parts and perform the write itself # at the same point. source $::env(SCRIPTS_DIR)/final_connect.tcl -orfs_write_db $::env(RESULTS_DIR)/6_final.odb +# final_outputs.tcl writes the endpoint slack after parasitic extraction +orfs_write_db $::env(RESULTS_DIR)/6_final.odb 0 orfs_write_sdc $::env(RESULTS_DIR)/6_final.sdc source $::env(SCRIPTS_DIR)/final_outputs.tcl diff --git a/flow/scripts/util.tcl b/flow/scripts/util.tcl index 912762199a..49ab5fd80f 100644 --- a/flow/scripts/util.tcl +++ b/flow/scripts/util.tcl @@ -302,7 +302,101 @@ proc find_physical_only_masters { } { return $physical_only_masters } -proc orfs_write_db { output_file } { +proc endpoint_slack_file { step } { + return $::env(REPORTS_DIR)/${step}_endpoint_slack.json +} + +# Write the setup and hold slack of every endpoint at the end of a flow +# step so they can be compared across steps with +# flow/util/endpointSlack.py. Parasitics are estimated the same way as +# open.tcl so that every step is timed consistently, unless the caller +# has already loaded parasitics (e.g. extracted ones) and passes their +# name as parasitics. +proc write_endpoint_slack { step { parasitics "" } } { + if { !$::env(REPORT_ENDPOINT_SLACK) || [string match "*-failed" $step] } { + return + } + set filename [endpoint_slack_file $step] + puts "Writing endpoint slack to $filename" + + if { $parasitics == "" } { + set design_stage [lindex [split $step _] 0] + set parasitics none + if { $design_stage >= 5 } { + if { [grt::have_routes] } { + set parasitics global_routing + } + } elseif { $design_stage >= 3 } { + set parasitics placement + } + if { $parasitics != "none" } { + estimate_parasitics -$parasitics + } + } + # Update timing before sta::endpoints: the endpoint set is cached the + # first time it is requested and misses gated clock enable checks if the + # clock network has not been found yet. + sta::find_timing + + set fileId [open $filename w] + puts $fileId "{" + puts $fileId "\"design\": \"$::env(DESIGN_NAME)\"," + puts $fileId "\"step\": \"$step\"," + puts $fileId "\"parasitics\": \"$parasitics\"," + # Seconds per reported time unit + puts $fileId "\"time_unit\": [sta::time_ui_sta 1]," + # endpoint: [setup slack, hold slack], null if unconstrained + puts $fileId "\"endpoints\": {" + set sep "" + foreach pin [sta::endpoints] { + set name [string map {\\ \\\\ \" \\\"} [get_full_name $pin]] + set slacks {} + foreach min_max {max min} { + set slack [get_property $pin slack_$min_max] + if { $slack eq "INF" || $slack eq "-INF" } { + set slack null + } + lappend slacks $slack + } + puts -nonewline $fileId "$sep\"$name\": \[[join $slacks ,]\]" + set sep ",\n" + } + puts $fileId "\n}" + puts $fileId "}" + close $fileId +} + +# A step that copies its input .odb leaves the design unchanged, so +# re-time it if it is loaded, otherwise copy the input step's snapshot. +proc copy_endpoint_slack { input_file output_file } { + if { !$::env(REPORT_ENDPOINT_SLACK) } { + return + } + set step [file rootname [file tail $output_file]] + set chip [[ord::get_db] getChip] + if { $chip != "NULL" && [$chip getBlock] != "NULL" } { + write_endpoint_slack $step + return + } + set input [endpoint_slack_file [file rootname [file tail $input_file]]] + if { ![file exists $input] } { + return + } + set fileId [open $input r] + set json [read $fileId] + close $fileId + regsub {"step": "[^"]*"} $json "\"step\": \"$step\"" json + set fileId [open [endpoint_slack_file $step] w] + puts -nonewline $fileId $json + close $fileId +} + +# endpoint_slack 0 is for callers that write the endpoint slack snapshot +# themselves, e.g. 6_final after parasitic extraction. +proc orfs_write_db { output_file { endpoint_slack 1 } } { + if { $endpoint_slack } { + write_endpoint_slack [file rootname [file tail $output_file]] + } if { !$::env(WRITE_ODB_AND_SDC_EACH_STAGE) } { return } @@ -326,6 +420,7 @@ proc orfs_write_sdc { output_file } { # mtime with second resolution, which breaks make's timestamp-based # up-to-date checks for fast builds. proc orfs_copy_db { input_file output_file } { + copy_endpoint_slack $input_file $output_file if { !$::env(WRITE_ODB_AND_SDC_EACH_STAGE) } { return } diff --git a/flow/scripts/variables.json b/flow/scripts/variables.json index 36a165116c..61a7b8b8e7 100644 --- a/flow/scripts/variables.json +++ b/flow/scripts/variables.json @@ -954,6 +954,19 @@ "final" ] }, + "REPORT_ENDPOINT_SLACK": { + "default": 0, + "description": "If set to 1, write the setup and hold slack of every timing endpoint at the end of every flow step (e.g. 3_3_place_gp) to $REPORTS_DIR/_endpoint_slack.json. These snapshots can be compared across steps with flow/util/endpointSlack.py. See docs/contrib/EndpointSlackTracking.md.\n", + "stages": [ + "synth", + "floorplan", + "place", + "cts", + "grt", + "route", + "final" + ] + }, "ROUTING_LAYER_ADJUSTMENT": { "default": 0.5, "description": "Adjusts routing layer capacities to manage congestion and improve detailed routing. High values ease detailed routing but risk excessive detours and long global routing times, while low values reduce global routing failure but can complicate detailed routing. The global routing running time normally reduces dramatically (entirely design specific, but going from hours to minutes has been observed) when the value is low (such as 0.10). Sometimes, global routing will succeed with lower values and fail with higher values. Exploring results with different values can help shed light on the problem. Start with a too low value, such as 0.10, and bisect to value that works by doing multiple global routing runs. As a last resort, `make global_route_issue` and using the tools/OpenROAD/etc/whittle.py can be useful to debug global routing errors. If there is something specific that is impossible to route, such as a clock line over a macro, global routing will terminate with DRC errors routes that could have been routed were it not for the specific impossible routes. whittle.py should weed out the possible routes and leave a minimal failing case that pinpoints the problem.\n", diff --git a/flow/scripts/variables.yaml b/flow/scripts/variables.yaml index 644427f34c..2052bd468d 100644 --- a/flow/scripts/variables.yaml +++ b/flow/scripts/variables.yaml @@ -110,6 +110,22 @@ REPORT_CLOCK_SKEW: - route - final default: 1 +REPORT_ENDPOINT_SLACK: + description: > + If set to 1, write the setup and hold slack of every timing endpoint at + the end of every flow step (e.g. 3_3_place_gp) to + $REPORTS_DIR/_endpoint_slack.json. These snapshots can be compared + across steps with flow/util/endpointSlack.py. See + docs/contrib/EndpointSlackTracking.md. + stages: + - synth + - floorplan + - place + - cts + - grt + - route + - final + default: 0 SKIP_REPORT_METRICS: description: > If set to 1, then metrics, report_metrics does nothing. Useful to speed up diff --git a/flow/test/test_endpointSlack.py b/flow/test/test_endpointSlack.py new file mode 100644 index 0000000000..6ed498d590 --- /dev/null +++ b/flow/test/test_endpointSlack.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 + +import json +import os +import sys +import tempfile +import unittest +from io import StringIO +from unittest.mock import patch + +sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "util")) + +import endpointSlack + + +def snapshot(step, endpoints): + return { + "design": "test", + "step": step, + "parasitics": "placement", + "time_unit": 1e-12, + "endpoints": endpoints, + } + + +class TestEndpointSlack(unittest.TestCase): + def setUp(self): + self.tmp_dir = tempfile.TemporaryDirectory() + # slacks in ps: [setup, hold] + self.a_file = self.write( + "3_5_place_dp", + { + "r1/D": [-10.0, 5.0], # improves to passing + "r2/D": [20.0, 5.0], # degrades to failing + "r3/D": [30.0, 5.0], # unchanged + "r4/D": [40.0, 5.0], # removed in B + "out": [None, None], # unconstrained + }, + ) + self.b_file = self.write( + "4_1_cts", + { + "r1/D": [15.0, 4.0], + "r2/D": [-5.0, 6.0], + "r3/D": [30.0, None], # hold constrained only in A + "r5/D": [50.0, 5.0], # new in B + "out": [None, None], + "clkload0/Y": [None, None], # new but unconstrained + }, + ) + os.utime(self.a_file, (1, 1)) + os.utime(self.b_file, (2, 2)) + + def tearDown(self): + self.tmp_dir.cleanup() + + def write(self, step, endpoints): + f = os.path.join(self.tmp_dir.name, step + endpointSlack.SUFFIX) + with open(f, "w") as fh: + json.dump(snapshot(step, endpoints), fh) + return f + + def test_load(self): + a = endpointSlack.load_snapshot(self.a_file) + self.assertEqual(a["name"], "3_5_place_dp") + self.assertEqual(len(a["endpoints"]), 5) + self.assertNotIn("out", a["setup"]) + self.assertAlmostEqual(a["setup"]["r1/D"], -10e-12) + s = endpointSlack.summary(a["setup"]) + self.assertEqual(s["violations"], 1) + self.assertAlmostEqual(s["wns"], -10e-12) + self.assertAlmostEqual(s["tns"], -10e-12) + + def test_compare(self): + a = endpointSlack.load_snapshot(self.a_file) + b = endpointSlack.load_snapshot(self.b_file) + setup = endpointSlack.compare(a, b, "setup") + self.assertEqual([ep for ep, _ in setup["only_a"]], ["r4/D"]) + self.assertEqual([ep for ep, _ in setup["only_b"]], ["r5/D"]) + self.assertEqual(len(setup["both"]), 3) + m = endpointSlack.movement(setup["both"]) + self.assertEqual( + m, + { + "improved": 1, + "degraded": 1, + # r3/D is unchanged but positive in both, so not counted + "unchanged": 0, + "newly_failing": 1, + "newly_passing": 1, + }, + ) + hold = endpointSlack.compare(a, b, "hold") + self.assertEqual([ep for ep, _ in hold["constrained_only_a"]], ["r3/D"]) + self.assertEqual(len(hold["both"]), 2) + + def test_tolerance(self): + both = [("r1/D", -10e-12, -9.9e-12), ("r2/D", -10e-12, -12e-12)] + m = endpointSlack.movement(both, tolerance=0.5e-12) + self.assertEqual((m["improved"], m["degraded"], m["unchanged"]), (0, 1, 1)) + m = endpointSlack.movement(both) + self.assertEqual((m["improved"], m["degraded"], m["unchanged"]), (1, 1, 0)) + + def test_worst(self): + a = endpointSlack.load_snapshot(self.a_file) + b = endpointSlack.load_snapshot(self.b_file) + setup = endpointSlack.compare(a, b, "setup", worst=1) + # worst of A is r1/D, worst of B is r2/D + self.assertEqual(sorted(ep for ep, _, _ in setup["both"]), ["r1/D", "r2/D"]) + + def test_spearman(self): + self.assertAlmostEqual(endpointSlack.spearman([1, 2, 3], [10, 20, 30]), 1.0) + self.assertAlmostEqual(endpointSlack.spearman([1, 2, 3], [3, 2, 1]), -1.0) + self.assertEqual(endpointSlack._ranks([5, 1, 5, 3]), [3.5, 1.0, 3.5, 2.0]) + self.assertIsNone(endpointSlack.spearman([1], [1])) + self.assertIsNone(endpointSlack.spearman([1, 1], [1, 2])) + + def test_metrics(self): + a = endpointSlack.load_snapshot(self.a_file) + b = endpointSlack.load_snapshot(self.b_file) + m = endpointSlack.compare(a, b, "setup")["metrics"] + # r3/D is positive in both and excluded; r1/D: -10 -> 15, r2/D: 20 -> -5 + self.assertEqual(m["n"], 2) + self.assertAlmostEqual(m["spearman"], -1.0) + self.assertAlmostEqual(m["bias"], 0.0) + self.assertAlmostEqual(m["mae"], 25e-12) + self.assertAlmostEqual(m["max_abs"], 25e-12) + self.assertAlmostEqual(m["top_agreement"], 1.0) + self.assertEqual(m["violator_recall"], 0.0) + self.assertEqual(m["violator_precision"], 0.0) + # With K=1 the worst of A (r1/D) and of B (r2/D) differ + m = endpointSlack.compare(a, b, "setup", critical=1)["metrics"] + self.assertEqual(m["n"], 2) + self.assertAlmostEqual(m["top_agreement"], 0.0) + # No hold violations: nothing critical + m = endpointSlack.compare(a, b, "hold")["metrics"] + self.assertEqual(m["n"], 0) + self.assertIsNone(m["spearman"]) + self.assertIsNone(m["violator_recall"]) + + def test_index_row_class(self): + def row_class(a_eps, b_eps): + a = endpointSlack.load_snapshot(self.write("3_3_place_gp", a_eps)) + b = endpointSlack.load_snapshot(self.write("3_4_place_resized", b_eps)) + cmps = {c: endpointSlack.compare(a, b, c) for c in endpointSlack.CHECKS} + page = endpointSlack.track_index_html([(a, b, cmps)], 1e9, "ns", None) + body = page[page.index(" 4_1_cts", out) + with open(html_file) as f: + content = f.read() + self.assertIn("const DATA = {", content) + self.assertNotIn("__DATA__", content) + with open(csv_file) as f: + self.assertEqual(len(f.readlines()), 1 + 5 + 6) + self.assertEqual( + sorted(os.listdir(track_dir)), + ["3_5_place_dp__4_1_cts.html", "index.html"], + ) + with open(os.path.join(track_dir, "index.html")) as f: + self.assertIn("3_5_place_dp__4_1_cts.html", f.read()) + + +if __name__ == "__main__": + unittest.main() diff --git a/flow/util/endpointSlack.py b/flow/util/endpointSlack.py new file mode 100755 index 0000000000..3297d2825f --- /dev/null +++ b/flow/util/endpointSlack.py @@ -0,0 +1,961 @@ +#!/usr/bin/env python3 +"""Compare endpoint slack across flow steps. + +Snapshots are written at the end of every flow step (e.g. 3_3_place_gp) when +REPORT_ENDPOINT_SLACK=1, as $REPORTS_DIR/_endpoint_slack.json, with the +setup and hold slack of every timing endpoint. + +Examples: + endpointSlack.py list reports/nangate45/gcd/base + endpointSlack.py track reports/nangate45/gcd/base + endpointSlack.py compare A_endpoint_slack.json B_endpoint_slack.json -o cmp.html + endpointSlack.py export reports/nangate45/gcd/base -o slack.csv +""" + +import argparse +import csv +import glob +import html +import json +import os +import statistics +import sys + +CHECKS = ("setup", "hold") +UNITS = {"s": 1.0, "ns": 1e9, "ps": 1e12} +SUFFIX = "_endpoint_slack.json" + + +def load_snapshot(path): + """Load a snapshot file. + + Returns a dict with metadata, "endpoints" (all endpoint names) and per + check a dict endpoint -> slack in seconds for constrained endpoints. + """ + with open(path) as f: + data = json.load(f) + unit = data.get("time_unit", 1.0) + name = os.path.basename(path) + if name.endswith(SUFFIX): + name = name[: -len(SUFFIX)] + snapshot = { + "file": path, + "name": name, + "design": data.get("design", ""), + "step": data.get("step", name), + "parasitics": data.get("parasitics", ""), + "endpoints": set(data["endpoints"]), + } + for i, check in enumerate(CHECKS): + snapshot[check] = { + ep: slacks[i] * unit + for ep, slacks in data["endpoints"].items() + if slacks[i] is not None + } + return snapshot + + +def find_snapshots(reports_dir): + """Snapshot files in execution (modification time) order.""" + files = glob.glob(os.path.join(reports_dir, "*" + SUFFIX)) + return sorted(files, key=lambda f: (os.path.getmtime(f), f)) + + +def load_dir(reports_dir): + return [load_snapshot(f) for f in find_snapshots(reports_dir)] + + +def summary(slacks): + """WNS, TNS and number of violating endpoints of a slack dict.""" + values = list(slacks.values()) + negative = [s for s in values if s < 0] + return { + "count": len(values), + "wns": min(values) if values else None, + "tns": sum(negative), + "violations": len(negative), + } + + +def worst_endpoints(slacks, n): + return set(sorted(slacks, key=slacks.get)[:n]) + + +def _ranks(values): + """Ranks starting at 1, with tied values given their average rank.""" + order = sorted(range(len(values)), key=values.__getitem__) + ranks = [0.0] * len(values) + i = 0 + while i < len(order): + j = i + while j + 1 < len(order) and values[order[j + 1]] == values[order[i]]: + j += 1 + for k in range(i, j + 1): + ranks[order[k]] = (i + j) / 2 + 1 + i = j + 1 + return ranks + + +def spearman(x, y): + """Spearman rank correlation, None if undefined (n < 2 or constant).""" + if len(x) < 2: + return None + try: + return statistics.correlation(_ranks(x), _ranks(y)) + except statistics.StatisticsError: + return None + + +def _ratio(num, den): + return num / den if den else None + + +def metrics(sa, sb, critical=1000): + """Agreement between the slacks of two steps for one check. + + sa, sb: endpoint -> slack of the endpoints constrained in both steps. + The critical set is the union of the worst `critical` endpoints of A and + of B, excluding endpoints with positive slack in both steps, which are + not being optimized. Correlation and error are measured on it. + """ + top_a = worst_endpoints(sa, critical) + top_b = worst_endpoints(sb, critical) + crit = [ep for ep in top_a | top_b if sa[ep] <= 0 or sb[ep] <= 0] + x = [sa[ep] for ep in crit] + y = [sb[ep] for ep in crit] + deltas = [b - a for a, b in zip(x, y)] + viol_a = {ep for ep, s in sa.items() if s < 0} + viol_b = {ep for ep, s in sb.items() if s < 0} + return { + "critical": critical, + "n": len(crit), + "spearman": spearman(x, y), + "bias": statistics.mean(deltas) if deltas else None, + "mae": statistics.mean(abs(d) for d in deltas) if deltas else None, + "rmse": (statistics.mean(d * d for d in deltas) ** 0.5 if deltas else None), + "max_abs": max(abs(d) for d in deltas) if deltas else None, + # Fraction of the critical set in the worst `critical` of both steps + "top_agreement": _ratio( + sum(ep in top_a and ep in top_b for ep in crit), len(crit) + ), + # Of B's violators, the fraction already violating in A + "violator_recall": _ratio(len(viol_a & viol_b), len(viol_b)), + # Of A's violators, the fraction still violating in B + "violator_precision": _ratio(len(viol_a & viol_b), len(viol_a)), + } + + +def compare(a, b, check, worst=None, critical=1000): + """Compare one check (setup or hold) of two snapshots. + + Unconstrained endpoints (e.g. outputs of CTS dummy loads) are ignored. + Returns a dict with: + only_a / only_b: constrained endpoints in only one netlist, + [(endpoint, slack)] + constrained_only_a / constrained_only_b: endpoints in both netlists + but constrained in only one, [(endpoint, slack)] + both: [(endpoint, slack_a, slack_b)], restricted to the union of the + worst `worst` endpoints of A and of B if `worst` is given + metrics: metrics() of all endpoints constrained in both + """ + sa, sb = a[check], b[check] + both = [ep for ep in sa if ep in sb] + m = metrics({ep: sa[ep] for ep in both}, {ep: sb[ep] for ep in both}, critical) + if worst: + keep = worst_endpoints(sa, worst) | worst_endpoints(sb, worst) + both = [ep for ep in both if ep in keep] + return { + "only_a": [(ep, sa[ep]) for ep in sa if ep not in b["endpoints"]], + "only_b": [(ep, sb[ep]) for ep in sb if ep not in a["endpoints"]], + "constrained_only_a": [ + (ep, sa[ep]) for ep in sa if ep not in sb and ep in b["endpoints"] + ], + "constrained_only_b": [ + (ep, sb[ep]) for ep in sb if ep not in sa and ep in a["endpoints"] + ], + "both": [(ep, sa[ep], sb[ep]) for ep in both], + "metrics": m, + } + + +def movement(both, tolerance=0.0): + """Counts of how endpoint slack changed between A and B. + + improved/degraded/unchanged exclude endpoints with positive slack in + both steps, which are not being optimized. Changes of at most + `tolerance` (seconds) count as unchanged. + """ + crit = [(s_a, s_b) for _, s_a, s_b in both if s_a <= 0 or s_b <= 0] + return { + "improved": sum(s_b - s_a > tolerance for s_a, s_b in crit), + "degraded": sum(s_a - s_b > tolerance for s_a, s_b in crit), + "unchanged": sum(abs(s_b - s_a) <= tolerance for s_a, s_b in crit), + "newly_failing": sum(s_a >= 0 > s_b for _, s_a, s_b in both), + "newly_passing": sum(s_b >= 0 > s_a for _, s_a, s_b in both), + } + + +# Digits after the decimal point for each time unit +DIGITS = {"s": 12, "ns": 3, "ps": 1} + + +def _fmt(value, scale, digits): + return "-" if value is None else f"{value * scale:.{digits}f}" + + +def _num(value, scale): + return None if value is None else float(f"{value * scale:.6g}") + + +def _label(snapshot): + """Step name, with the parasitics model the step was timed with.""" + if snapshot["parasitics"]: + return f"{snapshot['name']} ({snapshot['parasitics']} parasitics)" + return snapshot["name"] + + +def _summary_lines(cmp, check, scale, unit, digits, tolerance=0.0): + m = movement(cmp["both"], tolerance) + lines = [ + f"{check:5s} both={len(cmp['both'])} onlyA={len(cmp['only_a'])}" + f" onlyB={len(cmp['only_b'])}" + f" constrained only in A={len(cmp['constrained_only_a'])}" + f" only in B={len(cmp['constrained_only_b'])}", + f" not positive in both: improved={m['improved']}" + f" degraded={m['degraded']} unchanged={m['unchanged']}" + f" newly failing={m['newly_failing']}" + f" newly passing={m['newly_passing']}", + ] + if cmp["both"]: + deltas = [(s_b - s_a) * scale for _, s_a, s_b in cmp["both"]] + lines.append( + f" slack delta (B-A) {unit}:" + f" mean={statistics.mean(deltas):.{digits}f}" + f" median={statistics.median(deltas):.{digits}f}" + f" min={min(deltas):.{digits}f} max={max(deltas):.{digits}f}" + ) + mt = cmp["metrics"] + lines.append( + f" critical (worst {mt['critical']} of A or B, not positive in" + f" both): n={mt['n']} spearman={_fmt_num(mt['spearman'])}" + f" top agreement={_fmt_num(mt['top_agreement'])}" + ) + lines.append( + f" {unit}: bias={_fmt(mt['bias'], scale, digits)}" + f" MAE={_fmt(mt['mae'], scale, digits)}" + f" RMSE={_fmt(mt['rmse'], scale, digits)}" + f" max|delta|={_fmt(mt['max_abs'], scale, digits)}" + ) + lines.append( + f" violators: recall={_fmt_num(mt['violator_recall'])}" + f" precision={_fmt_num(mt['violator_precision'])}" + ) + return lines + + +def _fmt_num(value): + return "-" if value is None else f"{value:.3f}" + + +def cmd_list(args): + scale = UNITS[args.unit] + header = f"{'step':32s} {'parasitics':14s} {'endpoints':>9s}" + for check in CHECKS: + header += f" {check + ' WNS':>10s} {check + ' TNS':>10s} {'#viol':>6s}" + print(header + f" ({args.unit})") + for s in load_dir(args.reports_dir): + line = f"{s['name']:32s} {s['parasitics']:14s} {len(s['endpoints']):9d}" + for check in CHECKS: + sm = summary(s[check]) + line += ( + f" {_fmt(sm['wns'], scale, args.digits):>10s}" + f" {_fmt(sm['tns'], scale, args.digits):>10s}" + f" {sm['violations']:6d}" + ) + print(line) + + +def _pair_file(a, b): + return f"{a['name']}__{b['name']}.html" + + +def cmd_track(args): + scale = UNITS[args.unit] + snaps = load_dir(args.reports_dir) + pairs = list(zip(snaps, snaps[1:])) + if args.html: + os.makedirs(args.html, exist_ok=True) + rows = [] + for i, (a, b) in enumerate(pairs): + print(f"{_label(a)} -> {_label(b)}") + cmps = { + check: compare(a, b, check, args.worst, args.critical) for check in CHECKS + } + for check in CHECKS: + for line in _summary_lines( + cmps[check], check, scale, args.unit, args.digits, args.tolerance + ): + print(" " + line) + rows.append((a, b, cmps)) + if args.html: + links = [("index", "index.html")] + if i > 0: + links.insert(0, ("previous", _pair_file(*pairs[i - 1]))) + if i + 1 < len(pairs): + links.append(("next", _pair_file(*pairs[i + 1]))) + page = compare_html( + a, + b, + cmps, + scale, + args.unit, + args.worst, + links, + args.digits, + args.tolerance, + ) + with open(os.path.join(args.html, _pair_file(a, b)), "w") as f: + f.write(page) + if args.html: + index = os.path.join(args.html, "index.html") + with open(index, "w") as f: + f.write( + track_index_html( + rows, + scale, + args.unit, + args.worst, + args.critical, + args.digits, + args.tolerance, + ) + ) + print(f"Wrote {len(pairs)} comparisons to {args.html}, see {index}") + + +def track_index_html(rows, scale, unit, worst, critical=1000, digits=3, tolerance=0.0): + """Index of step-to-step comparisons with a summary of each.""" + design = rows[0][0]["design"] if rows else "" + + # Cells are strings, or (before, after) pairs shown as "before -> after" + table = [] + for a, b, cmps in rows: + # Endpoints with positive slack in both steps are not considered + changed = considered = improved = 0 + cells = [] + for check in CHECKS: + cmp = cmps[check] + m = movement(cmp["both"], tolerance) + sa, sb = summary(a[check]), summary(b[check]) + added_removed = sum( + s <= 0 + for key in ( + "only_a", + "only_b", + "constrained_only_a", + "constrained_only_b", + ) + for _, s in cmp[key] + ) + changed += m["improved"] + m["degraded"] + added_removed + considered += m["improved"] + m["degraded"] + m["unchanged"] + considered += added_removed + improved += m["improved"] + cells += [ + (_fmt(sa["wns"], scale, digits), _fmt(sb["wns"], scale, digits)), + (_fmt(sa["tns"], scale, digits), _fmt(sb["tns"], scale, digits)), + str(m["improved"]), + str(m["degraded"]), + str(m["newly_failing"]), + str(m["newly_passing"]), + _fmt_num(cmp["metrics"]["spearman"]), + _fmt_num(cmp["metrics"]["top_agreement"]), + _fmt(cmp["metrics"]["mae"], scale, digits), + ] + if not changed: + row_class = "same" + elif improved == considered: + row_class = "better" + else: + row_class = "" + table.append((a, b, row_class, cells)) + + # Pad both sides of the pairs to the column width so they line up + widths = {} + for _, _, _, cells in table: + for i, c in enumerate(cells): + if isinstance(c, tuple): + w = widths.get(i, (0, 0)) + widths[i] = (max(w[0], len(c[0])), max(w[1], len(c[1]))) + + def cell(i, c): + if isinstance(c, tuple): + wa, wb = widths[i] + c = ( + f"{html.escape(c[0])}" + f" → " + f"{html.escape(c[1])}" + ) + else: + c = html.escape(c) + return f"{c}" + + body = "" + for a, b, row_class, cells in table: + link = ( + f"" + f"{html.escape(a['name'])} → {html.escape(b['name'])}" + ) + parasitics = a["parasitics"] + if b["parasitics"] != a["parasitics"]: + parasitics += " → " + b["parasitics"] + cls = f" class='{row_class}'" if row_class else "" + tds = "".join(cell(i, c) for i, c in enumerate(cells)) + body += f"{link}{parasitics}{tds}\n" + head = "".join(f"{check}" for check in CHECKS) + sub = "".join( + "WNSTNSimproveddegraded" + "newly failingnewly passing" + "Spearmantop agreementMAE" + for _ in CHECKS + ) + note = ( + f" Only the worst {worst} endpoints of each pair are compared." if worst else "" + ) + return ( + TRACK_INDEX_TEMPLATE.replace( + "__TITLE__", html.escape(f"{design} endpoint slack by step") + ) + .replace("__UNIT__", unit) + .replace("__NOTE__", html.escape(note)) + .replace("__CRITICAL__", str(critical)) + .replace("__TOL__", f"{tolerance * scale:g} {unit}") + .replace("__HEAD__", head) + .replace("__SUB__", sub) + .replace("__BODY__", body) + ) + + +TRACK_INDEX_TEMPLATE = """ +__TITLE__ + +

__TITLE__

+
Each row compares a step with the next one (__UNIT__). +Endpoints with positive slack in both steps are not considered, and slack +changes of at most __TOL__ count as unchanged. Grey rows: +no considered endpoint changed. Green rows: every considered endpoint +improved (setup and hold). Spearman, top agreement and MAE are +measured on the critical set: the worst __CRITICAL__ endpoints of either step, +excluding those with positive slack in both.__NOTE__
+
+__HEAD__ +__SUB__ +__BODY__
stepsparasitics
+ +""" + + +def cmd_compare(args): + scale = UNITS[args.unit] + a = load_snapshot(args.a) + b = load_snapshot(args.b) + print(f"A: {_label(a)}\nB: {_label(b)}") + if args.worst: + print(f"Endpoints in both restricted to the worst {args.worst} of A or B") + cmps = {check: compare(a, b, check, args.worst, args.critical) for check in CHECKS} + for check in CHECKS: + for line in _summary_lines( + cmps[check], check, scale, args.unit, args.digits, args.tolerance + ): + print(line) + if args.output: + with open(args.output, "w") as f: + f.write( + compare_html( + a, + b, + cmps, + scale, + args.unit, + args.worst, + digits=args.digits, + tolerance=args.tolerance, + ) + ) + print(f"Wrote {args.output}") + + +def cmd_export(args): + scale = UNITS[args.unit] + out = open(args.output, "w", newline="") if args.output else sys.stdout + writer = csv.writer(out) + writer.writerow( + [ + "order", + "step", + "parasitics", + "endpoint", + f"setup_{args.unit}", + f"hold_{args.unit}", + ] + ) + for order, s in enumerate(load_dir(args.reports_dir)): + for ep in sorted(s["endpoints"]): + writer.writerow( + [ + order, + s["step"], + s["parasitics"], + ep, + _num(s["setup"].get(ep), scale), + _num(s["hold"].get(ep), scale), + ] + ) + if args.output: + out.close() + + +def compare_html(a, b, cmps, scale, unit, worst, links=(), digits=3, tolerance=0.0): + """Self-contained interactive HTML for a two-snapshot comparison. + + links: optional (label, href) navigation links shown above the title. + """ + data = {} + for check, cmp in cmps.items(): + data[check] = { + "both": [ + [ep, _num(s_a, scale), _num(s_b, scale)] for ep, s_a, s_b in cmp["both"] + ], + "only_a": sorted( + ([ep, _num(s, scale)] for ep, s in cmp["only_a"]), key=lambda r: r[0] + ), + "only_b": sorted( + ([ep, _num(s, scale)] for ep, s in cmp["only_b"]), key=lambda r: r[0] + ), + "constrained_only_a": len(cmp["constrained_only_a"]), + "constrained_only_b": len(cmp["constrained_only_b"]), + "summary_a": { + k: _num(v, scale) if k in ("wns", "tns") else v + for k, v in summary(a[check]).items() + }, + "summary_b": { + k: _num(v, scale) if k in ("wns", "tns") else v + for k, v in summary(b[check]).items() + }, + "metrics": { + k: _num(v, scale) if k in ("bias", "mae", "rmse", "max_abs") else v + for k, v in cmp["metrics"].items() + }, + } + title = f"{a['design']} {a['name']} vs {b['name']}" + note = f" Only the worst {worst} endpoints of A or B are plotted." if worst else "" + return ( + HTML_TEMPLATE.replace("__TITLE__", html.escape(title)) + .replace("__A__", html.escape(a["name"])) + .replace("__B__", html.escape(b["name"])) + .replace("__PAR_A__", html.escape(a["parasitics"] or "unknown")) + .replace("__PAR_B__", html.escape(b["parasitics"] or "unknown")) + .replace("__NOTE__", html.escape(note)) + .replace( + "__NAV__", + " | ".join( + f"{html.escape(label)}" + for label, href in links + ), + ) + .replace("__UNIT__", unit) + .replace("__DIGITS__", str(digits)) + .replace("__TOL__", repr(tolerance * scale)) + .replace("__DATA__", json.dumps(data).replace(" +__TITLE__ + + +

__TITLE__

+
Endpoint slack: x = A (__A__), y = B (__B__), __UNIT__. +Parasitics: A __PAR_A__, B __PAR_B__.__NOTE__ +Drag to zoom, double-click to reset.
+
+ +
+
+
Improved and degraded exclude endpoints with positive +slack in both steps.
+
+
+
+
+

Most degraded

+
Excluding endpoints with positive slack in both
+
+

Most improved

+
Excluding endpoints with positive slack in both
+
+
+
+

Endpoints only in A (__A__)

+

Endpoints only in B (__B__)

+
+ + +""" + + +def main(argv=None): + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + sub = parser.add_subparsers(dest="cmd", required=True) + + def common(p, worst=True): + p.add_argument("--unit", choices=UNITS, default="ns") + p.add_argument( + "--digits", + type=int, + help="digits after the decimal point for times" + " (default: 3 for ns, 1 for ps)", + ) + p.add_argument( + "--tolerance", + type=float, + help="slack changes of at most this much (in --unit) count as" + " unchanged (default: half the last displayed digit, so a change" + " shown as 0 is unchanged)", + ) + if worst: + p.add_argument( + "--worst", + type=int, + metavar="N", + help="only compare the worst N endpoints of A or B", + ) + p.add_argument( + "--critical", + type=int, + default=1000, + metavar="K", + help="correlation and error are measured on the worst K" + " endpoints of A or B, excluding those with positive slack in" + " both (default 1000)", + ) + + p = sub.add_parser("list", help="list snapshots with WNS/TNS per step") + p.add_argument("reports_dir") + common(p, worst=False) + p.set_defaults(func=cmd_list) + + p = sub.add_parser("track", help="compare consecutive snapshots") + p.add_argument("reports_dir") + p.add_argument( + "--html", + metavar="DIR", + help="write an HTML scatter for every step vs. its successor and an" + " index.html to DIR", + ) + common(p) + p.set_defaults(func=cmd_track) + + p = sub.add_parser("compare", help="compare two snapshots") + p.add_argument("a", help="earlier snapshot (x axis)") + p.add_argument("b", help="later snapshot (y axis)") + p.add_argument("-o", "--output", help="write interactive HTML scatter") + common(p) + p.set_defaults(func=cmd_compare) + + p = sub.add_parser("export", help="flatten all snapshots to CSV") + p.add_argument("reports_dir") + p.add_argument("-o", "--output", help="CSV file (default stdout)") + common(p, worst=False) + p.set_defaults(func=cmd_export) + + args = parser.parse_args(argv) + if args.digits is None: + args.digits = DIGITS[args.unit] + if args.tolerance is None: + args.tolerance = 0.5 * 10**-args.digits + # Internally slack is in seconds + args.tolerance /= UNITS[args.unit] + args.func(args) + + +if __name__ == "__main__": + main()