Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions docs/contrib/EndpointSlackTracking.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
# Tracking Endpoint Slack Across Flow Steps

When working on an optimization (resizing, buffering, CTS, repair timing,
...) it is useful to see how the slack of each timing endpoint moves from
one flow step to the next, not just how WNS/TNS change. ORFS can record the
setup and hold slack of every endpoint at the end of every flow step
(`1_synth`, `2_1_floorplan`, ..., `3_3_place_gp`, ..., `6_final`) and compare
any two steps, including a scatter plot of slack (x: earlier step, y: later
step).

## Recording

Set `REPORT_ENDPOINT_SLACK=1`:

``` shell
cd flow
make DESIGN_CONFIG=designs/nangate45/gcd/config.mk REPORT_ENDPOINT_SLACK=1
```

Every step then writes `$(REPORTS_DIR)/<step>_endpoint_slack.json`, e.g.
`3_3_place_gp_endpoint_slack.json`. The snapshot is taken by
`write_endpoint_slack` in `flow/scripts/util.tcl`, called from
`orfs_write_db` so the step name is the name of the `.odb` the step writes.
It works in the single-process `flow/scripts/flow.tcl` flow as well. Steps
that just copy their input (`orfs_copy_db`, e.g. `5_3_fillcell` without fill
cells) re-time the design if it is loaded, or else copy the previous step's
snapshot. Intermediate `.odb` files written through `orfs_write_db` (e.g. CTS
`save_progress`) also get a snapshot; `*-failed` ones don't.

So that all steps are timed the same way, parasitics are estimated before
reporting as in `open.tcl`, based on the step number:

| Steps | Parasitics |
| --- | --- |
| 1, 2 | none (cells are not placed) |
| 3, 4 | `estimate_parasitics -placement` |
| 5, `6_1_fill` | `estimate_parasitics -global_routing` if global routes exist |
| `6_final` | `rcx`: the extracted `6_final.spef`, as in `6_finish.rpt` (`global_routing` on platforms without `RCX_RULES`) |

The parasitics model is recorded in each snapshot and shown by
`endpointSlack.py`. Slack changes between steps with different models (e.g.
`2_4_floorplan_pdn` to `3_1_place_gp_skip_io`, `4_1_cts` to `5_1_grt`, or
`6_1_fill` to `6_final`) include the change of model. Detailed routing is
only extracted in `6_final`, so `5_2_route` to `6_1_fill` show global routing
estimates.

Each file contains `design`, `step`, `parasitics`, `time_unit` (seconds per
time unit of the values) and `endpoints`: a map from every endpoint pin
(`sta::endpoints`) to `[setup slack, hold slack]`, `null` where the endpoint
is unconstrained. The slack is the pin's worst slack over all path groups
and clocks (`get_property $pin slack_max|slack_min`). On asap7/swerv_wrapper
(about 30k endpoints) a snapshot is about 1 MB and takes about 10 s, mostly
the timing update.

## Analysis

`flow/util/endpointSlack.py` only uses the Python standard library.

``` shell
R=reports/nangate45/gcd/base

# Steps in execution order with parasitics, WNS, TNS and violation counts
python3 util/endpointSlack.py list $R

# Compare consecutive steps across the whole run; --html also writes a
# scatter for every step vs. its successor and an index.html to a directory
python3 util/endpointSlack.py track $R --html track

# Compare two steps; -o writes an interactive HTML scatter
python3 util/endpointSlack.py compare \
$R/3_5_place_dp_endpoint_slack.json $R/4_1_cts_endpoint_slack.json \
--unit ps -o compare.html

# Flatten all snapshots into one CSV (one row per endpoint per step)
python3 util/endpointSlack.py export $R -o slack.csv
```

For setup and hold, `compare` reports:

- endpoints in both steps, and how many improved, degraded, stayed the
same, became violating (`newly failing`) or stopped violating
(`newly passing`), with statistics of the slack change (B - A).
Improved, degraded and unchanged exclude endpoints with positive slack in
both steps, which are not being optimized;
- endpoints only in A or only in B, i.e. added or removed from the netlist;
- endpoints in both netlists but constrained in only one of them;
- how well the two steps agree on the critical endpoints (see below).

Unconstrained endpoints, such as the outputs of CTS dummy load cells, are
ignored.

### Agreement metrics

Correlation over all endpoints is dominated by endpoints with large positive
slack and is close to 1 even when the critical endpoints change a lot (0.9997
for setup on asap7/swerv_wrapper from `3_5_place_dp` to `4_1_cts`). The
metrics are therefore measured on a critical set: the worst K endpoints of A
or of B (`--critical K`, default 1000), excluding endpoints with positive
slack in both steps, which are not being optimized. When fewer than K
endpoints violate, the critical set is the endpoints violating in either
step; when nothing violates (often hold) it is empty and the metrics are
reported as `-`.

| Metric | Meaning |
| --- | --- |
| `spearman` | Spearman rank correlation of A and B slack (ties get average ranks): is the order of the critical endpoints preserved? |
| `top agreement` | Fraction of the critical set that is in the worst K of both steps. |
| `bias`, `MAE`, `RMSE`, `max\|delta\|` | Mean, mean absolute, root mean square and maximum absolute slack change B - A, i.e. distance from the y = x line. |
| `violator recall` | Of the endpoints violating in B, the fraction already violating in A (over all endpoints). |
| `violator precision` | Of the endpoints violating in A, the fraction still violating in B (over all endpoints). |

They are printed by `compare` and `track`, shown in the HTML stats, and
the `track` index has Spearman, top agreement and MAE columns for every step
transition.

Times are shown in `--unit` (default ns) with a fixed number of digits after
the decimal point (3 for ns, 1 for ps, or `--digits`), in a fixed-width font
in the HTML.
Slack changes smaller than half the last displayed digit (0.5 ps with the
default ns and 3 digits), or `--tolerance` in `--unit`, count as unchanged
for the improved/degraded counts, the tables of most degraded and improved
endpoints and the `track` row colors. Otherwise numerical noise, e.g. a few
femtoseconds between `5_1_grt` and `5_2_route`, shows up as changes that
display as 0.

`--worst N` restricts the endpoints in both to the union of the worst N
endpoints of A and of B, to focus on the critical part of the design.

The HTML is self-contained (no network access needed). It has:

- WNS, TNS and violation counts of both steps;
- a scatter of endpoint slack with a y = x line, where above the line means
the endpoint improved. It is drawn on a canvas so it handles hundreds of
thousands of endpoints. Drag to zoom, double-click to reset, hover for
the endpoint name;
- a setup/hold toggle;
- tables of the most degraded and most improved endpoints, excluding
endpoints with positive slack in both steps, and of the endpoints only in
one step.

`track --html DIR` writes the same page for every step and its successor
(`<step>__<next step>.html`, with previous/next links) and `DIR/index.html`,
a table of every transition with WNS/TNS and the improved, degraded, newly
failing and newly passing counts for setup and hold. Endpoints with positive
slack in both steps are not considered. Transitions where no considered
endpoint changed (e.g. `2_2_floorplan_macro` to `2_3_floorplan_tapcell`) are
greyed out, and transitions where every considered endpoint improved, for
setup and hold, are green.

For other analyses, import the module (e.g. from a notebook):
`load_snapshot()`, `load_dir()`, `summary()`, `compare()`, `movement()`,
`metrics()` and `spearman()`
return plain Python dicts and lists.

## Testing

`flow/test/test_endpointSlack.py` covers loading and comparison with
synthetic snapshots and runs in the util CI job:

``` shell
python3 flow/test/test_endpointSlack.py
```
2 changes: 2 additions & 0 deletions docs/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ entries:
title: Guide to Integrate a New Platform into the OpenROAD Flow
- file: contrib/CI.md
title: CI Guide
- file: contrib/EndpointSlackTracking.md
title: Tracking Endpoint Slack Across Flow Steps

- file: contrib/CODE_OF_CONDUCT.md
title: Code of Conduct
Expand Down
8 changes: 8 additions & 0 deletions docs/user/FlowVariables.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,7 @@ configuration file.
| <a name="REMOVE_CELLS_FOR_LEC"></a>REMOVE_CELLS_FOR_LEC| String patterns directly passed to write_verilog -remove_cells <> for lec checks.| |
| <a name="REPAIR_PDN_VIA_LAYER"></a>REPAIR_PDN_VIA_LAYER| Remove power grid vias which generate DRC violations after detailed routing.| |
| <a name="REPORT_CLOCK_SKEW"></a>REPORT_CLOCK_SKEW| Report clock skew as part of reporting metrics, starting at CTS, before which there is no clock skew. This metric can be quite time-consuming, so it can be useful to disable.| 1|
| <a name="REPORT_ENDPOINT_SLACK"></a>REPORT_ENDPOINT_SLACK| If set to 1, write the setup and hold slack of every timing endpoint at the end of every flow step (e.g. 3_3_place_gp) to $REPORTS_DIR/<step>_endpoint_slack.json. These snapshots can be compared across steps with flow/util/endpointSlack.py. See docs/contrib/EndpointSlackTracking.md.| 0|
| <a name="ROUTING_LAYER_ADJUSTMENT"></a>ROUTING_LAYER_ADJUSTMENT| Adjusts routing layer capacities to manage congestion and improve detailed routing. High values ease detailed routing but risk excessive detours and long global routing times, while low values reduce global routing failure but can complicate detailed routing. The global routing running time normally reduces dramatically (entirely design specific, but going from hours to minutes has been observed) when the value is low (such as 0.10). Sometimes, global routing will succeed with lower values and fail with higher values. Exploring results with different values can help shed light on the problem. Start with a too low value, such as 0.10, and bisect to value that works by doing multiple global routing runs. As a last resort, `make global_route_issue` and using the tools/OpenROAD/etc/whittle.py can be useful to debug global routing errors. If there is something specific that is impossible to route, such as a clock line over a macro, global routing will terminate with DRC errors routes that could have been routed were it not for the specific impossible routes. whittle.py should weed out the possible routes and leave a minimal failing case that pinpoints the problem.| 0.5|
| <a name="RTLMP_AREA_WT"></a>RTLMP_AREA_WT| Weight for the area of the current floorplan.| 0.1|
| <a name="RTLMP_ARGS"></a>RTLMP_ARGS| Overrides all other RTL macro placer arguments.| |
Expand Down Expand Up @@ -354,6 +355,7 @@ configuration file.
- [POST_SYNTH_TCL](#POST_SYNTH_TCL)
- [POS_CLKGATE_AND_PORTS](#POS_CLKGATE_AND_PORTS)
- [PRE_SYNTH_TCL](#PRE_SYNTH_TCL)
- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK)
- [SDC_FILE](#SDC_FILE)
- [SEC_CHECK](#SEC_CHECK)
- [SKIP_REPORT_METRICS](#SKIP_REPORT_METRICS)
Expand Down Expand Up @@ -425,6 +427,7 @@ configuration file.
- [PRE_PDN_TCL](#PRE_PDN_TCL)
- [PRE_TAPCELL_TCL](#PRE_TAPCELL_TCL)
- [REMOVE_ABC_BUFFERS](#REMOVE_ABC_BUFFERS)
- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK)
- [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT)
- [RTLMP_AREA_WT](#RTLMP_AREA_WT)
- [RTLMP_ARGS](#RTLMP_ARGS)
Expand Down Expand Up @@ -502,6 +505,7 @@ configuration file.
- [PRE_GLOBAL_PLACE_TCL](#PRE_GLOBAL_PLACE_TCL)
- [PRE_IO_PLACEMENT_TCL](#PRE_IO_PLACEMENT_TCL)
- [PRE_RESIZE_TCL](#PRE_RESIZE_TCL)
- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK)
- [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT)
- [SKIP_REPORT_METRICS](#SKIP_REPORT_METRICS)
- [TNS_END_PERCENT](#TNS_END_PERCENT)
Expand All @@ -525,6 +529,7 @@ configuration file.
- [POST_CTS_TCL](#POST_CTS_TCL)
- [PRE_CTS_TCL](#PRE_CTS_TCL)
- [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW)
- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK)
- [SETUP_MOVE_SEQUENCE](#SETUP_MOVE_SEQUENCE)
- [SETUP_SLACK_MARGIN](#SETUP_SLACK_MARGIN)
- [SKIP_BUFFER_REMOVAL](#SKIP_BUFFER_REMOVAL)
Expand Down Expand Up @@ -554,6 +559,7 @@ configuration file.
- [POST_GLOBAL_ROUTE_TCL](#POST_GLOBAL_ROUTE_TCL)
- [PRE_GLOBAL_ROUTE_TCL](#PRE_GLOBAL_ROUTE_TCL)
- [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW)
- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK)
- [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT)
- [SETUP_MOVE_SEQUENCE](#SETUP_MOVE_SEQUENCE)
- [SETUP_SLACK_MARGIN](#SETUP_SLACK_MARGIN)
Expand Down Expand Up @@ -587,6 +593,7 @@ configuration file.
- [PRE_DETAIL_ROUTE_TCL](#PRE_DETAIL_ROUTE_TCL)
- [PRE_FILLCELL_TCL](#PRE_FILLCELL_TCL)
- [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW)
- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK)
- [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT)
- [SKIP_ANTENNA_REPAIR_POST_DRT](#SKIP_ANTENNA_REPAIR_POST_DRT)
- [SKIP_DETAILED_ROUTE](#SKIP_DETAILED_ROUTE)
Expand All @@ -610,6 +617,7 @@ configuration file.
- [PRE_FINAL_REPORT_TCL](#PRE_FINAL_REPORT_TCL)
- [PWR_NETS_VOLTAGES](#PWR_NETS_VOLTAGES)
- [REPORT_CLOCK_SKEW](#REPORT_CLOCK_SKEW)
- [REPORT_ENDPOINT_SLACK](#REPORT_ENDPOINT_SLACK)
- [ROUTING_LAYER_ADJUSTMENT](#ROUTING_LAYER_ADJUSTMENT)
- [SEC_CHECK](#SEC_CHECK)
- [SKIP_DETAILED_ROUTE](#SKIP_DETAILED_ROUTE)
Expand Down
3 changes: 3 additions & 0 deletions flow/scripts/final_outputs.tcl
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ if {

# Read Spef for OpenSTA
read_spef $::env(RESULTS_DIR)/6_final.spef
set final_parasitics rcx

# Static IR drop analysis
if { [env_var_exists_and_non_empty PWR_NETS_VOLTAGES] } {
Expand All @@ -59,7 +60,9 @@ if {
puts "OpenRCX is not enabled for this platform."
puts "Falling back to global route-based estimates."
log_cmd estimate_parasitics -global_routing
set final_parasitics global_routing
}
write_endpoint_slack 6_final $final_parasitics

report_cell_usage

Expand Down
3 changes: 2 additions & 1 deletion flow/scripts/final_report.tcl
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# (scripts/flow.tcl) can source the parts and perform the write itself
# at the same point.
source $::env(SCRIPTS_DIR)/final_connect.tcl
orfs_write_db $::env(RESULTS_DIR)/6_final.odb
# final_outputs.tcl writes the endpoint slack after parasitic extraction
orfs_write_db $::env(RESULTS_DIR)/6_final.odb 0
orfs_write_sdc $::env(RESULTS_DIR)/6_final.sdc
source $::env(SCRIPTS_DIR)/final_outputs.tcl
97 changes: 96 additions & 1 deletion flow/scripts/util.tcl
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,101 @@ proc find_physical_only_masters { } {
return $physical_only_masters
}

proc orfs_write_db { output_file } {
proc endpoint_slack_file { step } {
return $::env(REPORTS_DIR)/${step}_endpoint_slack.json
}

# Write the setup and hold slack of every endpoint at the end of a flow
# step so they can be compared across steps with
# flow/util/endpointSlack.py. Parasitics are estimated the same way as
# open.tcl so that every step is timed consistently, unless the caller
# has already loaded parasitics (e.g. extracted ones) and passes their
# name as parasitics.
proc write_endpoint_slack { step { parasitics "" } } {
if { !$::env(REPORT_ENDPOINT_SLACK) || [string match "*-failed" $step] } {
Comment thread
maliberty marked this conversation as resolved.
Resolved
return
}
set filename [endpoint_slack_file $step]
puts "Writing endpoint slack to $filename"

if { $parasitics == "" } {
set design_stage [lindex [split $step _] 0]
set parasitics none
if { $design_stage >= 5 } {
if { [grt::have_routes] } {
set parasitics global_routing
}
} elseif { $design_stage >= 3 } {
set parasitics placement
}
if { $parasitics != "none" } {
estimate_parasitics -$parasitics
}
}
# Update timing before sta::endpoints: the endpoint set is cached the
# first time it is requested and misses gated clock enable checks if the
# clock network has not been found yet.
sta::find_timing

set fileId [open $filename w]
puts $fileId "{"
puts $fileId "\"design\": \"$::env(DESIGN_NAME)\","
puts $fileId "\"step\": \"$step\","
puts $fileId "\"parasitics\": \"$parasitics\","
# Seconds per reported time unit
puts $fileId "\"time_unit\": [sta::time_ui_sta 1],"
# endpoint: [setup slack, hold slack], null if unconstrained
puts $fileId "\"endpoints\": {"
set sep ""
foreach pin [sta::endpoints] {
set name [string map {\\ \\\\ \" \\\"} [get_full_name $pin]]
set slacks {}
foreach min_max {max min} {
set slack [get_property $pin slack_$min_max]
if { $slack eq "INF" || $slack eq "-INF" } {
set slack null
}
lappend slacks $slack
}
puts -nonewline $fileId "$sep\"$name\": \[[join $slacks ,]\]"
set sep ",\n"
}
puts $fileId "\n}"
puts $fileId "}"
close $fileId
}

# A step that copies its input .odb leaves the design unchanged, so
# re-time it if it is loaded, otherwise copy the input step's snapshot.
proc copy_endpoint_slack { input_file output_file } {
if { !$::env(REPORT_ENDPOINT_SLACK) } {
Comment thread
maliberty marked this conversation as resolved.
Resolved
return
}
set step [file rootname [file tail $output_file]]
set chip [[ord::get_db] getChip]
if { $chip != "NULL" && [$chip getBlock] != "NULL" } {
write_endpoint_slack $step
return
}
set input [endpoint_slack_file [file rootname [file tail $input_file]]]
if { ![file exists $input] } {
return
}
set fileId [open $input r]
set json [read $fileId]
close $fileId
regsub {"step": "[^"]*"} $json "\"step\": \"$step\"" json
set fileId [open [endpoint_slack_file $step] w]
puts -nonewline $fileId $json
close $fileId
}

# endpoint_slack 0 is for callers that write the endpoint slack snapshot
# themselves, e.g. 6_final after parasitic extraction.
proc orfs_write_db { output_file { endpoint_slack 1 } } {
if { $endpoint_slack } {
write_endpoint_slack [file rootname [file tail $output_file]]
}
if { !$::env(WRITE_ODB_AND_SDC_EACH_STAGE) } {
return
}
Expand All @@ -326,6 +420,7 @@ proc orfs_write_sdc { output_file } {
# mtime with second resolution, which breaks make's timestamp-based
# up-to-date checks for fast builds.
proc orfs_copy_db { input_file output_file } {
copy_endpoint_slack $input_file $output_file
if { !$::env(WRITE_ODB_AND_SDC_EACH_STAGE) } {
return
}
Expand Down
13 changes: 13 additions & 0 deletions flow/scripts/variables.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading