Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .github/workflows/github-actions-cron-update-OR.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,7 @@ jobs:
delete-branch: true
title: 'Update OpenROAD submodule'
body: |
Automated action to update tools/OpenROAD submodule and tighten CI rule checking.
[ci:rules-tighten]
labels: UpdateRules
Automated action to update tools/OpenROAD submodule.
reviewers: |
vvbandeira
maliberty
Expand Down
161 changes: 65 additions & 96 deletions docs/contrib/Metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,57 +5,26 @@ repository contains source files (e.g., LEF/DEF, Verilog, SDC, Liberty,
RC extraction) and configuration files (e.g. `config.mk`) that enable the user to run
a small set of example designs through our complete RTL-to-GDS flow.

To keep track of the quality of the results, each design directory holds a
rules file that gates the flow:

``` text
flow/designs/<platform>/<design>/rules-base.json
```

The file name comes from `RULES_JSON`, defined in
[`flow/util/utils.mk`](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/flow/util/utils.mk):

``` makefile
export RULES_JSON ?= $(DESIGN_DIR)/rules-$(FLOW_VARIANT).json
```

`FLOW_VARIANT` defaults to `base`, which is why the committed file is
`rules-base.json`. A run with a non-default `FLOW_VARIANT` looks for
`rules-<variant>.json`, or you can point `RULES_JSON` at any file explicitly.

Each entry in the rules file is a metric name, a reference value, and a
comparison operator. For example:

``` json
{
"synth__design__instance__area__stdcell": {
"value": 43.1,
"compare": "<="
},
"detailedplace__design__violations": {
"value": 0,
"compare": "=="
}
}
```

An entry may also carry `"level": "warning"`, which reports a mismatch without
failing the build.
To keep track of the quality of the results, every CI run uploads its metrics
to the [QoR dashboard](https://dashboard.precisioninno.com). The dashboard
stores one rule per tracked metric and design (a comparison direction and a
tolerance), and judges a run against a real baseline build: the latest
`master` build by default. That check is the QoR gate in `make metadata`.

The values a run is measured against are collected by
`flow/util/genMetrics.py` into `$(REPORTS_DIR)/metadata.json`
(`reports/<platform>/<design>/<variant>/metadata.json`). That file is generated
output — it is not committed. `flow/util/checkMetadata.py` compares it against
the rules file.
output — it is not committed. `flow/util/checkQorMetrics.py` sends its numeric
values to the dashboard and reports the verdict.

## Checking against the rules
## Checking against the dashboard

The evaluation checks key metrics (e.g., worst slack, number of DRCs) to
ensure that changes do not degrade the results.

After you make a significant change — e.g., fixing a bug in a piece of code, or
changing a configuration variable such as `PLACE_DENSITY` — review the results
and compare them against the rules. To perform the check, run:
and compare them against the baseline. To perform the check, run:

``` shell
cd OpenROAD-flow-scripts/flow
Expand All @@ -64,69 +33,69 @@ make [clean_metadata] metadata
```

`make metadata` runs three steps in order: `finish` (the full flow),
`metadata-generate` (write `metadata.json`), and `metadata-check` (compare it
against `RULES_JSON`). The check log is written to
`$(REPORTS_DIR)/metadata-check.log`.

If you only changed synthesis, you can gate a synthesis-only run against the
synthesis subset of the same rules file, without running the rest of the flow:
`metadata-generate` (write `metadata.json`), and `metadata-check` (send it to
the dashboard). The check log is written to
`$(REPORTS_DIR)/metadata-check.log`. It lists every failed rule with the
baseline value, the current value, the limit, and the delta, and ends with a
`QoR check: PASS`, `FAIL`, `INCONCLUSIVE`, or `ERROR` line.

A `FAIL` verdict fails the target. So does `ERROR`, which means
`metadata.json` was missing, malformed, or held no numeric metric. An
`INCONCLUSIVE` verdict does not fail the target: the dashboard had no
baseline for the design, evaluated nothing, or could not be reached, so the
check did not run. Read the `[WARN]` lines to see which. Pass `--strict` to
`checkQorMetrics.py` to get distinct exit codes instead (2 inconclusive,
3 unreachable).

Two environment variables shape the comparison:

- `DASHBOARD_API_KEY` grants access to private platforms. Without it, a
private design reports as inconclusive, not as an authentication error.
- `DASHBOARD_JOB_NAME` names the Jenkins pipeline whose `master` builds are the
baseline. The default is `OpenROAD-flow-scripts-Public`.

If you only changed synthesis, you can gate a synthesis-only run, without
running the rest of the flow. Only the `synth__` and `constraints__` metrics
are sent, so only their rules are evaluated:

``` shell
make metadata-synth
```

If the check reports any error, review it to make sure the change in metrics is
expected and justifiable. If so, proceed to the next section to update the
reference values.

## Update process

Updating the rules file is mandatory if a metric became worse than the value
allowed by `rules-base.json` (see the previous section on how to perform the
check). It is also a good idea to update it when your change improves a metric,
so that the improvement is not silently lost later.

To update the rules file for a design, from a tree that has just been measured
(so `metadata.json` exists):
To check every run under `reports/` at once, or to pin the baseline to a
specific commit, call the script directly:

``` shell
cd OpenROAD-flow-scripts/flow
make update_ok
```

`update_ok` is an alias for `update_rules`. It runs `flow/util/genRuleFile.py`
with `--failing --tighten`, which:

- relaxes rules the current run fails, using the measured value plus a margin,
so that small run-to-run variation does not break the flow; and
- tightens rules the current run passes by a wide margin, so improvements are
locked in.

The new file is written to `$(REPORTS_DIR)/rules.json` and then copied over
`$(RULES_JSON)` — that copy is the file you commit.

If you need to overwrite every rule from the current run rather than only the
failing and improved ones, use:

``` shell
make update_rules_force
```

This runs `genRuleFile.py --update`. Prefer `make update_ok`; use the forced
variant only when the metric set itself changed.

## Reference metadata (`metadata-<variant>-ok.json`)

There is a second, optional file per design:

``` shell
# writes flow/designs/<platform>/<design>/metadata-base-ok.json
make update_metadata
./util/checkQorMetrics.py # sweep, log in reports/inline-check.log
./util/checkQorMetrics.py --base-commit <sha> # pin the baseline build
./util/checkQorMetrics.py --verbose # show passing rules too
```

This copies the current `metadata.json` next to the design as
`metadata-$(FLOW_VARIANT)-ok.json`. It is **not** what gates the flow — nothing
in `make metadata` reads it. It is consumed only by
`flow/util/genReportTable.py` when building the golden-vs-current report table,
and almost no design commits one. Update the rules file, not this one, unless
you are specifically working on that report.
If the check reports a failed rule, review it to make sure the change in
metrics is expected and justifiable. If so, say so in the pull request: the
baseline moves with `master`, so a merged change becomes the new baseline
without a file update. Rule tolerances live in the dashboard, not in this
repository.

## Removed: rules files (`rules-<variant>.json`)

Before the dashboard check, each design directory held a
`rules-<variant>.json` file with absolute limits, and
`flow/util/checkMetadata.py` compared `metadata.json` against it. Those
files, that script, `flow/util/genRuleFile.py`, and the `RULES_JSON`
variable were removed. The make targets `update_ok`, `update_rules`, and
`update_rules_force` still exist as stubs that print a deprecation warning
and do nothing, so an old wrapper script does not break. Rule tolerances
live in the dashboard, so there is nothing to regenerate locally after an
accepted QoR change.

## Removed: golden metadata (`metadata-<variant>-ok.json`)

A design directory could also hold a `metadata-<variant>-ok.json` copy of a
known-good `metadata.json`, written by `make update_metadata`. Only
`flow/util/genReportTable.py` read it, to color a golden-versus-current
metrics table in the CI report. That table and the golden files were
removed; the script now writes only the per-design image galleries.
`make update_metadata` is a stub that prints a deprecation warning. The
baseline for a run is the `master` build on the dashboard.
4 changes: 2 additions & 2 deletions docs/tutorials/FlowTutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,10 +248,10 @@ underlying files are organized.

- `designs/sky130hd/ibex`
Files include: designs make file and SDC file for the `sky130hd`
platform and other files for autotuner and metrics.
platform and other files for autotuner and routing.

```
autotuner.json config.mk constraint_doe.sdc constraint.sdc metadata-base-ok.json rules.json
autotuner.json config.mk constraint_doe.sdc constraint.sdc fastroute.tcl
```

- `platforms`
Expand Down
5 changes: 0 additions & 5 deletions docs/user/FlowVariables.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,6 @@ configuration file.
| <a name="RTLMP_OUTLINE_WT"></a>RTLMP_OUTLINE_WT| Weight for violating the fixed outline constraint, meaning that all clusters should be placed within the shape of their parent cluster.| 100.0|
| <a name="RTLMP_RPT_DIR"></a>RTLMP_RPT_DIR| Path to the directory where reports are saved.| |
| <a name="RTLMP_WIRELENGTH_WT"></a>RTLMP_WIRELENGTH_WT| Weight for half-perimiter wirelength.| 100.0|
| <a name="RULES_JSON"></a>RULES_JSON| json files with the metrics baseline regression rules. In the ORFS Makefile, this defaults to $DESIGN_DIR/rules-base.json, but ORFS does not mandate the users source directory layout and this can be placed elsewhere when the user sets up an ORFS config.mk or from bazel-orfs.| |
| <a name="RUN_LOG_NAME_STEM"></a>RUN_LOG_NAME_STEM| Stem of the log file name, the log file will be named `$(LOG_DIR)/$(RUN_LOG_NAME_STEM).log`.| run|
| <a name="RUN_SCRIPT"></a>RUN_SCRIPT| Path to script to run from `make run`, python or tcl script detected by .py or .tcl extension.| |
| <a name="SC_LEF"></a>SC_LEF| Path to technology standard cell LEF file.| |
Expand Down Expand Up @@ -636,10 +635,6 @@ configuration file.

- [ABSTRACT_SOURCE](#ABSTRACT_SOURCE)

## test variables

- [RULES_JSON](#RULES_JSON)

## Uncategorized variables

- [ADDITIONAL_FILES](#ADDITIONAL_FILES)
Expand Down
10 changes: 6 additions & 4 deletions flow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,7 @@ For a design at `flow/designs/<platform>/<design>/` with `DESIGN_NAME =
| `<n>_route` | Detailed routing |
| `<n>_final` | Final + fill |
| `<n>_generate_abstract` | LEF/LIB abstract |
| `<n>_test` | Full flow + QoR check against `rules-base.json` |
| `<n>_update` | Rebuild and write thresholds back to `rules-base.json` |
| `<n>_test` | Full flow + QoR check |

Stages depend on the previous, so `_final` runs the whole flow.

Expand Down Expand Up @@ -48,8 +47,11 @@ load("//flow/designs:design.bzl", "files")
files("verilog")
```

A design counts as CI-tested iff `rules-base.json` exists; without it
the generated targets get `tags = ["manual"]`.
The QoR check is the dashboard check described in
`docs/contrib/Metrics.md`. The committed `rules-base.json` files it used
to read were removed, so bazel-orfs no longer has that file to mark a
design as CI-tested. Until bazel-orfs picks another marker, its generated
test targets carry `tags = ["manual"]` for every design.

## Parallelism

Expand Down
29 changes: 10 additions & 19 deletions flow/designs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

The maintainers' QoR signal comes from Jenkins runs published at
<https://dashboard.precisioninno.com/>. The targets described here are
its local counterpart: the same designs, gated by the same
`rules-base.json` files with the same `checkMetadata.py`, but built
locally through Bazel so the loop can be sharpened onto exactly the
its local counterpart: the same designs, judged by the same dashboard
rules through `make metadata-check` (see `docs/contrib/Metrics.md`), but
built locally through Bazel so the loop can be sharpened onto exactly the
design, engine, and stage being worked on — fast enough for an agent
(e.g. Claude) to iterate against.

Expand All @@ -16,10 +16,11 @@ Every design gets manual forced-engine flow variants (see
- `<name>_yosys_*` pins the yosys engine, so the baseline stays
reproducible after a design switches (asap7/gcd today).

Designs with a `rules-base.json` get two QoR tiers per variant:
CI-tested designs get two QoR tiers per variant:

- `<name>_<engine>_synth_test` — the fast tier: synthesis only,
checked against the `synth__`/`constraints__` subset of the rules.
checked against the `synth__`/`constraints__` subset of the rules
(`make metadata-synth`).
Minutes. QoR-only — without LEC it cannot prove correctness, so use
it to iterate, not to conclude.
- `<name>_<engine>_test` — the full flow against all rules: the real,
Expand Down Expand Up @@ -49,20 +50,10 @@ bazelisk test --keep_going //flow/designs/asap7:syn_synth_test
bazelisk test --keep_going //flow/designs/asap7:syn_test
```

Then display what you have — deliberately a separate concern from
deciding what to test or build:

```sh
bazelisk run //flow/designs:syn-dashboard
```

It prints a GitHub-markdown TL;DR table of every forced-engine test
result already in `bazel-testlogs` — pass/fail per tier and engine,
failing rule fields, result age — without building anything. Paste it
into a PR or issue as-is. Because the display reads whatever exists,
the loop stays fast: iterate on the one target you care about, and the
dashboard aggregates across everything you (or earlier sessions) have
built so far.
The `syn-dashboard` target that summarised these results as a markdown
table read the output of the removed rules-file check, so it was removed
with it. Read the `QoR check:` line in each test's `metadata-check.log`
instead.

The suites include designs that fail OpenROAD-SYN today on purpose —
they are a status signal, not a green-only gate. A build failure (e.g.
Expand Down
Loading
Loading