Skip to content

Commit 9275989

Browse files
authored
Merge pull request #4604 from The-OpenROAD-Project-staging/qor-check-ci-mode
Check QoR as a checkout of the recorded commit
2 parents f34d4d9 + 51c7b16 commit 9275989

4 files changed

Lines changed: 889 additions & 40 deletions

File tree

‎docs/contrib/Metrics.md‎

Lines changed: 50 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,9 @@ a small set of example designs through our complete RTL-to-GDS flow.
88
To keep track of the quality of the results, every CI run uploads its metrics
99
to the [QoR dashboard](https://dashboard.precisioninno.com). The dashboard
1010
stores one rule per tracked metric and design (a comparison direction and a
11-
tolerance), and judges a run against a real baseline build: the latest
12-
`master` build by default. That check is the QoR gate in `make metadata`.
11+
tolerance), and judges a run against a real baseline build: the `master`
12+
build at the run's merge base by default. That check is the QoR gate in
13+
`make metadata`.
1314

1415
The values a run is measured against are collected by
1516
`flow/util/genMetrics.py` into `$(REPORTS_DIR)/metadata.json`
@@ -47,12 +48,52 @@ check did not run. Read the `[WARN]` lines to see which. Pass `--strict` to
4748
`checkQorMetrics.py` to get distinct exit codes instead (2 inconclusive,
4849
3 unreachable).
4950

50-
Two environment variables shape the comparison:
51-
52-
- `DASHBOARD_API_KEY` grants access to private platforms. Without it, a
53-
private design reports as inconclusive, not as an authentication error.
54-
- `DASHBOARD_JOB_NAME` names the Jenkins pipeline whose `master` builds are the
55-
baseline. The default is `OpenROAD-flow-scripts-Public`.
51+
The check runs as the commit `metadata.json` records. The dashboard searches
52+
its default baseline pipelines in order and takes a `master` build that has the
53+
design: the one at the commit's merge base with `master`. A commit on `master`
54+
is its own merge base, so a run of a `master` commit, such as a downloaded CI
55+
artifact, is compared against that commit's own build.
56+
57+
The dashboard finds the merge base through GitHub, so it cannot find one for a
58+
commit that was never pushed. The check handles that commit itself: when no
59+
remote branch in your checkout contains the commit, it computes
60+
`git merge-base <commit> origin/master` locally and pins the baseline to that
61+
build. A commit that is on a remote, or that is not in your checkout at all, is
62+
left to the dashboard.
63+
64+
When the dashboard has no `master` build at the merge base, or no merge base,
65+
it compares against the latest `master` build instead (the latest one before
66+
the commit, if it knows the commit's date). That build may be newer than the
67+
commit's base, so changes merged since then can show up as regressions or hide
68+
real ones, and the result moves with `master`. The report warns whenever this
69+
happens; set `DASHBOARD_BASE_COMMIT` to compare against an exact build instead.
70+
71+
These environment variables shape the comparison:
72+
73+
- `DASHBOARD_API_KEY` grants access to private platforms. With it, a private
74+
design is resolved like a public one: the dashboard's default baseline
75+
pipelines include the private ones, and only a caller with a key can see
76+
their builds. Without it, a private design has no baseline and reports as
77+
inconclusive, not as an authentication error.
78+
- `DASHBOARD_JOB_NAME` checks the run as a build of that CI pipeline instead,
79+
with the pipeline's rules for the branch checked out in this repository: the
80+
previous build on a trunk branch such as `master`, the merge base on any
81+
other branch. It is meant for CI jobs, which also set `DASHBOARD_BRANCH` and
82+
`DASHBOARD_COMMIT` (or pass `--branch` and `--commit`) since their checkout
83+
has no branch. Earlier versions of the check used it to pick the pipeline
84+
that holds a private platform's baseline; that is no longer needed, so leave
85+
it unset for a local run, private or not. With it set, an unpushed commit is
86+
not pinned to its local merge base.
87+
- `DASHBOARD_MERGE_BASE` (`--merge-base`) names the ref an unpushed commit's
88+
merge base is computed against, `origin/master` by default. Set it to your
89+
upstream ref if `origin` is a fork with an outdated `master`, or to `none` to
90+
leave unpushed commits to the dashboard. It does not apply together with
91+
`DASHBOARD_BASE_COMMIT` or `DASHBOARD_JOB_NAME`.
92+
- `DASHBOARD_BASE_COMMIT` (`--base-commit`) pins the baseline to the build of
93+
that commit. Nothing else is tried: if that commit has no build of the
94+
design, the check is inconclusive.
95+
- `DASHBOARD_API_URL` (`--api-url`) points the check at another deployment and
96+
takes precedence over `--beta`.
5697

5798
If you only changed synthesis, you can gate a synthesis-only run, without
5899
running the rest of the flow. Only the `synth__` and `constraints__` metrics
@@ -62,8 +103,7 @@ are sent, so only their rules are evaluated:
62103
make metadata-synth
63104
```
64105

65-
To check every run under `reports/` at once, or to pin the baseline to a
66-
specific commit, call the script directly:
106+
To check every run under `reports/` at once, call the script directly:
67107

68108
``` shell
69109
cd OpenROAD-flow-scripts/flow

0 commit comments

Comments
 (0)