Time a chain of modules with the base's and the head's build systems

Why:
    A change to the build's scheduling, such as starting a module once the
    interfaces it reads are final, does the same work sooner. Counting
    allocated words and instructions can only show what such a change
    costs, never what it saves.

`time <base checkout> <head checkout> <out.json>` builds a generated
project of 150 modules in a chain, each using the one before it, and a
module beside each, clean with the two build systems in turn on the
default number of worker domains: one build each that isn't counted,
then 10 rounds, the order alternating each round. `compare --timings`
adds a section to the report with the base's and the head's median times
and the median of the head's time over the base's in the same round, with
a 95% confidence interval from a percentile bootstrap of the rounds with a
fixed seed. The section never counts as a regression or an improvement;
when the times' file is missing or can't be read, it says they weren't
measured. In CI, the step runs after both measurements, each build may
take 2 minutes and the whole command 10, and a failure only leaves the
times out.

Locally, with load around 5 to 9 on 8 cores, the ratio was 0.695 (0.646
to 0.825) for the pull request that starts dependents once a module's
interface is final, 0.991 (0.948 to 1.055) for one that only changes
namespaced packages, and 0.965 (0.921 to 1.078) with the same build
system on both sides. The command took about 45 seconds.

Signed-off-by: Jono Prest <jjprest@gmail.com>

Author: JonoPrest

.github/workflows/ci.yml

@@ -467,10 +467,17 @@ jobs:
           }
           measure "$RUNNER_TEMP/base" "$RUNNER_TEMP/base.json"
           measure "$GITHUB_WORKSPACE" "$RUNNER_TEMP/head.json"
+          # Wall time, with the two build systems taking turns on this
+          # runner. It only indicates a change that counting can't show, so
+          # a failure, or a run longer than 10 minutes, leaves it out of the
+          # report and fails nothing.
+          timeout --kill-after=30s 10m \
+            node scripts/compilerBenchmarks.js time "$RUNNER_TEMP/base" "$GITHUB_WORKSPACE" \
+            "$RUNNER_TEMP/timings.json" || echo "::warning::The wall-time benchmark failed."
           status=0
           report="$RUNNER_TEMP/benchmarks-report.md"
           node scripts/compilerBenchmarks.js compare "$RUNNER_TEMP/base.json" "$RUNNER_TEMP/head.json" \
-            --threshold 1 --summary "$report" || status=$?
+            --threshold 1 --summary "$report" --timings "$RUNNER_TEMP/timings.json" || status=$?
           cat "$report" >> "$GITHUB_STEP_SUMMARY"
           printf '\nPull request %s compared with base %s in [this run](%s).\n' \
             "$(git rev-parse --short=9 HEAD^2)" "$(git rev-parse --short=9 HEAD^1)" "$RUN_URL" \

changes/generated-project-benchmarks.internal.md

@@ -1 +1,2 @@
 - Count the words the build allocates on generated projects in the CI benchmarks: many modules using little of a few large dependencies, and a namespaced package of 1,000 modules.
+- Time clean builds of a generated chain of modules with the base's and the head's build systems in turn in the CI benchmarks, and report the median ratio with a 95% confidence interval. The times are only indicative and never fail the job.