Fail the build when the popular use cases allocate more (#529, #500) (#1014)

The benchmark job measured parse, Simplify, Solve and Differentiate on every push to the kernel
and then uploaded the numbers, which meant a regression was visible to whoever went looking and to
nobody else. #746's standing condition is that speed and memory on these paths are measured rather
than hoped for, and a measurement that nothing fails on is not that.

PerformanceGate compares a run against Sources/Tests/DotnetBenchmark/performance-baseline.json and
returns a non-zero exit code. It gates allocation and not the wall clock, because only one of the
two is a property of the code: allocated bytes per operation is what the program asked the
allocator for, and the same input asks for the same number of bytes, while a GitHub-hosted runner's
wall clock is a property of whoever else is on that host.

Measured before choosing the thresholds -- all eighteen benchmarks, three consecutive runs of one
unchanged build, on a machine deliberately left loaded -- allocation was at most 0.033% apart run
to run on the sixteen gated benchmarks and identical to the byte on seven of them, while the mean
time was up to 51.8% apart and 8-10% apart on nine of the eighteen. Three orders of magnitude
between the two is the argument: allocation is held to 3%, a hundred times the observed spread, and
the wall clock fails only above 3x, which is a catastrophe threshold and deliberately catches
nothing smaller.

CompileEasy and CompileHard are excluded by name, at 1.0% and 3.6% over those same three runs,
because what they measure includes the runtime building and JIT-compiling a delegate. The exclusion
lives in the baseline's ungated map with its reason and the gate prints it on every run, so it is a
decision somebody made rather than a benchmark missing from the file.

It fails on an improvement too, as the corpus gate does and for the same reason: a baseline that
overstates what the library allocates would let a later change give the gain back unnoticed. The
verdict says IMPROVED rather than REGRESSED and says to update the baseline.

The run writes its numbers in the baseline's own format, so updating the baseline is one file
copied over another -- including from the artifact of the job that just failed, which is why the
upload step keeps if: always() and why the gate runs before it.

#529
#500
#746


Claude-Session: https://claude.ai/code/session_011emtnRT6EWTrxXNtqDVK3e

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Rafael-SOWNet

.github/workflows/Benchmark.yml

@@ -55,6 +55,22 @@ jobs:
         cd Sources/Tests/DotnetBenchmark
         dotnet run -c Release RAMUsageTest
 
+    # The step that makes the two above a gate rather than a report. It runs no benchmark:
+    # the CPU step wrote what it measured to benchmark_results.csv/measured-*.json, and this
+    # reads that against the committed performance-baseline.json. Allocation is what it fails
+    # on, in either direction; the wall clock is reported and only fails at 3x, because a
+    # GitHub-hosted runner shares its host and a tight time threshold would be red for
+    # reasons that are not the change. The argument, and the variance it was measured from,
+    # are in PerformanceGate.cs.
+    #
+    # Last, so that a failure here does not cost the RAM figures, and before the upload so
+    # that the file to copy over the baseline is in the artifact of the run that failed.
+    # https://github.com/asc-community/AngouriMath/issues/529
+    - name: 'Kernel performance gate'
+      run: |
+        cd Sources/Tests/DotnetBenchmark
+        dotnet run -c Release --no-build PerformanceGate
+
     # The numbers used to exist only in this job's log, which expires -- so there was no way to
     # see whether a figure had moved without re-running an old commit by hand. Retaining them is
     # step 1 of https://github.com/asc-community/AngouriMath/issues/500, and the only step that

AGENTS.md

@@ -200,6 +200,26 @@ suite is paid for on every commit. The harnesses in `work/` are where a measurem
 inputs, takes minutes, and gets read by a person; the two are not substitutes, and a finding from a
 harness that is worth keeping belongs in the corpus as a new problem.
 
+### And a second gate, on what the popular use cases allocate
+
+`Sources/Tests/DotnetBenchmark/performance-baseline.json` records allocated bytes and mean
+nanoseconds per operation for parse, `Simplify`, `Solve` and `Differentiate` on textbook-sized input.
+The Kernel Benchmark workflow runs the benchmarks and then `PerformanceGate`, which compares the run
+against that file and fails the build. It behaves like the corpus gate — including failing when a
+number gets **better**, because a baseline that overstates what the library allocates would let a
+later change give the gain back unnoticed.
+
+**It gates allocation and reports time.** Allocated bytes are a property of the code: the same input
+asks the allocator for the same number of bytes. A shared runner's wall clock is a property of
+whoever else is on that host, so the time column fails only above 3x — a catastrophe threshold, not
+a performance one. The measurement that chose those two numbers is in the remarks of
+`PerformanceGate.cs`.
+
+It runs on the benchmark job rather than in the suite, because it costs about ten minutes and
+everything in the suite is paid for on every commit. What to do when it fails, and when updating the
+baseline is legitimate, is in
+[`WhatsNew/version_performance_control.md`](Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md).
+
 ## One structure under several features
 
 Four things in this library are the same shape, and three of them were written separately before
@@ -400,7 +420,7 @@ are short, and a stale one is worse than none — if you change what a file desc
 | [`Contributing/SimplificationContract.md`](Sources/AngouriMath/Docs/Contributing/SimplificationContract.md) | what a rewrite may assume, and the ten obligations one has to meet. Read it *before* adding or changing a rule |
 | [`Contributing/CanonicalForm.md`](Sources/AngouriMath/Docs/Contributing/CanonicalForm.md) | canonical versus simplest, why no canonical form exists for the whole language, and what one means per node class. Read it before comparing two expressions for equality |
 | [`Contributing/coding_rules.md`](Sources/AngouriMath/Docs/Contributing/coding_rules.md) | sealed-or-abstract, and immutability of `Entity` |
-| [`WhatsNew/version_performance_control.md`](Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md) | the inter-version performance table, and how to add a column |
+| [`WhatsNew/version_performance_control.md`](Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md) | the inter-version performance table, how to add a column, and when the CI performance baseline may be updated |
 | `Sources/Analyzers/` | the custom analyzers, including the static-field one behind `[ConstantField]` |
 
 Anything added for the library's own purposes is not `public` — see
@@ -434,7 +454,8 @@ whatever else it delivered:
    [`WhatsNew/version_performance_control.md`](Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md).
    Measure the previous column again on the same machine and publish the pair: columns taken on
    different hardware cannot be read as a ratio, and a uniform factor across every row is the machine
-   rather than the code.
+   rather than the code. Since #529 that is a gate and not only a record — see *And a second gate, on
+   what the popular use cases allocate* above.
 3. **Correctness coverage grows with the surface.** Each new layer adds ways to be wrong that the one
    below could not express.
 

Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md

@@ -47,6 +47,82 @@ take a millisecond in total, while a thousand evaluations of freshly built nodes
 The row is kept as it is so the column history stays comparable, and the two trigonometric
 benchmarks build their expression afresh on every call for exactly this reason.
 
+---
+
+## The gate, and when the baseline may be updated
+
+This file is a history a person reads. `Sources/Tests/DotnetBenchmark/performance-baseline.json`
+is the same measurement in the form a build can fail on, which is what
+[#746](https://github.com/asc-community/AngouriMath/issues/746)'s standing condition -- *speed and
+memory on popular use cases are measured, not hoped for* -- asks for and what
+[#529](https://github.com/asc-community/AngouriMath/issues/529) asked for before it. The Kernel
+Benchmark workflow runs the benchmarks and then `dotnet run -c Release PerformanceGate`, which
+compares that run against the committed file.
+
+**It gates allocation and not time, because only one of the two is a property of the code.**
+Allocated bytes per operation is what the program asked the allocator for: the same code on the
+same input asks for the same number of bytes. Wall-clock time on a GitHub-hosted runner is a
+property of whoever else is on that host. Measured before choosing the thresholds -- all eighteen
+benchmarks, three consecutive runs of one unchanged build, on a machine deliberately left loaded
+(eight cores, load average around 16, which is closer to a shared runner than an idle desktop is):
+
+| | run to run |
+|---|---|
+| allocation, the sixteen gated benchmarks | at most **0.033%** apart -- six bytes on eighteen kilobytes, for `ParseEasy` -- and identical to the byte on seven of them |
+| mean time | up to **51.8%** apart, and 8-10% on nine of the eighteen |
+
+Three orders of magnitude between the two is the whole argument. Allocation is held to 3%, a
+hundred times the observed spread, and the wall clock fails only above 3.00x, which is a
+catastrophe threshold and deliberately catches nothing smaller. A gate that is red for reasons that
+are not the change is a gate people learn to ignore, and that is worse than not having one.
+
+**Two benchmarks are excluded by name.** `CompileEasy` and `CompileHard` moved 1.0% and 3.6% over
+those same three runs, because what they measure includes the runtime building and JIT-compiling a
+delegate, which is not reproducible the way the library's own allocation is. They are listed in the
+baseline's `ungated` map with that reason, and the gate prints the reason every time it runs -- an
+exclusion is a decision somebody made, not a benchmark that happens to be missing from the file.
+
+**It fails when a number gets better, too**, exactly as the corpus gate does and for the same
+reason: a baseline that overstates what the library allocates silently permits giving the gain
+back. The failure says `IMPROVED` rather than `REGRESSED`, and says what to do about it.
+
+### How to update it
+
+The run writes its numbers in the baseline's own format, so updating the baseline is a copy rather
+than a transcription:
+
+```
+cd Sources/Tests/DotnetBenchmark
+dotnet run -c Release CommonFunctionsInterVersion
+cp benchmark_results.csv/measured-CommonFunctionsInterVersion.json performance-baseline.json
+dotnet run -c Release --no-build PerformanceGate
+```
+
+The same file is in the `benchmark-results-<sha>` artifact of the CI job -- including of the job
+that just failed, which is why the upload step runs `if: always()`. That copy is the better source
+of the two, because it was taken on the machine and the runtime the gate is checked against.
+
+### When updating it is legitimate
+
+- **The allocation went down.** Record it, in the same change. This is the case the gate exists to
+  stop being absorbed, and it is the only one where a red build means nothing is wrong.
+- **The allocation went up and buys something.** [#918](https://github.com/asc-community/AngouriMath/pull/918)
+  is the worked example, below in this file: the polynomial layer made `SolveMedium` allocate 19%
+  more and made an incomplete root set complete. Say which in the pull request, next to the
+  `BREAKING-CHANGES.md` entry that a changed answer already owes.
+- **A benchmark was added or removed.** The gate reports `NEW` or `MISSING`, and the baseline
+  follows in the same change.
+- **The runtime moved under it.** Allocation is a property of the code *and* of the BCL beneath it,
+  so a .NET version bump can move a row with nothing here having changed. The baseline records the
+  `runtime` and `machine` it was taken on, and the gate prints both against the run's own, so check
+  those two lines before concluding this -- and say so in the commit, because it is the one reason
+  on this list that leaves no trace in the diff.
+
+And when it is not: **because the build is red and the number is inconvenient.** The gate names the
+benchmark and the size of the move. A move nobody can explain is the finding, not the obstacle.
+
+---
+
 |          Method |         [331st](https://github.com/asc-community/AngouriMath/commit/10e6e5a90e7270336b68dc5fd6aa36f3e0e65d2b) |          [380th](https://github.com/asc-community/AngouriMath/commit/73ae36488ddb863c1d6f35db5ed2f5dcf1484a26) |     [391st](https://github.com/asc-community/AngouriMath/commit/c7e08e6936bfdc2373377bec81ffd160e406244f)      |         [410th](https://github.com/asc-community/AngouriMath/commit/20814936bc740a9f410af4a4368e9895eab7aaf7) |           [483rd](https://github.com/asc-community/AngouriMath/commit/355963dcdf0ff9da568e9f1144ad2b7b68c19584) |         [520th](https://github.com/asc-community/AngouriMath/commit/70aa71acb73307c9f7df0aac006faae31b06058c) |         [690th](https://github.com/asc-community/AngouriMath/commit/5cc894939cb3657f0aa7ef5a25fd55011058929f) |         [826th](https://github.com/asc-community/AngouriMath/commit/87e33ec3590a95dd4ec59ff5c1f77064a64196d1) |         [914th](https://github.com/asc-community/AngouriMath/commit/6134338df083a908369b6bcfb69e70a4269ec51b) |         [920th](https://github.com/asc-community/AngouriMath/commit/501a0a3a9b2e07cddf92c4446b73ad6e2748253a) |        [1034th](https://github.com/asc-community/AngouriMath/commit/a3f48b47795b2dc2b3435152989a6e15639a65b4) |        [1066th](https://github.com/asc-community/AngouriMath/commit/a33746651f56a380b6c17913aa844f162f258d8c) |        [1090th](https://github.com/asc-community/AngouriMath/commit/9530c5b04484e98023941f7693bbeb1a3282cee6) |           [1446th](https://github.com/asc-community/AngouriMath/commit/2abb2b537c03977281f3fc2cab1da2c78c36a5f5) | [1620th](https://github.com/asc-community/AngouriMath/commit/e05d71797f53a9ac7dbdad6075cd7302a91036e7) | [1671st](https://github.com/asc-community/AngouriMath/commit/253b5a8d598a9a1d721b777340bd53ae3be12f99) | [1769th](https://github.com/asc-community/AngouriMath/commit/3ac24bc241e898509533bc980ec5dc91c5d79a07) |
 |---------------- |--------------:|---------------:|---------------:|--------------:|----------------:|--------------:|--------------:|--------------:|--------------:|--------------:|--------------:|--------------:|--------------:|-----------------:|--------------:|--------------:|--:|
 |       ParseEasy |        28,599 |         73,669 |        134,120 |        44,328 |          54,675 |        21,722 |        32,212 |        32,138 |        34,702 |        32,199 |        33,008 |        27,483 |        33,043 |        34,664 | 11,760 ns | 10,857 ns | 10,080 ns |