Repository navigation
Commit f995d72
* Publish the comparisons against other systems (#184)
The measurements existed and nothing in the repository said what they found.
Docs/Usage/Comparison.md reproduces three of them -- the head-to-head against
Math.NET Symbolics 0.25.0 and Symbolism 1.0.4, the 80 feature probes against
SymPy 1.14.0, and 1,774 problems of Rubi's integration suite -- with the
version, date and library commit each was taken at, and with the rows this
library loses reproduced alongside the rows it wins.
Every sentence of judgement on the page is a count taken from the table
directly above it: 3 of 12 shared tasks identical ignoring spacing, Math.NET
shorter on 4 of the remaining 9 and all four of those rows ones where a
`provided` clause is attached here; eleven of 24 capability rows present here
and in neither of the other two, and two the other way; Math.NET faster on
all four operations both libraries can do.
Three caveats the tables do not state and a reader needs. SymPy's verdict
column is computed from this library's answer alone, so `answers` is not
`agrees` and no disagreement on that page is adjudicated. `absent` is a
keyword search of the public surface, which makes it a lower bound. And
604/1774 is a statement about Rubi's textbook problems at a 5s budget -- held
against the 116/119 the library's own corpus scores, the pair says more about
who wrote each list than about integration.
The harnesses are outside this repository and are not in CI, so the page says
that plainly rather than implying `dotnet test` reproduces a figure on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjumi5K7fg8yx6UK1mZTQd
* Correct two verdict counts against the report they came from
The SymPy table said internal 5 and unclear 2; work/sympyparity.md says 3 and
4. Two probes were attributed to the wrong verdict, and nothing caught it
because both readings sum to the same 80 rows.
Re-checked every other figure on the page against its source in the same pass:
3 of 12 shared tasks identical, Math.NET shorter on 4, and Rubi 604 of 1,774
at 34.0% all match.
* Re-measure the Rubi row on merged master, and say what its rate is worth
The page carried 604 of 1774 at 3ac24bc, three commits before the 2.3.0 tag.
The suite is downloaded rather than vendored, so that harness runs on its own
schedule and had not been re-run since. It has now, at e4eefde -- master with
nine pull requests merged -- and answers 602 of 1774.
The two-problem difference is not a change in the integrator and the page now
says so instead of leaving a reader to infer a regression. All three cases that
moved between the runs moved into the timeout bucket, none became unevaluated
and none became wrong, and timed alone against this same build they take
1744 ms, 1579 ms and 835 ms -- none near the 5 second budget. intbench cannot
abort a thread, so a case that does time out leaks one and the leaked threads
slow every problem after it. The rate is therefore +/-3 problems and the page
says to read the wrong-answer count, which is 0, as the number that means
something.
Every per-file row re-taken from the report rather than edited by hand.
* Take the Rubi figures from the report, all of them, including the row I edited by hand
A third run of the same commit answered 604 with 44 timeouts, against 602 and
46 from the run before it. Restoring the page to 604 I put the Welz row back to
its old 97/4 from memory of the earlier column -- and the report says 96/5,
because Welz 13 timed out in both runs and only the two Timofeev problems came
back. Every per-file row is now substituted from work/intcoverage.md rather
than edited, and the twelve rows sum to the headline in all seven columns.
The variance paragraph is stronger for it: two runs of one commit, same slice
and same budget, differing by two problems, is a better argument than two runs
of two commits. 0 wrong in every run.
---------
Co-authored-by: Rafael <darkfader@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 6ed4507 commit f995d72
2 files changed
Lines changed: 407 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
87 | 95 | | |
88 | 96 | | |
89 | 97 | | |
| |||
0 commit comments