Skip to content

Commit f995d72

Browse files
Rafael-SOWNetdarkfaderclaude
authored
Publish the comparisons against other systems (#184) (#1030)
* Publish the comparisons against other systems (#184) The measurements existed and nothing in the repository said what they found. Docs/Usage/Comparison.md reproduces three of them -- the head-to-head against Math.NET Symbolics 0.25.0 and Symbolism 1.0.4, the 80 feature probes against SymPy 1.14.0, and 1,774 problems of Rubi's integration suite -- with the version, date and library commit each was taken at, and with the rows this library loses reproduced alongside the rows it wins. Every sentence of judgement on the page is a count taken from the table directly above it: 3 of 12 shared tasks identical ignoring spacing, Math.NET shorter on 4 of the remaining 9 and all four of those rows ones where a `provided` clause is attached here; eleven of 24 capability rows present here and in neither of the other two, and two the other way; Math.NET faster on all four operations both libraries can do. Three caveats the tables do not state and a reader needs. SymPy's verdict column is computed from this library's answer alone, so `answers` is not `agrees` and no disagreement on that page is adjudicated. `absent` is a keyword search of the public surface, which makes it a lower bound. And 604/1774 is a statement about Rubi's textbook problems at a 5s budget -- held against the 116/119 the library's own corpus scores, the pair says more about who wrote each list than about integration. The harnesses are outside this repository and are not in CI, so the page says that plainly rather than implying `dotnet test` reproduces a figure on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bjumi5K7fg8yx6UK1mZTQd * Correct two verdict counts against the report they came from The SymPy table said internal 5 and unclear 2; work/sympyparity.md says 3 and 4. Two probes were attributed to the wrong verdict, and nothing caught it because both readings sum to the same 80 rows. Re-checked every other figure on the page against its source in the same pass: 3 of 12 shared tasks identical, Math.NET shorter on 4, and Rubi 604 of 1,774 at 34.0% all match. * Re-measure the Rubi row on merged master, and say what its rate is worth The page carried 604 of 1774 at 3ac24bc, three commits before the 2.3.0 tag. The suite is downloaded rather than vendored, so that harness runs on its own schedule and had not been re-run since. It has now, at e4eefde -- master with nine pull requests merged -- and answers 602 of 1774. The two-problem difference is not a change in the integrator and the page now says so instead of leaving a reader to infer a regression. All three cases that moved between the runs moved into the timeout bucket, none became unevaluated and none became wrong, and timed alone against this same build they take 1744 ms, 1579 ms and 835 ms -- none near the 5 second budget. intbench cannot abort a thread, so a case that does time out leaks one and the leaked threads slow every problem after it. The rate is therefore +/-3 problems and the page says to read the wrong-answer count, which is 0, as the number that means something. Every per-file row re-taken from the report rather than edited by hand. * Take the Rubi figures from the report, all of them, including the row I edited by hand A third run of the same commit answered 604 with 44 timeouts, against 602 and 46 from the run before it. Restoring the page to 604 I put the Welz row back to its old 97/4 from memory of the earlier column -- and the report says 96/5, because Welz 13 timed out in both runs and only the two Timofeev problems came back. Every per-file row is now substituted from work/intcoverage.md rather than edited, and the twelve rows sum to the headline in all seven columns. The variance paragraph is stronger for it: two runs of one commit, same slice and same budget, differing by two problems, is a better argument than two runs of two commits. 0 wrong in every run. --------- Co-authored-by: Rafael <darkfader@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 6ed4507 commit f995d72

2 files changed

Lines changed: 407 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,14 @@ equations, systems of equations,
8484
with <a href="https://am.angouri.org/wiki/10.-Matrices.html">matrices</a>, find limits,
8585
convert an expression to LaTeX, and <a href="https://am.angouri.org/wiki/">many other things</a>.
8686

87+
## How does it compare?
88+
89+
Measured rather than asserted:
90+
<a href="./Sources/AngouriMath/Docs/Usage/Comparison.md">Docs/Usage/Comparison.md</a> puts the
91+
library beside Math.NET Symbolics, Symbolism, SymPy 1.14.0 and Rubi's integration test-suite, and
92+
reproduces the rows it loses as well as the ones it wins &mdash; 604 of 1,774 Rubi integrals
93+
answered with no wrong answer, and Math.NET faster on all four operations both libraries can do.
94+
8795
## Where can be used?
8896

8997
<a href="#jupyter"><img src="./.github/additional/readme/side.PNG" align="right" width="25%" alt="AngouriMath now supports Jupyter integration"/></a>

0 commit comments

Comments
 (0)