Explain in plain language how accurate the heads actually are - #264
Merged
Merged
Conversation
Everything needed to answer "how good is this, really?" was scattered across a manifest, a training script and three PR descriptions. docs/ACCURACY.md collects it and writes it for someone without a machine-learning background — the document to hand a customer, a chemist, or a hiring manager who asks the obvious question. It covers what a head is, what 5-fold cross-validation buys us (every molecule graded by a model blind to it), and the distinction that matters most here: AUROC measures ranking and is blind to class imbalance, while precision is not. A head with 11 positives among 2,403 molecules can score 0.979 and be right one time in ten when it fires. Both numbers are true; only one of them was being published until recently. Then the per-head thresholds, the 50% precision floor, the F1 mistake that made the floor necessary, and what `indicative` means — marked, kept in full, never dressed up as confident. Finishes with the actual accuracy spread (25 aroma heads at 90-100% precision, 23 that have never been wrong on unseen molecules, 72 below 50%), the memorization audit, a one-paragraph version to say out loud, and a curl command so none of it has to be taken on faith. Every figure was cross-checked against the running app: 94 confident-capable, 73 indicative, 23 at perfect precision. Also adds a "How good is a head, really?" card to the site's own How-it-works grid, so the point lands for someone who will never open the docs folder, and links the new doc from HOW-IT-WORKS.md section 3, where the old text implied AUROC was the whole story. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>
rvnminers-A-and-N
added
area:aroma
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything needed to answer "how good is this, really?" was scattered across a manifest, a training script and three PR descriptions.
docs/ACCURACY.mdcollects it and writes it for someone without an ML background — the document to hand a customer, a chemist, or a hiring manager.What it covers
gingeris AUROC 0.979 / precision 0.10 — right one time in ten when it fires. Both true; only one was being publishedblackberrytuned to 0.18, where 96% of its calls were wrong)indicativemeans — marked, kept in full, never dressed up as confidentcurlso none of it has to be taken on faithThe honest spread, stated openly:
indicative)23 aroma heads have never been wrong on unseen molecules.
Also: a "How good is a head, really?" card in the site's own How-it-works grid, so the point lands for someone who'll never open
docs/; and a link fromHOW-IT-WORKS.md§3, whose old text implied AUROC was the whole story.Verified: every figure cross-checked live —
/api/headsreports 94 confident-capable, 73 indicative, 23 at perfect precision, matching the doc exactly. Suite 43 passed.