Publish each head's calibration in the catalog and on its bar - #263
Merged
Merged
Conversation
#261 gave every head its own decision threshold and a measured precision, but only a molecule read exposed them. The head catalog — the thing you consult to ask "how good is this head?" — still showed AUROC alone, which is precisely the number that hid the problem: AUROC is computed on ranking and is blind to class imbalance, so `ginger` reads 0.979 there while being right 10% of the time when it fires. /api/heads now carries threshold, precision, recall, n_pos and confident_capable per head, for aroma and mouthfeel alike. A head that fires at 0.16 and is right 6% of the time says so on its own row, not only buried inside one molecule's read. Also fixes headBar, which still hard-coded 0.5 to decide whether a bar was lit. That silently disagreed with the calibrated mouthfeel heads: `cooling` fires at 0.64 and `astringent` at 0.71, so a score of 0.55 was being drawn as firing when the head says it is not. Bars now read the head's own threshold, show where that bar sits and how precise it is there, and hatch the fill for indicative heads. Tox heads keep the flat 0.5 — they are not calibrated and, with hundreds of positives each, were never shy. Taste heads are published with an explicit threshold of 0.5 rather than a null, so a consumer never has to guess whether the field is missing or genuinely flat. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>
rvnminers-A-and-N
added
area:api
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#261 gave every head its own threshold and a measured precision, but only a molecule read exposed them. The head catalog — the thing you consult to ask how good is this head? — still showed AUROC alone, which is exactly the number that hid the problem.
gingerreads AUROC 0.979 there while being right 10% of the time when it fires./api/headsnow carriesthreshold,precision,recall,n_posandconfident_capableper head, for aroma and mouthfeel alike. A head that fires at 0.16 and is right 6% of the time says so on its own row.Also a real bug:
headBarstill hard-coded 0.5 to decide whether a bar was lit, which silently disagreed with the calibrated mouthfeel heads —coolingfires at 0.64 andastringentat 0.71, so a 0.55 score was being drawn as firing when the head says it isn't. Bars now read the head's own threshold, show where it sits and how precise it is there, and hatch the fill for indicative heads.Tox heads keep the flat 0.5 — not calibrated, and with hundreds of positives each they were never shy. Taste heads publish an explicit
0.5rather than null, so consumers never have to guess whether the field is missing or genuinely flat.Verified: e2e suite 13/13 in isolation; full suite 41/43 with the two known load-flaky compare tests, which pass on their own.