You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Calibration in #261 surfaced this, and it is the single biggest quality signal we have on the aroma roster: 73 of 167 heads never reach 50% out-of-fold precision at any threshold. They fire well above base rate — real evidence — but they are right less than half the time when they fire, so they ship as indicative rather than confident.
The point of this issue is that indicative is a starting state, not a resting state. Every one of these should be built up until it can make a confident call.
Why AUROC hid this. AUROC is computed on ranking and is insensitive to class imbalance; precision is not. Both of these are true of the same head:
head
AUROC
out-of-fold precision
ginger
0.979
0.10
rosemary
0.897
0.06
passionfruit
0.895
0.06
woody
0.812
0.07
fatty
0.854
0.17
coffee
0.960
0.50 (just clears)
A head can rank its positives above its negatives beautifully and still be swamped by false positives, because there are 200x more negatives than positives. Nothing we had been reporting would have shown this.
What actually fixes it — in order of expected effect:
Better negatives. Right now every unlabelled molecule is an implicit negative, which is wrong: the HSDB corpus is sparsely annotated, so a genuinely gingery molecule with no odor text counts against the head. Curated hard negatives, or treating unlabelled as unknown rather than negative, would raise precision without touching the positives.
A better representation (research(model): GNN for structure→property vs the RandomForest heads #199). Where 1 and 2 stall, this is the GNN case: a 2048-bit Morgan fingerprint may simply not separate these classes. Precision on the indicative 73 is a much sharper benchmark for a GNN than AUROC, which the forests already score highly on.
Calibrated probabilities.CalibratedClassifierCV (isotonic/Platt) over the forests would make the scores mean what they say, which may let some heads clear the floor without any new data.
Explicitly NOT the fix: dropping the 73 heads, or hiding their molecules from chips, palette match and the map. They keep everything — they are simply labelled honestly. That is settled; see the reverted gating in #261.
Measuring progress:training/audit_generalization.py prints precision and the capable/indicative split per head. Target is the count of confident-capable heads going up from 94/167, with novel discoveries not collapsing — a head that reaches the floor by refusing to fire has not improved.
Baseline as of #261: 94 confident-capable, 73 indicative, median threshold 0.46.
Calibration in #261 surfaced this, and it is the single biggest quality signal we have on the aroma roster: 73 of 167 heads never reach 50% out-of-fold precision at any threshold. They fire well above base rate — real evidence — but they are right less than half the time when they fire, so they ship as
indicativerather than confident.The point of this issue is that indicative is a starting state, not a resting state. Every one of these should be built up until it can make a confident call.
Why AUROC hid this. AUROC is computed on ranking and is insensitive to class imbalance; precision is not. Both of these are true of the same head:
gingerrosemarypassionfruitwoodyfattycoffeeA head can rank its positives above its negatives beautifully and still be swamped by false positives, because there are 200x more negatives than positives. Nothing we had been reporting would have shown this.
What actually fixes it — in order of expected effect:
ginger(11),passionfruit(10),violetleaf(11),woody(15) are the obvious first targets — all have deep, public, uncontroversial chemistry.CalibratedClassifierCV(isotonic/Platt) over the forests would make the scores mean what they say, which may let some heads clear the floor without any new data.Explicitly NOT the fix: dropping the 73 heads, or hiding their molecules from chips, palette match and the map. They keep everything — they are simply labelled honestly. That is settled; see the reverted gating in #261.
Measuring progress:
training/audit_generalization.pyprints precision and the capable/indicative split per head. Target is the count of confident-capable heads going up from 94/167, withnoveldiscoveries not collapsing — a head that reaches the floor by refusing to fire has not improved.Baseline as of #261: 94 confident-capable, 73 indicative, median threshold 0.46.