Skip to content

Build the 73 indicative aroma heads up to confident (precision, not AUROC, is the bar) #262

Description

@rvnminers-A-and-N

Calibration in #261 surfaced this, and it is the single biggest quality signal we have on the aroma roster: 73 of 167 heads never reach 50% out-of-fold precision at any threshold. They fire well above base rate — real evidence — but they are right less than half the time when they fire, so they ship as indicative rather than confident.

The point of this issue is that indicative is a starting state, not a resting state. Every one of these should be built up until it can make a confident call.

Why AUROC hid this. AUROC is computed on ranking and is insensitive to class imbalance; precision is not. Both of these are true of the same head:

head AUROC out-of-fold precision
ginger 0.979 0.10
rosemary 0.897 0.06
passionfruit 0.895 0.06
woody 0.812 0.07
fatty 0.854 0.17
coffee 0.960 0.50 (just clears)

A head can rank its positives above its negatives beautifully and still be swamped by false positives, because there are 200x more negatives than positives. Nothing we had been reporting would have shown this.

What actually fixes it — in order of expected effect:

  1. More positives per head. This is the same lever as model(aroma): 16 heads are memorizing, not generalizing — broaden their positives #256 and model(mouthfeel): broaden the tingling head — it barely generalizes #247, and it is the one that moves precision. A head with 11 positives cannot learn a decision boundary sharp enough to beat a 200:1 imbalance. ginger (11), passionfruit (10), violetleaf (11), woody (15) are the obvious first targets — all have deep, public, uncontroversial chemistry.
  2. Better negatives. Right now every unlabelled molecule is an implicit negative, which is wrong: the HSDB corpus is sparsely annotated, so a genuinely gingery molecule with no odor text counts against the head. Curated hard negatives, or treating unlabelled as unknown rather than negative, would raise precision without touching the positives.
  3. A better representation (research(model): GNN for structure→property vs the RandomForest heads #199). Where 1 and 2 stall, this is the GNN case: a 2048-bit Morgan fingerprint may simply not separate these classes. Precision on the indicative 73 is a much sharper benchmark for a GNN than AUROC, which the forests already score highly on.
  4. Calibrated probabilities. CalibratedClassifierCV (isotonic/Platt) over the forests would make the scores mean what they say, which may let some heads clear the floor without any new data.

Explicitly NOT the fix: dropping the 73 heads, or hiding their molecules from chips, palette match and the map. They keep everything — they are simply labelled honestly. That is settled; see the reverted gating in #261.

Measuring progress: training/audit_generalization.py prints precision and the capable/indicative split per head. Target is the count of confident-capable heads going up from 94/167, with novel discoveries not collapsing — a head that reaches the floor by refusing to fire has not improved.

Baseline as of #261: 94 confident-capable, 73 indicative, median threshold 0.46.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:aromaOpenPOM aroma model + sidecararea:dataDatasets, sources, column mappingarea:trainingPython dataset build + model trainingenhancementNew feature or request

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions