Ran the generalization test from docs/MOUTHFEEL.md (fire each head over all 8,834 molecules, count hits that were NOT in its training set) across all 164 aroma heads. The roster is mostly healthy — but 16 heads are lookup tables wearing high AUROCs, exactly the pattern that caught tingling in #247.
148 of 164 generalize (median 7 novel discoveries). The strongest are doing real work: coffee fires on 65 molecules from 12 positives (53 novel), fatty on 96 from 27 (69 novel).
16 memorize — 0 novel discoveries. Each fires on exactly its own training molecules:
| head |
AUROC |
n_pos |
fires |
novel |
| celery |
0.999 |
13 |
13 |
0 |
| elemi |
0.980 |
10 |
10 |
0 |
| turmeric |
0.975 |
11 |
11 |
0 |
| pine |
0.965 |
15 |
14 |
0 |
| costus |
0.961 |
11 |
11 |
0 |
| eucalyptus |
0.948 |
10 |
10 |
0 |
| rosemary |
0.943 |
12 |
12 |
0 |
| fennel |
0.936 |
12 |
12 |
0 |
| frankincense |
0.916 |
11 |
11 |
0 |
| narcissus |
0.911 |
10 |
10 |
0 |
| allspice |
0.885 |
12 |
12 |
0 |
| freesia |
0.875 |
12 |
12 |
0 |
(+4 more below 0.875.) All sit at 10–15 positives on a single narrow scaffold. A 0.999 AUROC here means "this class is structurally distinctive", not "this head predicts".
Fix: the #247 approach — add structurally diverse positives, not more of the same scaffold. Broadening tingling from one genus to four took it 0 → 2 novel. These heads have rich public chemistry to draw on (pine: pinenes/camphene/carene/bornyl acetate/terpinolene; eucalyptus: cineole/terpinen-4-ol/p-cymene/aromadendrene; fennel: anethole/fenchone/estragole/phellandrene; celery: phthalides).
Explicitly NOT fixable with molecules — worth recording so nobody wastes effort: sweet (0.724, 208 positives, 132 novel), pungent (0.786, 289, 142), musty (0.700, 52), floral (0.772, 39). These have plenty of data and generalize fine; they score low because the category is broad and fuzzy, not because the corpus is thin. More molecules won't teach a model what "sweet-smelling" means structurally — that needs a better representation (#199 GNN) or a narrower definition.
Ran the generalization test from
docs/MOUTHFEEL.md(fire each head over all 8,834 molecules, count hits that were NOT in its training set) across all 164 aroma heads. The roster is mostly healthy — but 16 heads are lookup tables wearing high AUROCs, exactly the pattern that caughttinglingin #247.148 of 164 generalize (median 7 novel discoveries). The strongest are doing real work:
coffeefires on 65 molecules from 12 positives (53 novel),fattyon 96 from 27 (69 novel).16 memorize — 0 novel discoveries. Each fires on exactly its own training molecules:
(+4 more below 0.875.) All sit at 10–15 positives on a single narrow scaffold. A 0.999 AUROC here means "this class is structurally distinctive", not "this head predicts".
Fix: the #247 approach — add structurally diverse positives, not more of the same scaffold. Broadening tingling from one genus to four took it 0 → 2 novel. These heads have rich public chemistry to draw on (pine: pinenes/camphene/carene/bornyl acetate/terpinolene; eucalyptus: cineole/terpinen-4-ol/p-cymene/aromadendrene; fennel: anethole/fenchone/estragole/phellandrene; celery: phthalides).
Explicitly NOT fixable with molecules — worth recording so nobody wastes effort:
sweet(0.724, 208 positives, 132 novel),pungent(0.786, 289, 142),musty(0.700, 52),floral(0.772, 39). These have plenty of data and generalize fine; they score low because the category is broad and fuzzy, not because the corpus is thin. More molecules won't teach a model what "sweet-smelling" means structurally — that needs a better representation (#199 GNN) or a narrower definition.