diff --git a/docs/METHODS.md b/docs/METHODS.md index 4d2ce05..ec628a9 100644 --- a/docs/METHODS.md +++ b/docs/METHODS.md @@ -86,9 +86,47 @@ and a rule is used only where it's *more honest* than a model. Companion to --- +## Does a head LEARN, or just MEMORIZE? +A high cross-validated AUROC is necessary but **not sufficient**, and this is the single easiest +place to fool yourself. A head trained on twelve molecules that all share one scaffold can score +0.99 by recognizing that scaffold and nothing else — it fires on exactly its own training +molecules and stays silent on every other molecule in the corpus. Cross-validation cannot see +this, because it only ever asks about molecules *inside* the labelled set. + +- **Generalization test** — `computed`. Fire the head over the **whole corpus**, then count the + hits that are **not** in its training positives. That count is the head's discovery power. + Run it with [`training/audit_generalization.py`](../training/audit_generalization.py). +- **Two thresholds, because zero is ambiguous.** A head with 13 positives against 2400 negatives + is calibrated conservatively even with balanced class weights — it can have learned its class + and still rarely clear 0.5 outside the molecules it was fit on. So the audit also scores at + **0.35**, which splits one number into two very different diagnoses: + - `novel@0.5 > 0` → **generalizes**. + - `novel@0.5 = 0 < novel@0.35` → **under-confident**. It found real unlabelled molecules just + below the bar. The class is learnable and the head isn't broken; more positives sharpen it. + - `novel@0.35 = 0` → **memorizing**. Fires on its training set and nothing else, at any + threshold. This is the real failure. +- Worth stating plainly because it bit us: `pine` and `rosemary` read as memorizing at 0.5 and + turned out to be under-confident (8 and 9 novel hits at 0.35). `celery` and `turmeric` were + memorizing at both. Same table, opposite verdicts, opposite fixes. +- **Reading the result.** A memorizing head is not worthless — it still labels its own positives + correctly — but it must not be presented as if it can *discover*, and it is not evidence the + model learned the class. +- **The fix is diversity, not volume.** `tingling` trained on nine *Zanthoxylum* sanshools learns + "sanshool"; the same head trained on sanshools **plus** *Echinacea*, *Anacyclus* and *Heliopsis* + amides learns "long-chain unsaturated N-alkylamide" and starts finding molecules nobody + labelled. More of the same scaffold never moves the count off zero. +- **Some heads can't be fixed with molecules.** A broad, fuzzy, multi-scaffold class like `sweet` + **odour** (AUROC 0.724 over 208 positives) isn't thin — it's genuinely hard. The honest answer + there is a better model (a GNN), not a longer list. + ## The honest ceiling on "deeper flavor description" -Taste tops out at the **5 basics + intensity + chemesthesis** on public data. The rich -descriptors people mean by "flavor" — *vanilla, fruity, green, woody, minty, caramel…* — are -**aroma**, not taste: a separate odor model that needs expert-labeled odor data (licensed or -customer; deferred — see [`AROMA.md`](AROMA.md)). That model is the real route to deeper flavor -language, and it's exactly the "comes with your data" piece. +Taste tops out at the **5 basics + intensity + chemesthesis** on public data — that ceiling is +real and hasn't moved. The rich descriptors people mean by "flavor" — *vanilla, fruity, green, +woody, minty, caramel…* — are **aroma**, a separate modality with its own heads (see +[`AROMA.md`](AROMA.md)), now trained from open sources rather than deferred to licensed data. + +What remains gated is **depth**, not vocabulary: the aroma heads are weakly labelled from public +odor text plus a hand-curated character-impact supplement, so they are strongest on the classic, +widely-documented associations and thinnest on the rare naturals — which is precisely what the +generalization test above measures and reports honestly. Expert-labelled odor panel data (licensed +or customer-supplied) is still the route to depth, and it remains the "comes with your data" piece. diff --git a/training/audit_generalization.py b/training/audit_generalization.py new file mode 100644 index 0000000..2c90243 --- /dev/null +++ b/training/audit_generalization.py @@ -0,0 +1,154 @@ +"""audit_generalization.py — does a head LEARN its class, or just memorize its training set? + +A high CV-AUROC is necessary but not sufficient. A head trained on twelve molecules that all share +one scaffold can score 0.99 by recognizing that scaffold and nothing else: it will fire on exactly +the molecules it was trained on and stay silent on every other molecule in the corpus. That head +has memorized. It is not useless — it still labels its own positives correctly — but it cannot +discover, so it must not be presented as if it can. + +The test is deliberately simple and hard to fool: + + fire the head over EVERY molecule in the corpus, then count the hits that are NOT in its + training positives. + +That count — `novel` below — is the head's discovery power. Zero means memorization. The AUROC +never reveals this, because cross-validation only ever asks about molecules inside the labelled +set; this asks what happens outside it. + +The fix for a memorizing head is structural DIVERSITY among its positives, not more of them: a +`tingling` head trained on nine Zanthoxylum sanshools learns "sanshool", whereas the same head +trained on sanshools + Echinacea + Anacyclus + Heliopsis amides learns "long-chain unsaturated +N-alkylamide" and starts finding molecules nobody labelled. See #247 (tingling) and #256 (the +sixteen memorizing aroma heads) for the method applied end to end. + +Some heads CANNOT be fixed with molecules. A broad, fuzzy, multi-scaffold class like `sweet` odour +(AUROC 0.724 over 208 positives) is not thin — it is genuinely hard, and the honest answer is a +better model (a GNN, #28), not a longer list. + +Usage: + python audit_generalization.py # all aroma heads + python audit_generalization.py --mouthfeel # the mouthfeel heads instead + python audit_generalization.py --threshold 0.6 # stricter definition of "fires" +""" +import argparse +import json +import sys +from pathlib import Path + +import numpy as np +import pandas as pd + +FIRE = 0.5 # a head "fires" on a molecule at or above this probability +RELAXED = 0.35 # ...and this is the "would it fire if it were less shy?" probe — see below + +# Why two thresholds. A head with 13 positives against 2400 negatives is calibrated conservatively +# even with balanced class weights: it can have genuinely learned its class and still rarely clear +# 0.5 outside the molecules it was fit on. Reading novel@0.5 alone therefore conflates two very +# different failures — a head that learned nothing, and a head that learned the class but is shy +# about saying so. Scoring both separates them, and the distinction changes what you'd do next: +# novel@0.5 > 0 -> generalizes. Nothing to fix. +# novel@0.5 == 0 < novel@0.35 -> UNDER-CONFIDENT. It found real molecules nobody labelled, +# just below the bar. More positives would firm it up; the +# class itself is learnable and the head is not broken. +# novel@0.35 == 0 -> MEMORIZING. It fires on its training set and nothing +# else at any reasonable threshold. This is the real +# failure, and the fix is structural diversity. +# `pine` and `rosemary` looked memorizing at 0.5 and turned out to be under-confident (8 and 9 +# novel at 0.35); `celery` and `turmeric` were memorizing at both. Same table, opposite verdicts. + + +def _positives(train_path, heads): + """{head -> set of InChIKey skeletons it was trained to call positive}.""" + if not Path(train_path).exists(): + return {} + df = pd.read_parquet(train_path) + cols = [h for h in heads if h in df.columns] + skel = df["inchikey"].astype(str).str.split("-").str[0] + return {h: set(skel[df[h].astype(int) == 1]) for h in cols} + + +def audit(modality="aroma", threshold=FIRE, relaxed=RELAXED): + """Return one row per head: name, auroc, n_pos, hits, novel, novel_lo, verdict.""" + import predict as P + P.MODELS_READY.wait() # heads load on a background thread — don't race it + + models, train_path, manifest = { + "aroma": (P._AROMA_MODELS, "aroma_train.parquet", "aroma_models/manifest.json"), + "mouthfeel": (P._MOUTHFEEL_MODELS, "mouthfeel_train.parquet", "mouthfeel_models/manifest.json"), + }[modality] + if not models: + print(f"no {modality} heads loaded", file=sys.stderr) + return [] + + meta = {} + if Path(manifest).exists(): + meta = json.loads(Path(manifest).read_text()).get("descriptors", {}) + + heads = sorted(models) + pos = _positives(train_path, heads) + + # score every head over the whole corpus in one pass — the enrichment table already holds a + # canonical molecule list, so the audit sees exactly what the app serves + from rdkit import Chem + df = pd.read_parquet("master_enrichment.parquet") + mols = [Chem.MolFromSmiles(str(s)) for s in df["smiles"]] + ok = [i for i, m in enumerate(mols) if m is not None] + x = np.vstack([P._feat(mols[i])[0] for i in ok]) + skel = [Chem.MolToInchiKey(mols[i]).split("-")[0] for i in ok] + + rows = [] + for h in heads: + p = models[h].predict_proba(x)[:, 1] + trained = pos.get(h, set()) + fired = {skel[i] for i in range(len(skel)) if p[i] >= threshold} + fired_lo = {skel[i] for i in range(len(skel)) if p[i] >= relaxed} + novel, novel_lo = len(fired - trained), len(fired_lo - trained) + rows.append({"head": h, + "auroc": meta.get(h, {}).get("auroc"), + "n_pos": len(trained), + "hits": len(fired), + "novel": novel, + "novel_lo": novel_lo, + "verdict": "ok" if novel else ("shy" if novel_lo else "memorizing")}) + return sorted(rows, key=lambda r: (r["novel"], r["novel_lo"], -(r["auroc"] or 0))) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--mouthfeel", action="store_true", help="audit mouthfeel heads instead of aroma") + ap.add_argument("--threshold", type=float, default=FIRE, help=f"fire threshold (default {FIRE})") + ap.add_argument("--relaxed", type=float, default=RELAXED, + help=f"under-confidence probe threshold (default {RELAXED})") + a = ap.parse_args() + + rows = audit("mouthfeel" if a.mouthfeel else "aroma", a.threshold, a.relaxed) + if not rows: + sys.exit(1) + + tag = {"ok": "", "shy": " <- under-confident", "memorizing": " <- MEMORIZING"} + print(f"{'head':16s} {'AUROC':>6s} {'n_pos':>6s} {'hits':>6s} " + f"{'novel':>6s} {f'@{a.relaxed:g}':>6s}") + for r in rows: + au = f"{r['auroc']:.3f}" if r["auroc"] is not None else " - " + print(f"{r['head']:16s} {au:>6s} {r['n_pos']:6d} {r['hits']:6d} " + f"{r['novel']:6d} {r['novel_lo']:6d}{tag[r['verdict']]}") + + shy = [r["head"] for r in rows if r["verdict"] == "shy"] + mem = [r["head"] for r in rows if r["verdict"] == "memorizing"] + novel = sorted(r["novel"] for r in rows) + print(f"\n{len(rows) - len(shy) - len(mem)}/{len(rows)} heads generalize at {a.threshold:g} " + f"(median {novel[len(novel) // 2]} novel discoveries).") + if shy: + print(f"\n{len(shy)} UNDER-CONFIDENT — found unlabelled molecules at {a.relaxed:g} but not " + f"{a.threshold:g}: {', '.join(shy)}") + print(" These learned their class; they're shy because their positives are heavily " + "outnumbered. More positives sharpen them. Not broken.") + if mem: + print(f"\n{len(mem)} MEMORIZING — fire on their own training molecules and nothing else, " + f"at any threshold: {', '.join(mem)}") + print(" Fix: add STRUCTURALLY DIVERSE positives to the curated supplement, then rebuild " + "and re-run. More of the same scaffold will not move these off zero.") + + +if __name__ == "__main__": + main() diff --git a/training/build_aroma_supplement.py b/training/build_aroma_supplement.py index bb6adfd..6e5fc53 100644 --- a/training/build_aroma_supplement.py +++ b/training/build_aroma_supplement.py @@ -82,6 +82,35 @@ "vegetable":["2-isobutyl-3-methoxypyrazine", "2-isopropyl-3-methoxypyrazine", "dimethyl sulfide", "2-acetylpyrrole"], "grassy": ["cis-3-hexenal", "cis-3-hexen-1-ol", "trans-2-hexenal", "hexanal", "trans-2-hexen-1-ol", "cis-3-hexenyl acetate"], + # --- Heads that were MEMORIZING (#256): each fired on exactly its own training molecules and + # nothing else. The fix is structural DIVERSITY within the class, not more of one scaffold — + # the same approach that moved `tingling` off zero in #247. All well-established + # character-impact chemistry for these materials, public-domain flavour/essential-oil facts. + "pine": ["alpha-pinene", "beta-pinene", "camphene", "delta-3-carene", "terpinolene", + "myrcene", "bornyl acetate", "isobornyl acetate", "longifolene", "borneol", + "verbenone", "alpha-terpineol"], + "eucalyptus": ["1,8-cineole", "alpha-terpineol", "terpinen-4-ol", "p-cymene", "aromadendrene", + "globulol", "alpha-phellandrene", "gamma-terpinene", "trans-pinocarveol"], + "fennel": ["anethole", "fenchone", "estragole", "alpha-phellandrene", "anisaldehyde", + "fenchyl alcohol", "limonene", "beta-phellandrene", "camphene"], + "celery": ["3-n-butylphthalide", "sedanolide", "sedanenolide", "neocnidilide", + "beta-selinene", "3-n-butyl-4,5-dihydrophthalide", "senkyunolide A", "ligustilide"], + "rosemary": ["1,8-cineole", "camphor", "borneol", "verbenone", "alpha-pinene", "bornyl acetate", + "camphene", "isoborneol", "rosmarinic acid"], + "turmeric": ["ar-turmerone", "alpha-turmerone", "beta-turmerone", "ar-curcumene", "zingiberene", + "beta-sesquiphellandrene", "curlone", "curcumene"], + "allspice": ["eugenol", "methyl eugenol", "beta-caryophyllene", "1,8-cineole", "chavicol", + "eugenyl acetate", "alpha-phellandrene", "terpinen-4-ol"], + "frankincense": ["incensole", "incensole acetate", "alpha-pinene", "octyl acetate", "verbenone", + "serratol", "cembrene", "alpha-thujene", "octanol"], + "narcissus": ["indole", "p-cresol", "benzyl acetate", "cinnamyl alcohol", "benzyl benzoate", + "methyl benzoate", "alpha-terpineol", "eugenol", "benzyl alcohol"], + "freesia": ["linalool", "alpha-ionone", "beta-ionone", "geraniol", "nerol", "benzaldehyde", + "dihydro-beta-ionone", "linalyl acetate", "citronellol"], + "elemi": ["elemol", "elemicin", "limonene", "alpha-phellandrene", "beta-elemene", + "dill apiole", "elemene", "sabinene"], + "costus": ["costunolide", "dehydrocostus lactone", "costol", "costic acid", + "dihydrocostunolide", "alpha-costene", "aplotaxene"], "green": ["cis-3-hexenal", "trans-2-hexenal", "cis-3-hexen-1-ol", "hexanal"], # pungent (an odor/chemesthesis head, also tagged mouthfeel): sharp biting Piper long/black-pepper # amides on top of the corpus's documented pungent molecules (piperine, isothiocyanates, etc.)