Broaden the memorizing aroma heads, and measure whether it worked

docs/METHODS.md

@@ -86,9 +86,47 @@ and a rule is used only where it's *more honest* than a model. Companion to
 
 ---
 
+## Does a head LEARN, or just MEMORIZE?
+A high cross-validated AUROC is necessary but **not sufficient**, and this is the single easiest
+place to fool yourself. A head trained on twelve molecules that all share one scaffold can score
+0.99 by recognizing that scaffold and nothing else — it fires on exactly its own training
+molecules and stays silent on every other molecule in the corpus. Cross-validation cannot see
+this, because it only ever asks about molecules *inside* the labelled set.
+
+- **Generalization test** — `computed`. Fire the head over the **whole corpus**, then count the
+  hits that are **not** in its training positives. That count is the head's discovery power.
+  Run it with [`training/audit_generalization.py`](../training/audit_generalization.py).
+- **Two thresholds, because zero is ambiguous.** A head with 13 positives against 2400 negatives
+  is calibrated conservatively even with balanced class weights — it can have learned its class
+  and still rarely clear 0.5 outside the molecules it was fit on. So the audit also scores at
+  **0.35**, which splits one number into two very different diagnoses:
+  - `novel@0.5 > 0` → **generalizes**.
+  - `novel@0.5 = 0 < novel@0.35` → **under-confident**. It found real unlabelled molecules just
+    below the bar. The class is learnable and the head isn't broken; more positives sharpen it.
+  - `novel@0.35 = 0` → **memorizing**. Fires on its training set and nothing else, at any
+    threshold. This is the real failure.
+- Worth stating plainly because it bit us: `pine` and `rosemary` read as memorizing at 0.5 and
+  turned out to be under-confident (8 and 9 novel hits at 0.35). `celery` and `turmeric` were
+  memorizing at both. Same table, opposite verdicts, opposite fixes.
+- **Reading the result.** A memorizing head is not worthless — it still labels its own positives
+  correctly — but it must not be presented as if it can *discover*, and it is not evidence the
+  model learned the class.
+- **The fix is diversity, not volume.** `tingling` trained on nine *Zanthoxylum* sanshools learns
+  "sanshool"; the same head trained on sanshools **plus** *Echinacea*, *Anacyclus* and *Heliopsis*
+  amides learns "long-chain unsaturated N-alkylamide" and starts finding molecules nobody
+  labelled. More of the same scaffold never moves the count off zero.
+- **Some heads can't be fixed with molecules.** A broad, fuzzy, multi-scaffold class like `sweet`
+  **odour** (AUROC 0.724 over 208 positives) isn't thin — it's genuinely hard. The honest answer
+  there is a better model (a GNN), not a longer list.
+
 ## The honest ceiling on "deeper flavor description"
-Taste tops out at the **5 basics + intensity + chemesthesis** on public data. The rich
-descriptors people mean by "flavor" — *vanilla, fruity, green, woody, minty, caramel…* — are
-**aroma**, not taste: a separate odor model that needs expert-labeled odor data (licensed or
-customer; deferred — see [`AROMA.md`](AROMA.md)). That model is the real route to deeper flavor
-language, and it's exactly the "comes with your data" piece.
+Taste tops out at the **5 basics + intensity + chemesthesis** on public data — that ceiling is
+real and hasn't moved. The rich descriptors people mean by "flavor" — *vanilla, fruity, green,
+woody, minty, caramel…* — are **aroma**, a separate modality with its own heads (see
+[`AROMA.md`](AROMA.md)), now trained from open sources rather than deferred to licensed data.
+
+What remains gated is **depth**, not vocabulary: the aroma heads are weakly labelled from public
+odor text plus a hand-curated character-impact supplement, so they are strongest on the classic,
+widely-documented associations and thinnest on the rare naturals — which is precisely what the
+generalization test above measures and reports honestly. Expert-labelled odor panel data (licensed
+or customer-supplied) is still the route to depth, and it remains the "comes with your data" piece.

training/audit_generalization.py

@@ -0,0 +1,154 @@
+"""audit_generalization.py — does a head LEARN its class, or just memorize its training set?
+
+A high CV-AUROC is necessary but not sufficient. A head trained on twelve molecules that all share
+one scaffold can score 0.99 by recognizing that scaffold and nothing else: it will fire on exactly
+the molecules it was trained on and stay silent on every other molecule in the corpus. That head
+has memorized. It is not useless — it still labels its own positives correctly — but it cannot
+discover, so it must not be presented as if it can.
+
+The test is deliberately simple and hard to fool:
+
+    fire the head over EVERY molecule in the corpus, then count the hits that are NOT in its
+    training positives.
+
+That count — `novel` below — is the head's discovery power. Zero means memorization. The AUROC
+never reveals this, because cross-validation only ever asks about molecules inside the labelled
+set; this asks what happens outside it.
+
+The fix for a memorizing head is structural DIVERSITY among its positives, not more of them: a
+`tingling` head trained on nine Zanthoxylum sanshools learns "sanshool", whereas the same head
+trained on sanshools + Echinacea + Anacyclus + Heliopsis amides learns "long-chain unsaturated
+N-alkylamide" and starts finding molecules nobody labelled. See #247 (tingling) and #256 (the
+sixteen memorizing aroma heads) for the method applied end to end.
+
+Some heads CANNOT be fixed with molecules. A broad, fuzzy, multi-scaffold class like `sweet` odour
+(AUROC 0.724 over 208 positives) is not thin — it is genuinely hard, and the honest answer is a
+better model (a GNN, #28), not a longer list.
+
+Usage:
+    python audit_generalization.py                 # all aroma heads
+    python audit_generalization.py --mouthfeel     # the mouthfeel heads instead
+    python audit_generalization.py --threshold 0.6 # stricter definition of "fires"
+"""
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+
+FIRE = 0.5   # a head "fires" on a molecule at or above this probability
+RELAXED = 0.35  # ...and this is the "would it fire if it were less shy?" probe — see below
+
+# Why two thresholds. A head with 13 positives against 2400 negatives is calibrated conservatively
+# even with balanced class weights: it can have genuinely learned its class and still rarely clear
+# 0.5 outside the molecules it was fit on. Reading novel@0.5 alone therefore conflates two very
+# different failures — a head that learned nothing, and a head that learned the class but is shy
+# about saying so. Scoring both separates them, and the distinction changes what you'd do next:
+#   novel@0.5  > 0                      -> generalizes. Nothing to fix.
+#   novel@0.5 == 0 < novel@0.35         -> UNDER-CONFIDENT. It found real molecules nobody labelled,
+#                                          just below the bar. More positives would firm it up; the
+#                                          class itself is learnable and the head is not broken.
+#   novel@0.35 == 0                     -> MEMORIZING. It fires on its training set and nothing
+#                                          else at any reasonable threshold. This is the real
+#                                          failure, and the fix is structural diversity.
+# `pine` and `rosemary` looked memorizing at 0.5 and turned out to be under-confident (8 and 9
+# novel at 0.35); `celery` and `turmeric` were memorizing at both. Same table, opposite verdicts.
+
+
+def _positives(train_path, heads):
+    """{head -> set of InChIKey skeletons it was trained to call positive}."""
+    if not Path(train_path).exists():
+        return {}
+    df = pd.read_parquet(train_path)
+    cols = [h for h in heads if h in df.columns]
+    skel = df["inchikey"].astype(str).str.split("-").str[0]
+    return {h: set(skel[df[h].astype(int) == 1]) for h in cols}
+
+
+def audit(modality="aroma", threshold=FIRE, relaxed=RELAXED):
+    """Return one row per head: name, auroc, n_pos, hits, novel, novel_lo, verdict."""
+    import predict as P
+    P.MODELS_READY.wait()  # heads load on a background thread — don't race it
+
+    models, train_path, manifest = {
+        "aroma": (P._AROMA_MODELS, "aroma_train.parquet", "aroma_models/manifest.json"),
+        "mouthfeel": (P._MOUTHFEEL_MODELS, "mouthfeel_train.parquet", "mouthfeel_models/manifest.json"),
+    }[modality]
+    if not models:
+        print(f"no {modality} heads loaded", file=sys.stderr)
+        return []
+
+    meta = {}
+    if Path(manifest).exists():
+        meta = json.loads(Path(manifest).read_text()).get("descriptors", {})
+
+    heads = sorted(models)
+    pos = _positives(train_path, heads)
+
+    # score every head over the whole corpus in one pass — the enrichment table already holds a
+    # canonical molecule list, so the audit sees exactly what the app serves
+    from rdkit import Chem
+    df = pd.read_parquet("master_enrichment.parquet")
+    mols = [Chem.MolFromSmiles(str(s)) for s in df["smiles"]]
+    ok = [i for i, m in enumerate(mols) if m is not None]
+    x = np.vstack([P._feat(mols[i])[0] for i in ok])
+    skel = [Chem.MolToInchiKey(mols[i]).split("-")[0] for i in ok]
+
+    rows = []
+    for h in heads:
+        p = models[h].predict_proba(x)[:, 1]
+        trained = pos.get(h, set())
+        fired = {skel[i] for i in range(len(skel)) if p[i] >= threshold}
+        fired_lo = {skel[i] for i in range(len(skel)) if p[i] >= relaxed}
+        novel, novel_lo = len(fired - trained), len(fired_lo - trained)
+        rows.append({"head": h,
+                     "auroc": meta.get(h, {}).get("auroc"),
+                     "n_pos": len(trained),
+                     "hits": len(fired),
+                     "novel": novel,
+                     "novel_lo": novel_lo,
+                     "verdict": "ok" if novel else ("shy" if novel_lo else "memorizing")})
+    return sorted(rows, key=lambda r: (r["novel"], r["novel_lo"], -(r["auroc"] or 0)))
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    ap.add_argument("--mouthfeel", action="store_true", help="audit mouthfeel heads instead of aroma")
+    ap.add_argument("--threshold", type=float, default=FIRE, help=f"fire threshold (default {FIRE})")
+    ap.add_argument("--relaxed", type=float, default=RELAXED,
+                    help=f"under-confidence probe threshold (default {RELAXED})")
+    a = ap.parse_args()
+
+    rows = audit("mouthfeel" if a.mouthfeel else "aroma", a.threshold, a.relaxed)
+    if not rows:
+        sys.exit(1)
+
+    tag = {"ok": "", "shy": "  <- under-confident", "memorizing": "  <- MEMORIZING"}
+    print(f"{'head':16s} {'AUROC':>6s} {'n_pos':>6s} {'hits':>6s} "
+          f"{'novel':>6s} {f'@{a.relaxed:g}':>6s}")
+    for r in rows:
+        au = f"{r['auroc']:.3f}" if r["auroc"] is not None else "  -  "
+        print(f"{r['head']:16s} {au:>6s} {r['n_pos']:6d} {r['hits']:6d} "
+              f"{r['novel']:6d} {r['novel_lo']:6d}{tag[r['verdict']]}")
+
+    shy = [r["head"] for r in rows if r["verdict"] == "shy"]
+    mem = [r["head"] for r in rows if r["verdict"] == "memorizing"]
+    novel = sorted(r["novel"] for r in rows)
+    print(f"\n{len(rows) - len(shy) - len(mem)}/{len(rows)} heads generalize at {a.threshold:g} "
+          f"(median {novel[len(novel) // 2]} novel discoveries).")
+    if shy:
+        print(f"\n{len(shy)} UNDER-CONFIDENT — found unlabelled molecules at {a.relaxed:g} but not "
+              f"{a.threshold:g}: {', '.join(shy)}")
+        print("  These learned their class; they're shy because their positives are heavily "
+              "outnumbered. More positives sharpen them. Not broken.")
+    if mem:
+        print(f"\n{len(mem)} MEMORIZING — fire on their own training molecules and nothing else, "
+              f"at any threshold: {', '.join(mem)}")
+        print("  Fix: add STRUCTURALLY DIVERSE positives to the curated supplement, then rebuild "
+              "and re-run. More of the same scaffold will not move these off zero.")
+
+
+if __name__ == "__main__":
+    main()

training/build_aroma_supplement.py

@@ -82,6 +82,35 @@
     "vegetable":["2-isobutyl-3-methoxypyrazine", "2-isopropyl-3-methoxypyrazine", "dimethyl sulfide", "2-acetylpyrrole"],
     "grassy":   ["cis-3-hexenal", "cis-3-hexen-1-ol", "trans-2-hexenal", "hexanal",
                  "trans-2-hexen-1-ol", "cis-3-hexenyl acetate"],
+    # --- Heads that were MEMORIZING (#256): each fired on exactly its own training molecules and
+    # nothing else. The fix is structural DIVERSITY within the class, not more of one scaffold —
+    # the same approach that moved `tingling` off zero in #247. All well-established
+    # character-impact chemistry for these materials, public-domain flavour/essential-oil facts.
+    "pine":     ["alpha-pinene", "beta-pinene", "camphene", "delta-3-carene", "terpinolene",
+                 "myrcene", "bornyl acetate", "isobornyl acetate", "longifolene", "borneol",
+                 "verbenone", "alpha-terpineol"],
+    "eucalyptus": ["1,8-cineole", "alpha-terpineol", "terpinen-4-ol", "p-cymene", "aromadendrene",
+                 "globulol", "alpha-phellandrene", "gamma-terpinene", "trans-pinocarveol"],
+    "fennel":   ["anethole", "fenchone", "estragole", "alpha-phellandrene", "anisaldehyde",
+                 "fenchyl alcohol", "limonene", "beta-phellandrene", "camphene"],
+    "celery":   ["3-n-butylphthalide", "sedanolide", "sedanenolide", "neocnidilide",
+                 "beta-selinene", "3-n-butyl-4,5-dihydrophthalide", "senkyunolide A", "ligustilide"],
+    "rosemary": ["1,8-cineole", "camphor", "borneol", "verbenone", "alpha-pinene", "bornyl acetate",
+                 "camphene", "isoborneol", "rosmarinic acid"],
+    "turmeric": ["ar-turmerone", "alpha-turmerone", "beta-turmerone", "ar-curcumene", "zingiberene",
+                 "beta-sesquiphellandrene", "curlone", "curcumene"],
+    "allspice": ["eugenol", "methyl eugenol", "beta-caryophyllene", "1,8-cineole", "chavicol",
+                 "eugenyl acetate", "alpha-phellandrene", "terpinen-4-ol"],
+    "frankincense": ["incensole", "incensole acetate", "alpha-pinene", "octyl acetate", "verbenone",
+                 "serratol", "cembrene", "alpha-thujene", "octanol"],
+    "narcissus": ["indole", "p-cresol", "benzyl acetate", "cinnamyl alcohol", "benzyl benzoate",
+                 "methyl benzoate", "alpha-terpineol", "eugenol", "benzyl alcohol"],
+    "freesia":  ["linalool", "alpha-ionone", "beta-ionone", "geraniol", "nerol", "benzaldehyde",
+                 "dihydro-beta-ionone", "linalyl acetate", "citronellol"],
+    "elemi":    ["elemol", "elemicin", "limonene", "alpha-phellandrene", "beta-elemene",
+                 "dill apiole", "elemene", "sabinene"],
+    "costus":   ["costunolide", "dehydrocostus lactone", "costol", "costic acid",
+                 "dihydrocostunolide", "alpha-costene", "aplotaxene"],
     "green":    ["cis-3-hexenal", "trans-2-hexenal", "cis-3-hexen-1-ol", "hexanal"],
     # pungent (an odor/chemesthesis head, also tagged mouthfeel): sharp biting Piper long/black-pepper
     # amides on top of the corpus's documented pungent molecules (piperine, isothiocyanates, etc.)