feat(model): per-head calibrated thresholds, with a precision floor (#261)

Every head decided it fired at a flat 0.5. That is a coin-flip line, not a quality bar, and it
was silently withholding real matches from the thin heads: with 13 positives against 2400
negatives a forest hedges even with balanced class weights, so a genuine pine match could land
at 0.42 and never be shown. Each head now carries its own cut-off, fitted on OUT-OF-FOLD
probabilities at training time and stored in its manifest.

The first attempt maximised F1 and was wrong in an instructive way. On a badly imbalanced head
F1 peaks in a low-precision regime, because recall climbs faster than precision falls: blackberry
tuned itself to 0.18, where 96% of its calls were false, and sweet fired on 1317 of 8855
molecules at 18% precision. A high AUROC gives no protection here — AUROC is computed on ranking
and is insensitive to class imbalance, precision is not. coffee has AUROC 0.960 and out-of-fold
precision 0.46; both are true of the same head.

So the threshold must now clear a 50% precision floor: a call labelled confident has to be right
more often than not. Thresholds moved hard in BOTH directions — pine UP to 0.75 where it is
100% precise, rosemary down to 0.16 — and no head sits at either bound, so the data is choosing
rather than the clamp.

73 of 167 aroma heads cannot reach the floor at any threshold. They are marked INDICATIVE, not
deleted: they keep their score, their place in the 178-dim profile, their chips, their map colour
and their molecules, because firing well above base rate is real evidence. What changes is that
they are never dressed up as a confident call — the read returns `indicative` plus the head's
measured precision, and the UI hatches their bar. Hiding them would have deleted reach from 73
notes to paper over a labelling problem. All five mouthfeel heads clear the floor (0.57-1.00).

Applied everywhere a threshold was hard-coded, not just the live read: the precomputed profile
index, the flavor map and the export card each had their own 0.5, and leaving those would have
made a molecule's chips disagree with its own modal. The UI's likely/possible/weak levels are
now read against each head's own threshold too, for the same reason.

audit_generalization.py scores at the calibrated thresholds and grew a fourth verdict:
PRECISION-LIMITED, for a head that is strict on purpose. Calling pine "under-confident" at 0.75
would be exactly backwards — loosening it trades away the accuracy it was tuned for.

Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>

Author: rvnminers-A-and-N

docs/METHODS.md

@@ -119,6 +119,32 @@ this, because it only ever asks about molecules *inside* the labelled set.
   **odour** (AUROC 0.724 over 208 positives) isn't thin — it's genuinely hard. The honest answer
   there is a better model (a GNN), not a longer list.
 
+## When does a head "fire"? (per-head thresholds, and the precision floor)
+Not at a flat 0.5. A head with 13 positives against 2,400 negatives is calibrated conservatively
+even with balanced class weights, so a shared cut-off silently withheld real matches from exactly
+the thin heads — a genuine pine match could land at 0.42 and never be shown.
+
+- **Per-head thresholds** — `computed`. Each head's cut-off is fitted on **out-of-fold**
+  probabilities at training time (`train_aroma._calibrate`) and stored in its manifest. Bounded to
+  [0.15, 0.85]; both directions are allowed, and both are used — `astringent` moved **up** to 0.71,
+  `rosemary` down to 0.16. No head sits at either bound, so the data is choosing, not the clamp.
+- **This changes labels, never numbers.** The raw probability is computed, returned and displayed
+  identically. Only the *confident* flag moves. The threshold is in the API response too.
+- **A precision floor of 0.50, because F1 alone was not safe.** Our first pass maximised F1 and
+  produced thresholds where the head was mostly wrong: `blackberry` tuned to 0.18, where **96% of
+  its calls were false**. On a badly imbalanced head F1 peaks in a low-precision regime, since
+  recall climbs faster than precision falls. **A high AUROC does not protect you** — AUROC is
+  computed on ranking and is insensitive to class imbalance; precision is not. `coffee` has
+  AUROC 0.960 and out-of-fold precision **0.46**; both are true of the same head.
+- **Heads that cannot clear the floor are `indicative`, not deleted.** 73 of 167 aroma heads never
+  reach 50% precision at any threshold. They keep their score, their place in the 178-dim profile,
+  their chips and their map colour — firing well above base rate is real evidence. They are simply
+  never presented as a *confident* call: the UI marks them, and the read returns `indicative: true`
+  plus the head's measured precision. Hiding them would delete reach to paper over a labelling
+  problem; the honest fix is to label them.
+- All five **mouthfeel** heads clear the floor (precision 0.57–1.00), so the whole modality is
+  confident-capable.
+
 ## The honest ceiling on "deeper flavor description"
 Taste tops out at the **5 basics + intensity + chemesthesis** on public data — that ceiling is
 real and hasn't moved. The rich descriptors people mean by "flavor" — *vanilla, fruity, green,

training/app.py

@@ -589,16 +589,19 @@ def font(path, size):
     # reader knows they're seeing the firing subset, not the whole model.
     AROMA_CAP = 18                                   # 3 rows of 6 — keeps the card readable
     pa = P.predict_aroma(smi)
-    _all_aroma = sorted(((d["odor"], d["score"]) for d in pa.get("descriptors", [])),
-                        key=lambda kv: -kv[1])
-    _fired = [c for c in _all_aroma if c[1] >= 0.5]
+    # rank by score but decide "fired" from the head's OWN calibrated threshold, which
+    # predict_aroma already applied — re-thresholding at a flat 0.5 here would disagree with the
+    # modal for exactly the thin heads that needed calibrating
+    _descs = sorted(pa.get("descriptors", []), key=lambda d: -d["score"])
+    _all_aroma = [(d["odor"], d["score"]) for d in _descs]
+    _fired = [(d["odor"], d["score"]) for d in _descs if d.get("confident")]
     aroma_cells = (_fired or _all_aroma[:3])[:AROMA_CAP]   # nothing firing -> top 3, never a blank card
     aroma_total, aroma_fired = len(_all_aroma), len(_fired)
 
     _mol = Chem.MolFromSmiles(smi)
     mouth_cells = [(d["sensation"], d["score"])
                    for d in (P.predict_mouthfeel(_mol).get("descriptors", []) if _mol else [])
-                   if d["score"] >= 0.5]
+                   if d.get("confident")]
     tox_cells = [(a["assay"], a["probability"])
                  for a in ((out.get("safety") or {}).get("tox_screen") or {}).get("assays", [])
                  if (a.get("probability") or 0) >= 0.5]

training/audit_generalization.py

@@ -100,7 +100,11 @@ def audit(modality="aroma", threshold=FIRE, relaxed=RELAXED):
     for h in heads:
         p = models[h].predict_proba(x)[:, 1]
         trained = pos.get(h, set())
-        fired = {skel[i] for i in range(len(skel)) if p[i] >= threshold}
+        # score at the head's OWN calibrated threshold when it has one, so the audit measures the
+        # product's actual behaviour rather than a hypothetical flat cut-off it no longer uses
+        thr = meta.get(h, {}).get("threshold")
+        thr = float(thr) if isinstance(thr, (int, float)) else threshold
+        fired = {skel[i] for i in range(len(skel)) if p[i] >= thr}
         fired_lo = {skel[i] for i in range(len(skel)) if p[i] >= relaxed}
         novel, novel_lo = len(fired - trained), len(fired_lo - trained)
         rows.append({"head": h,
@@ -109,7 +113,19 @@ def audit(modality="aroma", threshold=FIRE, relaxed=RELAXED):
                      "hits": len(fired),
                      "novel": novel,
                      "novel_lo": novel_lo,
-                     "verdict": "ok" if novel else ("shy" if novel_lo else "memorizing")})
+                     "threshold": round(thr, 2),
+                     "precision": meta.get(h, {}).get("cv_precision"),
+                     "capable": meta.get(h, {}).get("confident_capable", True),
+                     # A head whose calibrated threshold sits ABOVE the relaxed probe is not shy —
+                     # it was deliberately made strict to reach the precision floor, and loosening
+                     # it would trade away the accuracy it was tuned for. That is precision-limited,
+                     # a different diagnosis with a different fix (more positives, not a lower bar).
+                     # Order matters: a head that finds nothing at EITHER threshold is memorizing,
+                     # whatever its cut-off. Only once we know relaxing would actually find
+                     # something does a high threshold mean "strict on purpose" rather than "broken".
+                     "verdict": ("ok" if novel else
+                                 "memorizing" if not novel_lo else
+                                 "precision-limited" if thr > relaxed else "shy")})
     return sorted(rows, key=lambda r: (r["novel"], r["novel_lo"], -(r["auroc"] or 0)))
 
 
@@ -125,24 +141,39 @@ def main():
     if not rows:
         sys.exit(1)
 
-    tag = {"ok": "", "shy": "  <- under-confident", "memorizing": "  <- MEMORIZING"}
-    print(f"{'head':16s} {'AUROC':>6s} {'n_pos':>6s} {'hits':>6s} "
+    tag = {"ok": "", "shy": "  <- under-confident",
+           "precision-limited": "  <- precision-limited (strict on purpose)",
+           "memorizing": "  <- MEMORIZING"}
+    print(f"{'head':16s} {'AUROC':>6s} {'thr':>5s} {'prec':>5s} {'n_pos':>6s} {'hits':>6s} "
           f"{'novel':>6s} {f'@{a.relaxed:g}':>6s}")
     for r in rows:
         au = f"{r['auroc']:.3f}" if r["auroc"] is not None else "  -  "
-        print(f"{r['head']:16s} {au:>6s} {r['n_pos']:6d} {r['hits']:6d} "
-              f"{r['novel']:6d} {r['novel_lo']:6d}{tag[r['verdict']]}")
+        pc = f"{r['precision']:.2f}" if r["precision"] is not None else "  -  "
+        ind = "" if r["capable"] else "  (indicative)"
+        print(f"{r['head']:16s} {au:>6s} {r['threshold']:5.2f} {pc:>5s} {r['n_pos']:6d} "
+              f"{r['hits']:6d} {r['novel']:6d} {r['novel_lo']:6d}{tag[r['verdict']]}{ind}")
 
+    indicative = [r["head"] for r in rows if not r["capable"]]
     shy = [r["head"] for r in rows if r["verdict"] == "shy"]
+    plim = [r["head"] for r in rows if r["verdict"] == "precision-limited"]
     mem = [r["head"] for r in rows if r["verdict"] == "memorizing"]
     novel = sorted(r["novel"] for r in rows)
-    print(f"\n{len(rows) - len(shy) - len(mem)}/{len(rows)} heads generalize at {a.threshold:g} "
-          f"(median {novel[len(novel) // 2]} novel discoveries).")
+    print(f"\n{len(rows) - len(shy) - len(plim) - len(mem)}/{len(rows)} heads generalize at their "
+          f"calibrated threshold (median {novel[len(novel) // 2]} novel discoveries).")
+    if plim:
+        print(f"\n{len(plim)} PRECISION-LIMITED — strict by design, to hold the 50% precision "
+              f"floor: {', '.join(plim)}")
+        print("  Not a defect and not a shy head: lowering the bar would trade away the accuracy "
+              "it was tuned for. More positives is the only real fix.")
     if shy:
         print(f"\n{len(shy)} UNDER-CONFIDENT — found unlabelled molecules at {a.relaxed:g} but not "
               f"{a.threshold:g}: {', '.join(shy)}")
         print("  These learned their class; they're shy because their positives are heavily "
               "outnumbered. More positives sharpen them. Not broken.")
+    if indicative:
+        print(f"\n{len(indicative)}/{len(rows)} are INDICATIVE — they never reach 50% out-of-fold "
+              f"precision at any threshold, so they fire as evidence, never as a confident call. "
+              f"They are still far better than the base rate; they are not yet trustworthy alone.")
     if mem:
         print(f"\n{len(mem)} MEMORIZING — fire on their own training molecules and nothing else, "
               f"at any threshold: {', '.join(mem)}")

training/build_flavor_map.py

@@ -172,21 +172,27 @@ def _all_structures():
                 dp = doc.get(ik)
                 if dp:                                   # rarest documented aroma (tie-break by model)
                     best[i] = min(dp, key=lambda h: (docfreq[h], -prob[h][i]))
-                else:                                    # else the strongest predicted head >= 0.5
-                    bp = 0.5
+                else:  # else the strongest head clearing its OWN calibrated threshold
+                    # a flat 0.5 here left thin heads uncoloured on the map even where they were
+                    # the best available call — see docs/METHODS.md on per-head thresholds
+                    bp = 0.0
+                    # indicative heads still colour points — the map is exploratory, and greying
+                    # out 73 heads' molecules would hide more than it clarifies
                     for name in heads:
-                        if prob[name][i] >= bp:
-                            bp, best[i] = prob[name][i], name
+                        p = prob[name][i]
+                        if p >= P._head_threshold(P._AROMA_META, name) and p > bp:
+                            bp, best[i] = p, name
             out["aroma_label"] = best
-            # dominant MOUTHFEEL sensation (for the map's "color by mouthfeel" mode). Straight
-            # strongest-head-over-0.5: unlike aroma there's no documented corpus to prefer, and
-            # with only five heads there's no rarity problem to correct for.
+            # dominant MOUTHFEEL sensation (for the map's "color by mouthfeel" mode): the strongest
+            # head that clears its own calibrated threshold. Unlike aroma there's no documented
+            # corpus to prefer, and with only five heads there's no rarity problem to correct for.
             if P._MOUTHFEEL_MODELS:
-                mbest, mbp = ["other"] * len(out), [0.5] * len(out)
+                mbest, mbp = ["other"] * len(out), [0.0] * len(out)
                 for name, clf in P._MOUTHFEEL_MODELS.items():
                     col = clf.predict_proba(Xf)[:, 1]
+                    thr = P._head_threshold(P._MOUTHFEEL_META, name)
                     for i in range(len(out)):
-                        if col[i] >= mbp[i]:
+                        if col[i] >= thr and col[i] > mbp[i]:
                             mbp[i], mbest[i] = col[i], name
                 out["mouthfeel_label"] = mbest
     out.to_parquet("flavor_map.parquet")

training/build_profile_index.py

@@ -11,7 +11,7 @@
 Output profile_index.npz:
   smiles            (N,)  canonical SMILES, deduped by connectivity skeleton
   taste_documented  (N,)  comma-separated documented tastes ("" if none)
-  aromas            (N,)  comma-separated confident aroma heads (score >= 0.5)
+  aromas            (N,)  comma-separated confident aroma heads (score >= that head's calibrated threshold)
   profiles          (N,D) float32 head-score matrix: taste heads then aroma heads (sorted names)
   dims              (D,)  column labels ("taste:sweet", "aroma:citrus", ...)
 
@@ -106,11 +106,22 @@ def main():
     dims = ([f"taste:{t}" for t in taste] + [f"aroma:{a}" for a in aroma]
             + [f"mouthfeel:{h}" for h in mouth])
     profiles = np.column_stack([scores[k] for k in dims]).astype("float32")  # scores keyed by dim
+    # Per-head calibrated thresholds, read straight from the manifest: this module runs under
+    # NO_MODELS so predict's own _AROMA_META is empty, and defaulting to a flat 0.5 here would make
+    # the precomputed chip list disagree with a live read for exactly the thin heads (#261).
+    import json
+    _mf = AROMA_DIR / "manifest.json"
+    aroma_meta = json.loads(_mf.read_text()).get("descriptors", {}) if _mf.exists() else {}
     aromas = [[] for _ in smis]
+    # Indicative heads (those that never reach 50% out-of-fold precision) are INCLUDED here on
+    # purpose. Dropping them would silently delete 73 of 167 notes from chip search and palette
+    # match — a real loss of reach to avoid a labelling problem. The honest fix is to mark them,
+    # which the read does (`indicative` per descriptor), not to hide the molecules they find.
     for a in aroma:
         col = scores[f"aroma:{a}"]
+        thr = P._head_threshold(aroma_meta, a)
         for i in range(len(smis)):
-            if col[i] >= 0.5:
+            if col[i] >= thr:
                 aromas[i].append(a)
     tmp_out = "profile_index.building.npz"  # write then atomically replace so the live index is never half-written
     np.savez_compressed(

training/predict.py

@@ -1101,8 +1101,38 @@ def _aroma_scores(smiles):
     return _aroma_scores_canon(Chem.MolToSmiles(m))
 
 
+def _head_threshold(meta, name, override=None):
+    """The probability at or above which a head counts as FIRING.
+
+    Not a flat 0.5. Each head carries its own threshold, fitted on out-of-fold predictions at
+    training time (train_aroma._calibrate) and stored in its manifest. The thin heads need this:
+    with 13 positives against 2400 negatives a forest hedges, so a genuine pine match can land at
+    0.42 and a flat cut-off would silently withhold it — while a head with 800 positives has no
+    such problem and keeps a threshold near 0.5.
+
+    This decides only whether a descriptor is marked *confident*. The raw probability is returned
+    and displayed either way, so nothing is hidden and nothing is inflated.
+    """
+    if override is not None:
+        return override
+    t = meta.get(name, {}).get("threshold")
+    return float(t) if isinstance(t, (int, float)) else 0.5
+
+
+def _head_capable(meta, name):
+    """Whether this head may be presented as making a CONFIDENT call.
+
+    False for heads that never reach 50% out-of-fold precision at any threshold — they are right
+    less than half the time when they fire, so calling them confident would be a lie no matter
+    where the cut-off sits. Such a head keeps its score and its place in the profile (it is still
+    real evidence, and often far better than the base rate); it is reported as INDICATIVE instead.
+    Heads trained before calibration shipped have no flag, and are trusted as before.
+    """
+    return meta.get(name, {}).get("confident_capable", True)
+
+
 @lru_cache(maxsize=8192)
-def predict_aroma(smiles, top_k=8, threshold=0.5):
+def predict_aroma(smiles, top_k=8, threshold=None):
     """Predicted odor descriptors from RandomForest heads trained on the PUBLIC-DOMAIN HSDB
     odor corpus (see docs/AROMA.md). Returns the descriptors the model scores above threshold,
     each with its probability and the head's CV-AUROC. This is PRESENCE/ABSENCE (the free-text
@@ -1115,17 +1145,27 @@ def predict_aroma(smiles, top_k=8, threshold=0.5):
     if scores is None:
         return {"available": False,
                 "note": "aroma model not trained here — build with train_aroma.py"}
-    preds = [{"odor": name, "score": p, "confident": p >= threshold,
-              "auroc": _AROMA_META.get(name, {}).get("auroc"),
-              "desc": AROMA_DESC.get(name)}
-             for name, p in scores.items()]
+    preds = []
+    for name, p in scores.items():
+        thr = _head_threshold(_AROMA_META, name, threshold)
+        fires, capable = p >= thr, _head_capable(_AROMA_META, name)
+        preds.append({"odor": name, "score": p, "threshold": thr,
+                      # a head that never reaches 50% out-of-fold precision fires as INDICATIVE,
+                      # never as confident — see _head_capable
+                      "confident": fires and capable,
+                      "indicative": fires and not capable,
+                      "precision": _AROMA_META.get(name, {}).get("cv_precision"),
+                      "auroc": _AROMA_META.get(name, {}).get("auroc"),
+                      "desc": AROMA_DESC.get(name)})
     preds.sort(key=lambda d: -d["score"])
     # Return EVERY head (like the taste meters list every taste), ranked, each flagged confident
     # or not — so the read shows the full aroma profile across all trained descriptor models, not
     # just the ones that fired. `top` is the confident shortlist for compact tag uses elsewhere.
     confident = [d for d in preds if d["confident"]]
+    indicative = [d for d in preds if d["indicative"]]
     return {"available": True, "predicted": True, "descriptors": preds,
             "top": (confident or preds[:3]), "any_confident": bool(confident),
+            "indicative": indicative,
             "note": "presence/absence model on public-domain HSDB odor text; not intensity"}
 
 
@@ -1150,7 +1190,11 @@ def predict_mouthfeel(mol):
     preds = []
     for name, clf in sorted(_MOUTHFEEL_MODELS.items()):
         p = round(float(clf.predict_proba(x)[0, 1]), 3)
-        preds.append({"sensation": name, "score": p, "confident": p >= 0.5,
+        thr = _head_threshold(_MOUTHFEEL_META, name)
+        fires, capable = p >= thr, _head_capable(_MOUTHFEEL_META, name)
+        preds.append({"sensation": name, "score": p, "threshold": thr,
+                      "confident": fires and capable, "indicative": fires and not capable,
+                      "precision": _MOUTHFEEL_META.get(name, {}).get("cv_precision"),
                       "auroc": _MOUTHFEEL_META.get(name, {}).get("auroc"),
                       "desc": _MOUTHFEEL_DESC.get(name)})
     preds.sort(key=lambda d: -d["score"])
@@ -1314,8 +1358,11 @@ def _build_sub_index():
             for name in aroma_heads:
                 col = _AROMA_MODELS[name].predict_proba(X)[:, 1]
                 cols.append(col)
+                # per-head threshold, same as a live read — otherwise the precomputed chip list
+                # and the on-the-fly one disagree for exactly the thin heads this fixes
+                thr = _head_threshold(_AROMA_META, name)
                 for i in range(len(smis)):
-                    if col[i] >= 0.5:
+                    if col[i] >= thr:
                         aromas[i].append(name)
             cols += [_MOUTHFEEL_MODELS[h].predict_proba(X)[:, 1] for h in mouthfeel_heads]
             profile_dims = ([f"taste:{t}" for t in taste_heads] + [f"aroma:{a}" for a in aroma_heads]