Per-head calibrated thresholds, with a precision floor by rvnminers-A-and-N · Pull Request #261 · echelonts/flavormancer

rvnminers-A-and-N · 2026-07-29T15:25:31Z

Every head decided it fired at a flat 0.5. That's a coin-flip line, not a quality bar — and it was silently withholding real matches from the thin heads. With 13 positives against 2,400 negatives a forest hedges even with balanced class weights, so a genuine pine match could land at 0.42 and never be shown.

Each head now carries its own cut-off, fitted on out-of-fold probabilities at training time and stored in its manifest.

The first attempt was wrong, instructively

Maximising F1 produced thresholds where heads were mostly wrong:

head	F1-optimal thr	out-of-fold precision
`blackberry`	0.18	0.04
`cassis`	0.16	0.10
`sweet`	0.21	0.18
`grassy`	0.23	0.15

sweet would have fired on 1,317 of 8,855 molecules. On a badly imbalanced head F1 peaks in a low-precision regime, because recall climbs faster than precision falls.

A high AUROC gives no protection here. AUROC is computed on ranking and is insensitive to class imbalance; precision is not. coffee has AUROC 0.960 and out-of-fold precision 0.46 — both true of the same head. That's worth internalising beyond this PR.

So: a 50% precision floor

A call labelled confident must be right more often than not. Thresholds then moved hard in both directions — pine up to 0.75 where it's 100% precise, rosemary down to 0.16 — and no head sits at either bound (min 0.16, max 0.85), so the data is choosing, not the clamp.

73 heads can't reach the floor. They're marked, not removed

73 of 167 aroma heads never reach 50% precision at any threshold. They keep their score, their place in the 178-dim profile, their chips, their map colour, and every molecule they find — firing well above base rate is real evidence. What changes is that they're never dressed up as confident: the read returns indicative: true plus the head's measured precision, and the UI hatches their bar.

I initially gated chips and map colour on capability and reverted it — that would have deleted reach from 73 notes to paper over a labelling problem. Mark, don't hide.

All five mouthfeel heads clear the floor (precision 0.57–1.00).

Applied everywhere, not just the live read

The precomputed profile index, the flavor map and the export card each had their own hard-coded 0.5. Leaving those would have made an in-corpus molecule's chips disagree with its own modal. The UI's likely/possible/weak levels are read against each head's own threshold now, for the same reason.

Audit tool

audit_generalization.py scores at the calibrated thresholds and grew a fourth verdict — PRECISION-LIMITED, for a head that is strict on purpose. Calling pine "under-confident" at 0.75 would be exactly backwards: loosening it trades away the accuracy it was tuned for.

Current roster across 167 aroma heads:

verdict	count
generalizes at its calibrated threshold	146 (median 9 novel discoveries)
precision-limited (strict by design)	19
memorizing	2 — `turmeric`, `celery`
(cross-cutting) indicative	73

Bug caught in passing

Inserting the threshold helper immediately above predict_aroma put it between an @lru_cache decorator and the function it decorated. The helper became memoised (and threw on its dict argument, which is how it surfaced) and predict_aroma silently lost its cache — the fix behind the 40s→0.01s modal. Both restored. Had the helper taken hashable arguments this would have shipped as a silent performance regression.

Verification

ruff check training/ tests/ mcp-server/ clean; full suite 43 passed
artifacts rebuilt against calibrated heads: master_enrichment 8,855 · profile_index 8,836 × 178 · flavor_map 8,836
needs sudo systemctl restart flavormancer — the running app still holds the pre-calibration heads

Every head decided it fired at a flat 0.5. That is a coin-flip line, not a quality bar, and it was silently withholding real matches from the thin heads: with 13 positives against 2400 negatives a forest hedges even with balanced class weights, so a genuine pine match could land at 0.42 and never be shown. Each head now carries its own cut-off, fitted on OUT-OF-FOLD probabilities at training time and stored in its manifest. The first attempt maximised F1 and was wrong in an instructive way. On a badly imbalanced head F1 peaks in a low-precision regime, because recall climbs faster than precision falls: blackberry tuned itself to 0.18, where 96% of its calls were false, and sweet fired on 1317 of 8855 molecules at 18% precision. A high AUROC gives no protection here — AUROC is computed on ranking and is insensitive to class imbalance, precision is not. coffee has AUROC 0.960 and out-of-fold precision 0.46; both are true of the same head. So the threshold must now clear a 50% precision floor: a call labelled confident has to be right more often than not. Thresholds moved hard in BOTH directions — pine UP to 0.75 where it is 100% precise, rosemary down to 0.16 — and no head sits at either bound, so the data is choosing rather than the clamp. 73 of 167 aroma heads cannot reach the floor at any threshold. They are marked INDICATIVE, not deleted: they keep their score, their place in the 178-dim profile, their chips, their map colour and their molecules, because firing well above base rate is real evidence. What changes is that they are never dressed up as a confident call — the read returns `indicative` plus the head's measured precision, and the UI hatches their bar. Hiding them would have deleted reach from 73 notes to paper over a labelling problem. All five mouthfeel heads clear the floor (0.57-1.00). Applied everywhere a threshold was hard-coded, not just the live read: the precomputed profile index, the flavor map and the export card each had their own 0.5, and leaving those would have made a molecule's chips disagree with its own modal. The UI's likely/possible/weak levels are now read against each head's own threshold too, for the same reason. audit_generalization.py scores at the calibrated thresholds and grew a fourth verdict: PRECISION-LIMITED, for a head that is strict on purpose. Calling pine "under-confident" at 0.75 would be exactly backwards — loosening it trades away the accuracy it was tuned for. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>

…263) #261 gave every head its own decision threshold and a measured precision, but only a molecule read exposed them. The head catalog — the thing you consult to ask "how good is this head?" — still showed AUROC alone, which is precisely the number that hid the problem: AUROC is computed on ranking and is blind to class imbalance, so `ginger` reads 0.979 there while being right 10% of the time when it fires. /api/heads now carries threshold, precision, recall, n_pos and confident_capable per head, for aroma and mouthfeel alike. A head that fires at 0.16 and is right 6% of the time says so on its own row, not only buried inside one molecule's read. Also fixes headBar, which still hard-coded 0.5 to decide whether a bar was lit. That silently disagreed with the calibrated mouthfeel heads: `cooling` fires at 0.64 and `astringent` at 0.71, so a score of 0.55 was being drawn as firing when the head says it is not. Bars now read the head's own threshold, show where that bar sits and how precise it is there, and hatch the fill for indicative heads. Tox heads keep the flat 0.5 — they are not calibrated and, with hundreds of positives each, were never shy. Taste heads are published with an explicit threshold of 0.5 rather than a null, so a consumer never has to guess whether the field is missing or genuinely flat. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>

…269) * feat: calibrate every head, unify the chip language, fix the modal layout Closes #265, #266, #267, #268. CALIBRATION EVERYWHERE (#265). Taste and Tox21 still ran on a flat 0.5 with only AUROC shown — the exact reporting gap #261 closed for aroma and mouthfeel. Both now use the same out-of-fold calibration with a 50% precision floor, and /api/heads publishes threshold, precision, recall and confident_capable for all four modalities. The taste heads were mildly shy, as expected with hundreds of positives each (sweet 0.33, umami 0.33, bitter 0.51), and all six clear the floor at 0.50-0.87. The Tox21 result is the one that matters: every one of the twelve calibrated UPWARD — NR-AR to 0.69, NR-AR-LBD 0.65, NR-ER 0.63 — which means a flat 0.5 had them OVER-flagging. That is the opposite direction from the thin aroma heads and the more dangerous one, because a caution flag that cries wolf teaches people to ignore the flags that matter. CHIP LANGUAGE (#266). Flavor, note, taste and mouthfeel chips had each been given a different border STYLE at a different time — dashed, dotted, thicker, plus per-taste text recolouring — so they read as four accidents rather than one system. They now share a pill, a size and a border weight, and carry identity in a leading dot in the family's own hue. Each studio section also gained a plain-language paragraph on what that dimension IS and why it matters: a flavor is the whole percept, a note is what the nose reads, taste is the basics the tongue reads, mouthfeel is touch reported by the trigeminal nerve. Menthol is cold without a temperature change; that distinction is load-bearing and nothing in the UI had ever said it. ALL MODALITIES ON CARDS (#267). Substitute and neighbor cards showed taste and aroma but not mouthfeel. Read straight off the profile matrix, whose mouthfeel columns were already there, so it costs no extra inference — and each sensation must clear its own calibrated threshold, so a card never shows something the modal would call indicative. MODAL LAYOUT (#268). Two real bugs, both measured rather than guessed: - Mobile: the swap grid used `1fr`, which is minmax(auto,1fr), and `auto` will not shrink below min-content — so the substitutes and neighbors cards rendered 468px wide inside a 358px modal. minmax(0,1fr) fixes it; .modal-top already had this and .swap-grid did not. - Desktop: #results is a flex column whose 14px gap separates the cards, but the render path set an inline display:block on it, and block boxes ignore `gap`. The Heads card had been sitting flush against the Substitutes card below it. Clearing the inline style lets the stylesheet win. Verified with a Playwright probe at 390px and 1440px: no card overflows its container at either size, and no two cards are within 8px of each other on desktop. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com> * docs: define AUROC, threshold and precision where people actually land HOW-IT-WORKS.md used CV-AUROC eleven times and never said what it is beyond a parenthetical, and never mentioned threshold, precision or recall at all — the real explanation lived only in ACCURACY.md, one click further than most readers go. A six-term glossary now sits at section 0.5, before any number is used: head, AUROC (0.5 is a coin flip, 1.0 is never wrong), threshold (not a flat 0.5 — 0.16 to 0.85, fitted per head), precision (when it says yes, how often is it right), recall, out-of-fold, and confident vs indicative. Two sentences each. The same glossary, compressed to a definition list, is now a card in the site's own How-it-works grid, so it reaches someone who will never open the docs folder. Also tightens the taste wording, which was loose in both places: there are FIVE basic tastes and SIX taste heads. `tasteless` is a head that predicts the documented absence of taste — genuinely useful, since it is how you find a clean carrier — but it is not a sixth basic taste, and calling it one is the kind of small inaccuracy a flavorist would notice immediately. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com> * docs(arch): pin the frontend UI kit and record what we deliberately excluded Three stack questions kept resurfacing, so they are now answered in the one place someone would look, with the reasoning rather than just the verdict. Frontend is pinned to React 19 + Vite + TypeScript with Tailwind and shadcn/ui, explicitly NOT MUI. shadcn is copy-in rather than import, so the components land in our own source tree where they can be read and owned — which for a portfolio is the difference between showing you used a component library and showing you can build one. Flavormancer also already has a visual identity (the purple/teal gradient, the gilded scroll rods, the flask loader) that Material would flatten, and the port would be spent overriding it. The trade is stated honestly: MUI wins when you need an enterprise data-grid and date pickers tomorrow. This app is cards, chips, charts and a modal. Node.js is scoped to build tooling and an optional TypeScript MCP server beside the Python one — never a third backend. Two API stacks is breadth; three is sprawl, and it reads as indecision. Laravel is deliberately absent and the reasoning is written down so nobody re-opens it: it is a good fit for CRUD-and-content products and is used heavily elsewhere in this portfolio, but Flavormancer is on-prem scientific computing. Adding a familiar framework that proves nothing new would blur that. Choosing against your most comfortable stack when it does not fit is the point. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com> --------- Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>

…reshold (#277) * fix(map): the legend counted the wrong thing, and members used the wrong threshold Two bugs behind the same confusing symptom — heads reading "(0)" in the map legend while clicking them lights up plenty of molecules. THE LEGEND COUNTED DOMINANCE, NOT MEMBERSHIP. Each map point gets ONE colour, from whichever head wins a winner-take-all tie-break, and the legend counted those wins. So `raspberry` reads 0 while firing on 19 molecules; `pine` reads 6 while firing on 17. Both numbers are true, they answer different questions, and only one of them is what a person means by "how many raspberry molecules are there". A tooltip explained the distinction, which is no help — nobody hovers a number that already looks like an answer. The legend now shows MEMBERSHIP, from a new /api/map_counts that returns per-head fire counts for every dimension in one call. Dominance is still what colours the dot, and the tooltip now gives both figures ("19 molecules carry raspberry. 0 of them show it as their DOMINANT note"). MAP MEMBERS USED A FLAT 0.5. Since #261 every head fires at its own calibrated threshold, but api_map_members had kept a hardcoded 0.5 — so a head like `quince` (fires at 0.24) had most of its molecules hidden from the map while the molecule modal happily called them matches. Two surfaces disagreeing about the same head. It now defaults to the head's own threshold, with the explicit `threshold` query parameter still available for callers that want a different cut. Neither bug would have raised an error; both just quietly showed a smaller, wronger number. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com> * feat(map): show both counts with a toggle, and overlay safety instead of colouring by it BOTH COUNTS, SWITCHABLE. The legend number was ambiguous because two true things were competing for it: how many molecules a note COLOURS (it wins the single dominant label) and how many molecules CARRY it (the head fires). raspberry reads 0 and 19 respectively. Rather than pick one and bury the other in a tooltip nobody hovers, both are available from a carries/dominant toggle, and a key line under the map names which is showing — in words, not a hover. Membership is the default because that is how the number is read, and because it is the question a formulator has. Dominance stays what colours the dot, which is why it is still offered. SAFETY AS AN OVERLAY. A colour-by-tox mode was the obvious thing to build and would have been wrong: colouring a molecule by its worst assay asserts the molecule IS that hazard, when the true statement is "this is a flavour molecule that also warrants review". So the ⚠ toggle RINGS flagged molecules on top of whatever flavour colouring is active — ring weight and warmth scale with the number of flags, so severity reads without a second legend. You keep the flavour map you were reading and see hazard laid over it, which is the actual question: where are the risky ones in the space I am working in? It respects an active spotlight and uses the same projection as the dots, so it lines up in 2D, 3D and the property-axes view. Tox flag counts ride along in /api/map, read from master_enrichment's precomputed columns — no inference at request time. Nothing was removed. taste / aroma / both / mouthfeel colour modes all remain (mouthfeel already existed and is unchanged), highlight-by-head still works, and the tooltip still explains both numbers for anyone who does hover. Deliberately NOT built: an "all heads" colour mode. 195 heads is 195 colours; past roughly twenty the eye cannot separate them and the legend becomes a wall. The perceptual families are the right granularity for colour, and individual heads are already served by highlight. Verified in a real browser: all controls present, no page errors, all four colour modes intact. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com> --------- Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>

rvnminers-A-and-N added area:aroma OpenPOM aroma model + sidecar area:training Python dataset build + model training area:ui React workbench frontend enhancement New feature or request labels Jul 29, 2026

rvnminers-A-and-N merged commit 40caeb7 into main Jul 29, 2026
4 checks passed

rvnminers-A-and-N deleted the feat/per-head-calibrated-thresholds branch July 29, 2026 15:36

This was referenced Jul 29, 2026

Build the 73 indicative aroma heads up to confident (precision, not AUROC, is the bar) #262

Open

research(model): GNN for structure→property vs the RandomForest heads #199

Open

Publish each head's calibration in the catalog and on its bar #263

Merged

This was referenced Jul 30, 2026

model(taste,safety): calibrate the taste and Tox21 heads like the aroma heads #265

Closed

Calibrate every head, unify the chip language, fix the modal layout #269

Merged

This was referenced Jul 30, 2026

Build up the food side of sweet/ethereal/pungent rather than cutting the industrial odorants out #257

Closed

The map legend counted the wrong thing, and members used the wrong threshold #277

Merged

Navigation Menu

Uh oh!

Per-head calibrated thresholds, with a precision floor - #261

Per-head calibrated thresholds, with a precision floor#261
rvnminers-A-and-N merged 1 commit into
mainfrom
feat/per-head-calibrated-thresholds

rvnminers-A-and-N commented Jul 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

rvnminers-A-and-N commented Jul 29, 2026

The first attempt was wrong, instructively

So: a 50% precision floor

73 heads can't reach the floor. They're marked, not removed

Applied everywhere, not just the live read

Audit tool

Bug caught in passing

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant