feat: chip legend, measured-property crawler, and the data-pipeline doc (#229, #26, #25) (#273)

CHIP LEGEND (#229). Four dimensions with four different meanings sit side by side in the studios,
and after the chip-system work in #269 the dot colour was the only thing distinguishing them —
which works only if you already know the code. A key now states it once, up front: what each
family means, and what solid / outline / hatched marking indicates (documented, predicted,
indicative). The hatched swatch matters most: it is the one that says "right less than half the
time — a hint, not an answer".

MEASURED PROPERTIES (#209, in progress). 73% of the enrichment table had no measured boiling
point, and the assumption was that the data did not exist. It does. build_properties.py reads
PubChem's property table, which only ever carries COMPUTED values; measured boiling and melting
points live in PUG-View under Experimental Properties, an endpoint this codebase had never called.
A sample of molecules with no BP found 8 of 8 had a PubChem record.

build_measured_properties.py crawls that endpoint. Parsing is the real work — values are free text
written by whoever took the measurement ("246 °C", "115-116 °C at 12 mm Hg", "410 °F") — so it
converts Fahrenheit, takes the midpoint of a range, and DISCARDS anything measured at reduced
pressure. A boiling point at 12 mmHg is not comparable to one at atmospheric, and silently mixing
them would corrupt the volatility ordering the formulation studio depends on. properties.parquet
already carries boiling_point_pressure_mmhg for exactly this reason.

Smoke test resolved 11 of 12 with values that check out (pyrrolidine 86.6 °C). The full crawl is
~7,200 molecules at roughly 57% hit rate and takes hours; it checkpoints every 100 so it can be
folded in whenever it finishes.

_by_skel() would have silently discarded all of it. It required an `inchikey` column, but both
backfill tables are keyed by `inchikey_skel` — so it returned an empty dict, with no error, and
every backfilled name and boiling point would simply never have appeared. It now accepts either.

DATA-PIPELINE.md (#26, #25). What every build script reads, the REAL column names in each table
read off the built artifacts rather than copied from the loader's hopes, and a clean-machine first
run in dependency order with timings and an end-to-end verification that checks a prediction
rather than just that the server started.

Two claims in the first draft were wrong and are corrected: the verification example printed
`.aroma.top[0]`, which is the highest-scoring CONFIDENT head and not the one you searched for; and
I had written that `n_sources` breaks label conflicts by weight of evidence. It does not — nothing
reads it, and conflicts resolve POSITIVE-WINS: one source calling a molecule sweet makes it sweet.
That is the right default for sparse partially-annotated sources, but it means a single mislabelled
source can assert a taste the others deny, which is worth knowing and is now written down.

Also resolves the [OBTAIN]/[VERIFY] markers: flavordb_taste.csv is permanently excluded
(NonCommercial), umami_list.csv is an unused nice-to-have, sweeteners_db.csv is present and drives
the intensity regressor.

Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>

Author: rvnminers-A-and-N

docs/DATA-PIPELINE.md

@@ -0,0 +1,161 @@
+# Data pipeline — sources, exact columns, and a clean-machine first run
+
+What every build script reads, the **real** column names in each table (not the ones the code
+hopes for), and the order to run things in from nothing. Companion to
+[`training/SETUP.md`](../training/SETUP.md) (install steps) and
+[`SOURCES.md`](SOURCES.md) (licensing and attribution).
+
+Written because the loaders carried `[VERIFY]` markers where nobody had opened the file and
+confirmed what was actually in it.
+
+---
+
+## 1. First run from a clean machine
+
+Assumes Python 3.12 and ~8 GB free. The whole chain is CPU-only; no GPU anywhere.
+
+```bash
+git clone https://github.com/echelonts/flavormancer.git && cd flavormancer
+python -m venv .venv && . .venv/bin/activate
+pip install -r requirements.txt
+pip install fastapi "uvicorn[standard]" pydantic umap-learn openpyxl   # serving + map + xlsx
+```
+
+Then, **in this order** — each step consumes the previous step's output:
+
+| # | command | produces | ~time |
+|---|---|---|---|
+| 1 | `python build_taste_dataset.py` | `taste_master.parquet` | 1 min |
+| 2 | `python train_taste.py` | `taste_models/` + manifest | 3 min |
+| 3 | `python build_odor_notes.py --pubchem-all` | `odor_notes.parquet` | **hours** (crawl) |
+| 4 | `python build_aroma_supplement.py` | `aroma_supplement.csv` | 10 min (crawl) |
+| 5 | `python build_aroma_dataset.py` | `aroma_train.parquet` | 2 min |
+| 6 | `python train_aroma.py` | `aroma_models/` + manifest | 10 min (32 cores) |
+| 7 | `python build_mouthfeel_supplement.py && python build_mouthfeel_dataset.py && python train_mouthfeel.py` | `mouthfeel_models/` | 5 min |
+| 8 | `python train_tox.py` | `tox_models/` | 5 min |
+| 9 | `python build_enrichment.py` | `master_enrichment.parquet` | 8 min |
+| 10 | `python build_profile_index.py` | `profile_index.npz` | 4 min |
+| 11 | `python build_flavor_map.py` | `flavor_map.parquet` | 6 min |
+| 12 | `uvicorn app:app --host 0.0.0.0 --port 8000` | the workbench | 50 s to warm |
+
+**Steps 3 and 4 are network crawls against PubChem.** They cache, so a re-run is cheap, but a
+cold step 3 is genuinely hours — start it and go and do something else. Everything downstream is
+local compute.
+
+**Verify it worked** (this is the honest end-to-end check, not just "the server started"):
+
+```bash
+curl -s localhost:8000/api/status                      # {"ready":true,"total":190,...}
+curl -s localhost:8000/api/heads | jq '.aroma | length' # 166
+# vanillin — the vanilla head should fire at 1.0, confident, with its calibrated threshold
+curl -s -X POST localhost:8000/api/predict \
+  -H 'Content-Type: application/json' \
+  -d '{"smiles":"COc1cc(C=O)ccc1O"}' \
+  | jq '.aroma.descriptors[] | select(.odor=="vanilla")'
+# {"odor":"vanilla","score":1.0,"threshold":0.42,"confident":true,"precision":1.0,"auroc":0.93,...}
+```
+
+`.aroma.top` is the *confident* subset ordered by score, so `top[0]` is whichever head scored
+highest — not necessarily the one you are looking for. Query `descriptors[]` by name, as above.
+
+If `/api/status` reports fewer than 190 heads, a training step was skipped — the app degrades
+gracefully rather than failing, so it will start regardless.
+
+---
+
+## 2. What each table actually contains
+
+Column names below were read off the built artifacts, not copied from the loader's expectations.
+Where a loader accepts several spellings it is noted, because the upstream files change.
+
+### Taste
+
+| table | rows | key columns |
+|---|---|---|
+| `taste_master.parquet` | 3,845 | `inchikey`, `smiles`, `multitaste`, `sweet`, `bitter`, `umami`, `sour`, `salty`, `n_sources` |
+| `taste_notes.parquet` | 676 | `inchikey`, `smiles`, `name`, `taste`, `taste_source` |
+
+`taste_master` is one row per molecule with a 0/1 column per basic taste; `multitaste` marks
+molecules ChemTastesDB records under more than one class. **There is no `tasteless` column** —
+that head trains from documented-tasteless text in `taste_notes`, which is why
+`build_flavor_map.py` overlays it separately rather than reading a column. `n_sources` counts the distinct upstream datasets a molecule appeared in. It is **provenance
+metadata only** — nothing downstream reads it today.
+
+Worth knowing how conflicts actually resolve, because it is not a vote: `build_taste_dataset`
+aggregates with **positive-wins** — if any source labels a molecule sweet, it is sweet; only if
+every source says 0 does it become 0. That maximises recall on sparse, partially-annotated
+sources, which is the right default here, but it does mean one mislabelled source can assert a
+taste the other four deny. `n_sources` is the column that would let a future weighted rule
+override that, which is why it is recorded even though it is unused.
+
+
+### Aroma
+
+| table | rows | key columns |
+|---|---|---|
+| `odor_notes.parquet` | 2,258 | `inchikey`, `smiles`, `name`, `odor`, `odor_source`, `odor_threshold_ppm`, `odor_threshold_note` |
+| `aroma_supplement.csv` | 2,159 | `flavor`, `molecule`, `smiles`, `category` |
+| `aroma_train.parquet` | 2,417 | `inchikey`, `smiles`, + one 0/1 column per descriptor |
+
+`odor` is **free text** as recorded ("sweet, floral, slightly minty"), normalised into the
+controlled vocabulary by `build_aroma_dataset.tag()`. `odor_threshold_ppm` is populated only where
+PubChem cites one — treat it as a lookup, never a prediction.
+
+### Properties
+
+`properties.parquet` — 9,154 rows: `inchikey`, `common_name`, `iupac_name`, `boiling_point_c`,
+`boiling_point_pressure_mmhg`, `vapor_pressure_pa`, `melting_point_c`.
+
+**`boiling_point_pressure_mmhg` is the column that makes the BP usable.** A boiling point measured
+at reduced pressure is not comparable to one at atmospheric, and averaging them together would
+corrupt any volatility ordering. `build_measured_properties.py` therefore *discards* reduced-pressure
+values outright rather than storing them without their pressure.
+
+Backfill tables, both keyed by **`inchikey_skel`** rather than `inchikey`:
+
+- `iupac_backfill.parquet` — `common_name`, `iupac_name` from PubChem **Title + IUPACName**
+- `measured_properties.parquet` — `boiling_point_c`, `melting_point_c` from PUG-View
+
+> That key difference is a real trap. `build_enrichment._by_skel()` accepts either column for
+> exactly this reason — an `inchikey`-only reader returns an empty dict for these two, silently,
+> and every backfilled value simply never appears.
+
+### Food-use registers
+
+| table | rows | key columns |
+|---|---|---|
+| `saf_universe.parquet` | 2,782 | `smiles`, `inchikey`, `name`, `cas` (FDA Substances Added to Food) |
+| `gb_union_list.parquet` | 2,164 | `smiles`, `inchikey`, `name`, `fl` (EU/GB flavourings, FL number) |
+| `gras_reference.parquet` | 2,781 | `inchikey` |
+
+These carry **listing status only**. A row here means a regulator lists the substance — it is not
+a safety clearance, and the UI must never render it as one. See
+[`food-safety provenance`](SOURCES.md).
+
+### Master table
+
+`master_enrichment.parquet` — 8,861 rows, the single table the app reads for everything that is
+not a live prediction: `inchikey_skel`, `smiles`, `name`, `mw`, `logp`, `tpsa`, `hbd`, `hba`,
+`rot_bonds`, `rings`, `melting_point_c`, `boiling_point_c`, `gras`, `taste_documented`, plus a
+`tox_*` column per Tox21 assay.
+
+`name` is **never null**: it falls back to the molecular formula, with multi-component structures
+labelled as mixtures (`CaI2O6 (mixture: Ca+2 + IO3- + IO3-)`). A raw SMILES in the grid looks
+broken; a formula is honest.
+
+---
+
+## 3. Unresolved source markers
+
+`build_taste_dataset.py` carries `[OBTAIN]` markers for three sources. Their status, so nobody
+re-investigates:
+
+| source | status |
+|---|---|
+| `flavordb_taste.csv` | **Excluded, permanently.** FlavorDB is NonCommercial — incompatible with this project's commercial-clean requirement. The loader tolerates its absence. |
+| `umami_list.csv` | Optional. UMP442/BIOPEP-UWM umami SMILES. Not currently used; the umami head trains from ChemTastesDB alone and reaches AUROC 0.990, so this is a nice-to-have. |
+| `sweeteners_db.csv` | **Present and used.** Cheron SweetenersDB (MIT) drives the sweetness-intensity regressor (R² 0.836, n=316 — small, treat as indicative). |
+
+The `[VERIFY]` marker at `build_taste_dataset.py:72` is a defensive `KeyError` for when an upstream
+file renames a column. It is working as intended: the loader tries several known spellings and
+fails loudly with the actual column list rather than silently producing an empty dataset.

docs/README.md

@@ -8,6 +8,8 @@ Architecture, capability catalogue, and design decisions.
 - `ACCURACY.md` — **how accurate is it, really?** AUROC vs precision, per-head thresholds,
   and what every reported number does and doesn't mean. Written to be read without an ML background
 - `METHODS.md` — every rule, threshold, heuristic, and trick the system uses, with rationale and ceiling
+- `DATA-PIPELINE.md` — what every build script reads, the **real** column names in each table,
+  and a clean-machine first run in order
 - `SOURCES.md` — data sources, libraries, research, and license attribution
 - `API-CONTRACT.md` — the fixed JSON contract the .NET API exposes and the React UI consumes
 - `DATA-SOURCES.md` — data acquisition tracker: what each source unlocks and how to get it

training/build_enrichment.py

@@ -47,17 +47,24 @@ def _all_structures():
 
 
 def _by_skel(path, cols):
-    """{skeleton -> {col: val}} from a parquet keyed by full inchikey."""
+    """{skeleton -> {col: val}} from a parquet keyed by either `inchikey` or `inchikey_skel`.
+
+    Accepting BOTH keys matters: properties.parquet is keyed by the full InChIKey, while the
+    backfill tables written by build_iupac_backfill.py and build_measured_properties.py are keyed
+    by the skeleton. An inchikey-only reader returns an empty dict for those two — silently, with
+    no error — so the backfilled names and measured boiling points would simply never appear.
+    """
     out = {}
     try:
         d = pd.read_parquet(path)
     except Exception:  # noqa: BLE001 — table absent; return what we have
         return out
-    if "inchikey" not in d.columns:
+    key = next((k for k in ("inchikey", "inchikey_skel") if k in d.columns), None)
+    if key is None:
         return out
     have = [c for c in cols if c in d.columns]
     for _, r in d.iterrows():
-        ik = r["inchikey"]
+        ik = r[key]
         if isinstance(ik, str):
             out[ik.split("-")[0]] = {c: r[c] for c in have}
     return out
@@ -237,6 +244,21 @@ def _taste_by_skel():
                     structs.setdefault(Chem.MolToInchiKey(_m).split("-")[0], Chem.MolToSmiles(_m))
     props = _by_skel("properties.parquet",
                      ["common_name", "iupac_name", "melting_point_c", "boiling_point_c"])
+    # Names PubChem's property table missed (build_iupac_backfill.py: Title AND IUPACName).
+    for _skel, _row in _by_skel("iupac_backfill.parquet", ["common_name", "iupac_name"]).items():
+        tgt = props.setdefault(_skel, {})
+        for _k in ("common_name", "iupac_name"):
+            if not isinstance(tgt.get(_k), str) and isinstance(_row.get(_k), str):
+                tgt[_k] = _row[_k]
+    # MEASURED boiling/melting points from PUG-View (build_measured_properties.py). The property
+    # table only ever carried COMPUTED values, which is why 73% of rows had no measured BP despite
+    # every one of those molecules having a PubChem record — we had never asked the right endpoint.
+    for _skel, _row in _by_skel("measured_properties.parquet",
+                                ["melting_point_c", "boiling_point_c"]).items():
+        tgt = props.setdefault(_skel, {})
+        for _k in ("melting_point_c", "boiling_point_c"):
+            if tgt.get(_k) is None and _row.get(_k) is not None:
+                tgt[_k] = _row[_k]
     taste_doc = _taste_by_skel()
     curated = _curated_names()   # human names for the molecules we hand-curated
     print(f"{len(structs)} molecules; {len(props)} with crawl properties; "

training/build_measured_properties.py

@@ -0,0 +1,145 @@
+"""build_measured_properties.py — crawl EXPERIMENTAL boiling/melting points from PubChem PUG-View.
+
+`build_properties.py` uses PubChem's property table, which only carries COMPUTED values (MW, logP,
+TPSA). Measured boiling and melting points live somewhere else entirely: in PUG-View, under the
+Experimental Properties section, sourced from HSDB and other public-domain records. That is why
+73% of the enrichment table had no measured BP despite every one of those molecules having a
+PubChem record — we had never asked the endpoint that holds them.
+
+Values arrive as free text written by whoever recorded the measurement, so parsing is the real
+work here: "246 °C", "115-116 °C at 12 mm Hg", "-11.5 °C", "410 °F". We take the first plausible
+Celsius figure, convert Fahrenheit, take the midpoint of a range, and DROP anything reported at
+reduced pressure — a boiling point at 12 mmHg is not comparable to one at atmospheric and
+silently mixing them would corrupt the volatility ordering the formulation studio relies on.
+
+Public-domain PubChem/HSDB data, cached locally.
+
+    python build_measured_properties.py            # -> measured_properties.parquet
+    python build_measured_properties.py --limit 50 # smoke test
+"""
+import argparse
+import contextlib
+import json
+import re
+import time
+import urllib.parse
+import urllib.request
+
+import pandas as pd
+from rdkit import Chem
+
+ENRICH = "master_enrichment.parquet"
+OUT = "measured_properties.parquet"
+PAUSE = 0.25          # PUG-View is heavier than the property table; stay well under the guidance
+TIMEOUT = 10
+
+# "115-116 °C at 12 mm Hg" — reduced-pressure boiling points are not comparable to atmospheric
+# ones and must not be mixed into the same column.
+_REDUCED = re.compile(r"\bat\s+[\d.]+\s*(mm\s*hg|torr|kpa|mbar|hpa)\b", re.IGNORECASE)
+_TEMP = re.compile(r"(-?\d+(?:\.\d+)?)\s*(?:-\s*(-?\d+(?:\.\d+)?)\s*)?°?\s*([CF])\b")
+
+
+def _parse_temp_c(text):
+    """First plausible temperature in the string, as Celsius. None if unusable."""
+    if not text or _REDUCED.search(text):
+        return None
+    m = _TEMP.search(text)
+    if not m:
+        return None
+    lo = float(m.group(1))
+    val = (lo + float(m.group(2))) / 2 if m.group(2) else lo   # midpoint of a reported range
+    if m.group(3).upper() == "F":
+        val = (val - 32) * 5 / 9
+    return round(val, 1) if -200 <= val <= 800 else None       # outside this, it isn't a BP or MP
+
+
+def _get(url):
+    try:
+        with urllib.request.urlopen(url, timeout=TIMEOUT) as r:
+            return json.load(r)
+    except Exception:  # noqa: BLE001 — not found / throttled / timeout
+        return None
+
+
+def _cid(smiles):
+    d = _get("https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/smiles/"
+             f"{urllib.parse.quote(smiles)}/cids/JSON")
+    with contextlib.suppress(Exception):
+        return d["IdentifierList"]["CID"][0]
+    return None
+
+
+def _experimental(cid, heading):
+    """All free-text values PubChem lists under one Experimental Properties heading."""
+    d = _get(f"https://pubchem.ncbi.nlm.nih.gov/rest/pug_view/data/compound/{cid}/JSON"
+             f"?heading={urllib.parse.quote(heading)}")
+    if not d:
+        return []
+    out, stack = [], [d]
+    while stack:                      # the response nests sections arbitrarily deep
+        node = stack.pop()
+        if isinstance(node, dict):
+            for info in node.get("Information", []) or []:
+                for sw in (info.get("Value", {}) or {}).get("StringWithMarkup", []) or []:
+                    if sw.get("String"):
+                        out.append(sw["String"])
+            stack.extend(v for v in node.values() if isinstance(v, (dict, list)))
+        elif isinstance(node, list):
+            stack.extend(node)
+    return out
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    ap.add_argument("--limit", type=int, default=0, help="only crawl N molecules (smoke test)")
+    a = ap.parse_args()
+
+    enrich = pd.read_parquet(ENRICH)
+    done = set()
+    with contextlib.suppress(Exception):
+        done = set(pd.read_parquet(OUT)["inchikey_skel"])
+
+    todo = {}
+    for _, r in enrich.iterrows():
+        skel, smi = r.get("inchikey_skel"), r.get("smiles")
+        if not isinstance(skel, str) or not isinstance(smi, str) or skel in done:
+            continue
+        # only chase molecules that are actually missing a measured value
+        if pd.notna(r.get("boiling_point_c")) and pd.notna(r.get("melting_point_c")):
+            continue
+        todo.setdefault(skel, smi)
+    if a.limit:
+        todo = dict(list(todo.items())[:a.limit])
+
+    print(f"{len(done)} already crawled; {len(todo)} to go.", flush=True)
+    rows, got = [], 0
+    for i, (skel, smi) in enumerate(todo.items(), 1):
+        mol = Chem.MolFromSmiles(smi)
+        cid = _cid(Chem.MolToSmiles(mol)) if mol else None
+        if cid:
+            bp = next((v for v in (_parse_temp_c(t) for t in _experimental(cid, "Boiling Point"))
+                       if v is not None), None)
+            mp = next((v for v in (_parse_temp_c(t) for t in _experimental(cid, "Melting Point"))
+                       if v is not None), None)
+            if bp is not None or mp is not None:
+                rows.append({"inchikey_skel": skel, "boiling_point_c": bp, "melting_point_c": mp})
+                got += 1
+        if i % 100 == 0:
+            print(f"  {i}/{len(todo)} crawled, {got} with a measured value", flush=True)
+            _save(rows)
+        time.sleep(PAUSE)
+    _save(rows)
+    print(f"Done. {got}/{len(todo)} gained a measured property -> {OUT}")
+
+
+def _save(rows):
+    if not rows:
+        return
+    new = pd.DataFrame(rows).drop_duplicates("inchikey_skel")
+    with contextlib.suppress(Exception):
+        new = pd.concat([pd.read_parquet(OUT), new]).drop_duplicates("inchikey_skel", keep="last")
+    new.to_parquet(OUT)
+
+
+if __name__ == "__main__":
+    main()