From 78a87372bae31792a964555afcef7bdd9c30ef14 Mon Sep 17 00:00:00 2001 From: "Austin L." <86896075+rvnminers-A-and-N@users.noreply.github.com> Date: Fri, 26 Jun 2026 17:10:34 +0000 Subject: [PATCH] =?UTF-8?q?feat(data):=20add=20SweetenersDB=20=E2=86=92=20?= =?UTF-8?q?sweetness-intensity=20regressor?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Map SweetenersDB v2.0's 'logSw' column (MIT, github.com/chemosim-lab/SweetenersDB, 316 compounds) so build_intensity finds it. Trains the relative-to-sucrose sweetness-intensity head that was skipped (R2=0.82, indicative on small data) and exports it to ONNX alongside the taste classifiers. Tracker updated. Closes #36. --- docs/DATA-SOURCES.md | 9 +++++---- training/build_taste_dataset.py | 4 +++- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/DATA-SOURCES.md b/docs/DATA-SOURCES.md index 09e3f92..bc3d4c3 100644 --- a/docs/DATA-SOURCES.md +++ b/docs/DATA-SOURCES.md @@ -15,7 +15,7 @@ What feeds the models, what each source unlocks, its license, and how to get it. | **cosylab/bittersweet** | more sweet/bitter volume | AGPL-3.0 | ⬜ to get | `git clone github.com/cosylabiiit/bittersweet` → `bittersweet/data/*.tsv`. AGPL: keep a CC-BY-clean build with `INCLUDE_COSYLAB=False`, or opt in deliberately. | | **FlavorDB** | ~25k molecules taste+odor + natural-source mapping | confirm terms | ⬜ to get | REST/JSON API at `cosylab.iiitd.edu.in/flavordb` → `flavordb_taste.csv`. Map taste fields; **[VERIFY] columns** before trusting. | | **UMP442 / BIOPEP-UWM** | more umami examples (only 283 now) | confirm terms | ⬜ to get | BIOPEP-UWM umami DB (form-driven) → `umami_list.csv`. | -| **SweetenersDB (Chéron 2017)** | the sweetness-**intensity** regressor (currently skipped) | publisher supp. | ⬜ to get | paper supplementary (~316 compounds, relative-to-sucrose) → `sweeteners_db.csv`. Likely **paywalled → manual grab**. | +| **SweetenersDB v2.0** | the sweetness-**intensity** regressor | **MIT** | ✅ in use | direct CSV from `github.com/chemosim-lab/SweetenersDB` (`SweetenersDB_v2.0.csv`, 316 cmpds, `logSw` column). R²≈0.82. | | **Pyrfume / Leffingwell** | the **aroma model** (OpenPOM) — issues #17 / #18 | per-set; confirm | ⬜ to get | `git clone github.com/pyrfume/pyrfume-data`; Leffingwell odor set. Separate, version-fragile effort. | | **FEMA GRAS / FDA SAF** | GRAS cross-reference + dosing/OAV lookups | gov public / FEMA | ⬜ to get | FDA "Substances Added to Food" (public domain) → `gras_reference.parquet`; FEMA use-level PDFs → `properties.parquet`. **Verify every scraped dosing number.** | @@ -27,6 +27,7 @@ What feeds the models, what each source unlocks, its license, and how to get it. - **Column verification.** Each new source's column names must be checked against the loaders (the `[VERIFY]` markers) — see issue **#25**. ChemTastesDB's mapping is resolved (both the coarse `Class taste` and the granular `Taste` columns are parsed). -- **Suggested priority.** (1) `Taste`-column mining — **done** ✅ → (2) cosylab / - FlavorDB for taste volume → (3) SweetenersDB for the intensity head → (4) - Pyrfume / Leffingwell for the aroma model. +- **Suggested priority.** (1) `Taste`-column mining — **done** ✅ → (2) SweetenersDB + intensity head — **done** ✅ → (3) Pyrfume / Leffingwell for the aroma model → + (4) cosylab for taste volume (AGPL — pending decision). **FlavorDB is + CC BY-NC-SA (NonCommercial) — incompatible with a commercial product; skip.** diff --git a/training/build_taste_dataset.py b/training/build_taste_dataset.py index 61d77e4..abf86e3 100644 --- a/training/build_taste_dataset.py +++ b/training/build_taste_dataset.py @@ -248,7 +248,9 @@ def build_intensity(path): return None df = pd.read_csv(path) sc = _find(df, ["smiles", "canonical smiles"]) - vc = _find(df, ["relative_sweetness", "log_sweetness", "sweetness", "rs"]) + # SweetenersDB v2.0 (MIT) names it 'logSw' (log sweetness vs sucrose, ~316 cmpds); + # also accept the generic names. [VERIFIED against SweetenersDB_v2.0.csv, 2026-06-26.] + vc = _find(df, ["log_sweetness", "logsw", "relative_sweetness", "sweetness", "rs"]) rows = [] for _, r in df.iterrows(): _, cs = canon(r[sc])