From b3ce46f3f4763a9de8af523b9ce7d487b5eb9b9f Mon Sep 17 00:00:00 2001
From: "Austin L." <86896075+rvnminers-A-and-N@users.noreply.github.com>
Date: Fri, 26 Jun 2026 15:44:07 +0000
Subject: [PATCH] docs(data): add data-sources acquisition tracker

Record what each taste/aroma/safety source unlocks, its license, status, and how
to acquire it (ChemTastesDB in use; cosylab, FlavorDB, UMP442, SweetenersDB,
Pyrfume/Leffingwell, FEMA/FDA to get). Data files stay out of the repo; this
tracks provenance + the enrichment roadmap.
---
 docs/DATA-SOURCES.md | 32 ++++++++++++++++++++++++++++++++
 docs/README.md       |  1 +
 2 files changed, 33 insertions(+)
 create mode 100644 docs/DATA-SOURCES.md

diff --git a/docs/DATA-SOURCES.md b/docs/DATA-SOURCES.md
new file mode 100644
index 0000000..09e3f92
--- /dev/null
+++ b/docs/DATA-SOURCES.md
@@ -0,0 +1,32 @@
+# Data Sources — acquisition tracker
+
+What feeds the models, what each source unlocks, its license, and how to get it.
+
+> **The data files themselves are never committed** (licensing + size — see
+> `.gitignore`). The training scripts download them; this file records provenance,
+> status, and the acquisition steps. Loaders **skip cleanly** when a file is absent,
+> so adding a source is just dropping its file in and re-running the pipeline.
+
+## Status
+
+| Source | Unlocks | License | Status | How to get |
+|--------|---------|---------|--------|-----------|
+| **ChemTastesDB v2.0** | sweet/bitter/umami training + sour/salty rule data (primary) | CC-BY-4.0 | ✅ in use | Zenodo record `14963136`, direct download (see `training/SETUP.md`). Column mapping verified. |
+| **cosylab/bittersweet** | more sweet/bitter volume | AGPL-3.0 | ⬜ to get | `git clone github.com/cosylabiiit/bittersweet` → `bittersweet/data/*.tsv`. AGPL: keep a CC-BY-clean build with `INCLUDE_COSYLAB=False`, or opt in deliberately. |
+| **FlavorDB** | ~25k molecules taste+odor + natural-source mapping | confirm terms | ⬜ to get | REST/JSON API at `cosylab.iiitd.edu.in/flavordb` → `flavordb_taste.csv`. Map taste fields; **[VERIFY] columns** before trusting. |
+| **UMP442 / BIOPEP-UWM** | more umami examples (only 283 now) | confirm terms | ⬜ to get | BIOPEP-UWM umami DB (form-driven) → `umami_list.csv`. |
+| **SweetenersDB (Chéron 2017)** | the sweetness-**intensity** regressor (currently skipped) | publisher supp. | ⬜ to get | paper supplementary (~316 compounds, relative-to-sucrose) → `sweeteners_db.csv`. Likely **paywalled → manual grab**. |
+| **Pyrfume / Leffingwell** | the **aroma model** (OpenPOM) — issues #17 / #18 | per-set; confirm | ⬜ to get | `git clone github.com/pyrfume/pyrfume-data`; Leffingwell odor set. Separate, version-fragile effort. |
+| **FEMA GRAS / FDA SAF** | GRAS cross-reference + dosing/OAV lookups | gov public / FEMA | ⬜ to get | FDA "Substances Added to Food" (public domain) → `gras_reference.parquet`; FEMA use-level PDFs → `properties.parquet`. **Verify every scraped dosing number.** |
+
+## Notes
+
+- **Licensing.** ChemTastesDB (CC-BY-4.0) is the clean base. cosylab is AGPL — gated
+  behind the `INCLUDE_COSYLAB` flag. Get an IP/OSS-license review before any
+  commercial release; full attribution lives in [`SOURCES.md`](SOURCES.md).
+- **Column verification.** Each new source's column names must be checked against the
+  loaders (the `[VERIFY]` markers) — see issue **#25**. ChemTastesDB's mapping is
+  resolved (both the coarse `Class taste` and the granular `Taste` columns are parsed).
+- **Suggested priority.** (1) `Taste`-column mining — **done** ✅ → (2) cosylab /
+  FlavorDB for taste volume → (3) SweetenersDB for the intensity head → (4)
+  Pyrfume / Leffingwell for the aroma model.
diff --git a/docs/README.md b/docs/README.md
index a437136..c416268 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -7,5 +7,6 @@ Architecture, capability catalogue, and design decisions.
   honest limits of public-data prediction
 - `SOURCES.md` — data sources, libraries, research, and license attribution
 - `API-CONTRACT.md` — the fixed JSON contract the .NET API exposes and the React UI consumes
+- `DATA-SOURCES.md` — data acquisition tracker: what each source unlocks and how to get it
 
 These document *what we're building and why*, so decisions don't get re-argued.