|
| 1 | +# Data Sources β acquisition tracker |
| 2 | + |
| 3 | +What feeds the models, what each source unlocks, its license, and how to get it. |
| 4 | + |
| 5 | +> **The data files themselves are never committed** (licensing + size β see |
| 6 | +> `.gitignore`). The training scripts download them; this file records provenance, |
| 7 | +> status, and the acquisition steps. Loaders **skip cleanly** when a file is absent, |
| 8 | +> so adding a source is just dropping its file in and re-running the pipeline. |
| 9 | +
|
| 10 | +## Status |
| 11 | + |
| 12 | +| Source | Unlocks | License | Status | How to get | |
| 13 | +|--------|---------|---------|--------|-----------| |
| 14 | +| **ChemTastesDB v2.0** | sweet/bitter/umami training + sour/salty rule data (primary) | CC-BY-4.0 | β
in use | Zenodo record `14963136`, direct download (see `training/SETUP.md`). Column mapping verified. | |
| 15 | +| **cosylab/bittersweet** | more sweet/bitter volume | AGPL-3.0 | β¬ to get | `git clone github.com/cosylabiiit/bittersweet` β `bittersweet/data/*.tsv`. AGPL: keep a CC-BY-clean build with `INCLUDE_COSYLAB=False`, or opt in deliberately. | |
| 16 | +| **FlavorDB** | ~25k molecules taste+odor + natural-source mapping | confirm terms | β¬ to get | REST/JSON API at `cosylab.iiitd.edu.in/flavordb` β `flavordb_taste.csv`. Map taste fields; **[VERIFY] columns** before trusting. | |
| 17 | +| **UMP442 / BIOPEP-UWM** | more umami examples (only 283 now) | confirm terms | β¬ to get | BIOPEP-UWM umami DB (form-driven) β `umami_list.csv`. | |
| 18 | +| **SweetenersDB (ChΓ©ron 2017)** | the sweetness-**intensity** regressor (currently skipped) | publisher supp. | β¬ to get | paper supplementary (~316 compounds, relative-to-sucrose) β `sweeteners_db.csv`. Likely **paywalled β manual grab**. | |
| 19 | +| **Pyrfume / Leffingwell** | the **aroma model** (OpenPOM) β issues #17 / #18 | per-set; confirm | β¬ to get | `git clone github.com/pyrfume/pyrfume-data`; Leffingwell odor set. Separate, version-fragile effort. | |
| 20 | +| **FEMA GRAS / FDA SAF** | GRAS cross-reference + dosing/OAV lookups | gov public / FEMA | β¬ to get | FDA "Substances Added to Food" (public domain) β `gras_reference.parquet`; FEMA use-level PDFs β `properties.parquet`. **Verify every scraped dosing number.** | |
| 21 | + |
| 22 | +## Notes |
| 23 | + |
| 24 | +- **Licensing.** ChemTastesDB (CC-BY-4.0) is the clean base. cosylab is AGPL β gated |
| 25 | + behind the `INCLUDE_COSYLAB` flag. Get an IP/OSS-license review before any |
| 26 | + commercial release; full attribution lives in [`SOURCES.md`](SOURCES.md). |
| 27 | +- **Column verification.** Each new source's column names must be checked against the |
| 28 | + loaders (the `[VERIFY]` markers) β see issue **#25**. ChemTastesDB's mapping is |
| 29 | + resolved (both the coarse `Class taste` and the granular `Taste` columns are parsed). |
| 30 | +- **Suggested priority.** (1) `Taste`-column mining β **done** β
β (2) cosylab / |
| 31 | + FlavorDB for taste volume β (3) SweetenersDB for the intensity head β (4) |
| 32 | + Pyrfume / Leffingwell for the aroma model. |
0 commit comments