Skip to content

Commit b3ce46f

Browse files
docs(data): add data-sources acquisition tracker
Record what each taste/aroma/safety source unlocks, its license, status, and how to acquire it (ChemTastesDB in use; cosylab, FlavorDB, UMP442, SweetenersDB, Pyrfume/Leffingwell, FEMA/FDA to get). Data files stay out of the repo; this tracks provenance + the enrichment roadmap.
1 parent d970c0f commit b3ce46f

2 files changed

Lines changed: 33 additions & 0 deletions

File tree

β€Ždocs/DATA-SOURCES.mdβ€Ž

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Data Sources β€” acquisition tracker
2+
3+
What feeds the models, what each source unlocks, its license, and how to get it.
4+
5+
> **The data files themselves are never committed** (licensing + size β€” see
6+
> `.gitignore`). The training scripts download them; this file records provenance,
7+
> status, and the acquisition steps. Loaders **skip cleanly** when a file is absent,
8+
> so adding a source is just dropping its file in and re-running the pipeline.
9+
10+
## Status
11+
12+
| Source | Unlocks | License | Status | How to get |
13+
|--------|---------|---------|--------|-----------|
14+
| **ChemTastesDB v2.0** | sweet/bitter/umami training + sour/salty rule data (primary) | CC-BY-4.0 | βœ… in use | Zenodo record `14963136`, direct download (see `training/SETUP.md`). Column mapping verified. |
15+
| **cosylab/bittersweet** | more sweet/bitter volume | AGPL-3.0 | ⬜ to get | `git clone github.com/cosylabiiit/bittersweet` β†’ `bittersweet/data/*.tsv`. AGPL: keep a CC-BY-clean build with `INCLUDE_COSYLAB=False`, or opt in deliberately. |
16+
| **FlavorDB** | ~25k molecules taste+odor + natural-source mapping | confirm terms | ⬜ to get | REST/JSON API at `cosylab.iiitd.edu.in/flavordb` β†’ `flavordb_taste.csv`. Map taste fields; **[VERIFY] columns** before trusting. |
17+
| **UMP442 / BIOPEP-UWM** | more umami examples (only 283 now) | confirm terms | ⬜ to get | BIOPEP-UWM umami DB (form-driven) β†’ `umami_list.csv`. |
18+
| **SweetenersDB (ChΓ©ron 2017)** | the sweetness-**intensity** regressor (currently skipped) | publisher supp. | ⬜ to get | paper supplementary (~316 compounds, relative-to-sucrose) β†’ `sweeteners_db.csv`. Likely **paywalled β†’ manual grab**. |
19+
| **Pyrfume / Leffingwell** | the **aroma model** (OpenPOM) β€” issues #17 / #18 | per-set; confirm | ⬜ to get | `git clone github.com/pyrfume/pyrfume-data`; Leffingwell odor set. Separate, version-fragile effort. |
20+
| **FEMA GRAS / FDA SAF** | GRAS cross-reference + dosing/OAV lookups | gov public / FEMA | ⬜ to get | FDA "Substances Added to Food" (public domain) β†’ `gras_reference.parquet`; FEMA use-level PDFs β†’ `properties.parquet`. **Verify every scraped dosing number.** |
21+
22+
## Notes
23+
24+
- **Licensing.** ChemTastesDB (CC-BY-4.0) is the clean base. cosylab is AGPL β€” gated
25+
behind the `INCLUDE_COSYLAB` flag. Get an IP/OSS-license review before any
26+
commercial release; full attribution lives in [`SOURCES.md`](SOURCES.md).
27+
- **Column verification.** Each new source's column names must be checked against the
28+
loaders (the `[VERIFY]` markers) β€” see issue **#25**. ChemTastesDB's mapping is
29+
resolved (both the coarse `Class taste` and the granular `Taste` columns are parsed).
30+
- **Suggested priority.** (1) `Taste`-column mining β€” **done** βœ… β†’ (2) cosylab /
31+
FlavorDB for taste volume β†’ (3) SweetenersDB for the intensity head β†’ (4)
32+
Pyrfume / Leffingwell for the aroma model.

β€Ždocs/README.mdβ€Ž

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,5 +7,6 @@ Architecture, capability catalogue, and design decisions.
77
honest limits of public-data prediction
88
- `SOURCES.md` β€” data sources, libraries, research, and license attribution
99
- `API-CONTRACT.md` β€” the fixed JSON contract the .NET API exposes and the React UI consumes
10+
- `DATA-SOURCES.md` β€” data acquisition tracker: what each source unlocks and how to get it
1011

1112
These document *what we're building and why*, so decisions don't get re-argued.

0 commit comments

Comments
Β (0)