diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..8574e03 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,25 @@ +# Keep the build context small and the image free of anything that isn't runtime code. +.git +.github +.claude +**/__pycache__ +**/*.pyc +.venv +venv +# Trained artifacts are MOUNTED, never baked in — see the Dockerfile header. +**/*.joblib +**/*.parquet +**/*.npz +**/*.onnx +models/ +aroma_models/ +taste_models/ +mouthfeel_models/ +tox_models/ +onnx_models/ +# Dev-only +tests/ +docs/ +node_modules/ +*.md +!README.md diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..e4e984d --- /dev/null +++ b/.env.example @@ -0,0 +1,7 @@ +# Copy to .env and adjust. Every value has a working default, so `docker compose up` runs as-is. +APP_PORT=8000 +# Where the trained models + parquet tables live on the host (~1 GB, mounted read-only). +MODELS_DIR=./models +# 0 = auto (three quarters of available cores) +INFER_WORKERS=0 +POSTGRES_PASSWORD=flavormancer diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..c77274f --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,63 @@ +# Changelog + +All notable changes to Flavormancer. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); +versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +Flavormancer is pre-1.0. **v1.0.0 is the MVP** — the point at which both tracks (the Python +service and the .NET/React Track B) are running, packaged and documented. Until then, minor +versions ship working increments. + +## [Unreleased] + +## [0.2.0] — 2026-07-30 + +The honesty release. Every trained head now publishes how good it actually is, and the packaging +exists to install it somewhere other than the machine it was built on. + +### Added +- **Per-head calibrated thresholds with a 50% precision floor.** Each of the 190 heads carries a + decision threshold fitted on out-of-fold predictions, plus its measured precision and recall, + published in `/api/heads` and shown on every bar. Heads that cannot be right more than half the + time are marked `indicative` rather than confident — kept in full, never dressed up. +- **`docs/ACCURACY.md`** — a plain-language explanation of AUROC, precision, thresholds and + cross-validation, written to be read without a machine-learning background. +- **`training/audit_generalization.py`** — fires every head across the whole corpus and counts + discoveries outside its training set, to catch heads that memorise rather than learn. +- **Mouthfeel modality** — 5 trigeminal/chemesthesis heads (cooling, warming, pungent, tingling, + astringent), surfaced across reads, cards, chips and the map. +- **Docker packaging** — `Dockerfile`, `docker-compose.yml` with a pgvector-backed Postgres, and a + schema for the substitution index. Artifacts mount at runtime rather than baking into the image. +- `FLAVORMANCER_HOME` so the code and the ~1 GB of trained artifacts can live in different places. +- Numbers glossary in `HOW-IT-WORKS.md` and in the app's own How-it-works panel. + +### Changed +- Aroma roster **164 → 166 heads**; confident-capable heads **94 → 108**. +- Chip families (flavor / note / taste / mouthfeel) share one visual language instead of four + accidental ones, and each studio section explains what its dimension *is*. +- Every molecule has a display name: names fall back to molecular formula, with multi-component + structures labelled as mixtures. **8,861 of 8,861 named**, down from 770 blank. +- Milestones relabelled to say what kind of work they hold (Foundations / Track B / Ship). + +### Fixed +- `predict_aroma` had silently lost its `@lru_cache` to an orphaned decorator — the fix behind the + 40s → 0.01s modal read. +- Substitute/neighbor cards overflowed the modal on mobile (`1fr` will not shrink below + min-content; needed `minmax(0,1fr)`). +- The modal's 14px card gap had never applied, because an inline `display:block` overrode the flex + column and block boxes ignore `gap`. +- A duplicate `pungent` key in the curated supplement that would have deleted six molecules. + +### Known limits +- `sweet`, `ethereal` and `pungent` odour heads sit exactly at the precision floor. They are broad + *and* chemically incoherent, so curation cannot lift them — tracked for the GNN work (#199). +- 58 aroma heads remain `indicative` (#262). +- Track B (.NET API, React workbench) is scaffolded but not running (M2/M3). + +## [0.1.0] — 2026-07-16 + +First public demo: taste heads, the aroma descriptor model, substitution search, the flavor-space +map, formulation studio, and the on-prem workbench UI. + +[Unreleased]: https://github.com/echelonts/flavormancer/compare/v0.2.0...HEAD +[0.2.0]: https://github.com/echelonts/flavormancer/compare/v0.1.0...v0.2.0 +[0.1.0]: https://github.com/echelonts/flavormancer/releases/tag/v0.1.0 diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..b413a03 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,52 @@ +# Flavormancer — the on-prem prediction service. +# +# Deliberately a RUNTIME image, not a training one. Model training needs a GPU box, 32 cores and +# several hours; serving needs none of that. Trained artifacts (aroma_models/, taste_models/, +# mouthfeel_models/, tox_models/ and the parquet tables) are mounted at run time rather than baked +# in — they are ~1 GB, they change on every retrain, and burning them into a layer would make the +# image both enormous and stale the moment a head is retrained. +# +# docker build -t flavormancer:latest . +# docker compose up # see docker-compose.yml for the volume wiring +# +# RDKit is the reason for the slim-bookworm base rather than alpine: it ships manylinux wheels +# that need glibc, and building it from source on musl is hours of pain for no benefit. + +FROM python:3.12-slim-bookworm AS base + +# libxrender/libxext are RDKit's molecule-drawing dependencies (the structure SVGs); libgomp is +# OpenMP, which scikit-learn's forests use for parallel predict_proba. +RUN apt-get update && apt-get install -y --no-install-recommends \ + libxrender1 libxext6 libgomp1 curl \ + && rm -rf /var/lib/apt/lists/* + +ENV PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 \ + PIP_NO_CACHE_DIR=1 \ + FLAVORMANCER_HOME=/app/data + +WORKDIR /app + +# Dependencies first, in their own layer: they change far less often than the source, so an app +# edit rebuilds in seconds instead of reinstalling RDKit. +COPY requirements.txt ./ +RUN pip install --no-cache-dir -r requirements.txt \ + && pip install --no-cache-dir "fastapi" "uvicorn[standard]" pydantic + +COPY training/ ./training/ + +# Serving runs as an unprivileged user. The mounted model directory only needs to be readable. +RUN useradd --create-home --shell /usr/sbin/nologin flavormancer \ + && mkdir -p /app/data && chown -R flavormancer:flavormancer /app +USER flavormancer + +WORKDIR /app/training +EXPOSE 8000 + +# /healthz answers before the models finish loading (the app serves a warming page meanwhile), so +# a long start-period is what keeps the container from being killed during a legitimate ~50s +# cold start. See docs/METHODS.md on why loading is serial. +HEALTHCHECK --interval=15s --timeout=5s --start-period=180s --retries=4 \ + CMD curl -fsS http://localhost:8000/healthz || exit 1 + +CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..e7ed86a --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,62 @@ +# Flavormancer — single-box deployment. +# +# One machine, a handful of users, no orchestration. That is the actual shape of an on-prem +# install for a flavour house, so this is Compose rather than Kubernetes — see +# docs/ARCHITECTURE.md. +# +# docker compose up -d +# curl localhost:8000/healthz +# +# The trained models are NOT in the image. Point MODELS_DIR at wherever they live on the host +# (default ./models) — they are ~1 GB of joblib forests plus the parquet tables, rebuilt by the +# training scripts on a box with cores to spare. + +name: flavormancer + +services: + app: + build: . + image: flavormancer:latest + restart: unless-stopped + ports: + - "${APP_PORT:-8000}:8000" + environment: + # Serving is CPU-bound on forest inference; leave the box some headroom for Postgres. + FLAVORMANCER_INFER_WORKERS: "${INFER_WORKERS:-0}" # 0 = auto (3/4 of cores) + FLAVORMANCER_HOME: /app/data # where the mounted artifacts live + DATABASE_URL: "postgresql://flavormancer:${POSTGRES_PASSWORD:-flavormancer}@db:5432/flavormancer" + volumes: + # One clean mount, because the app now resolves artifacts under FLAVORMANCER_HOME rather + # than the working directory. Before that, ANY bind that reached the models also shadowed + # app.py — the container would have started with no application code. + - "${MODELS_DIR:-./models}:/app/data:ro" + depends_on: + db: + condition: service_healthy + healthcheck: + test: ["CMD", "curl", "-fsS", "http://localhost:8000/healthz"] + interval: 15s + timeout: 5s + start_period: 180s # a cold start loads 190 heads; see docs/METHODS.md + retries: 4 + + db: + # pgvector, not plain Postgres: the substitution index is a nearest-neighbour search over the + # 177-dimension flavour-profile vector, which is exactly what pgvector exists for (#20). + image: pgvector/pgvector:pg16 + restart: unless-stopped + environment: + POSTGRES_DB: flavormancer + POSTGRES_USER: flavormancer + POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:-flavormancer}" + volumes: + - pgdata:/var/lib/postgresql/data + - ./infra/initdb:/docker-entrypoint-initdb.d:ro + healthcheck: + test: ["CMD-SHELL", "pg_isready -U flavormancer -d flavormancer"] + interval: 10s + timeout: 5s + retries: 5 + +volumes: + pgdata: diff --git a/infra/initdb/01-schema.sql b/infra/initdb/01-schema.sql new file mode 100644 index 0000000..21631cd --- /dev/null +++ b/infra/initdb/01-schema.sql @@ -0,0 +1,31 @@ +-- Flavormancer schema (#20). +-- +-- The substitution index is a nearest-neighbour search over the flavour-profile vector, so the +-- vector lives in the database rather than being recomputed per query. 177 dimensions: +-- 6 taste + 166 aroma + 5 mouthfeel. Tox is deliberately NOT in the vector — safety is not a +-- flavour-match dimension, and letting it steer "what tastes similar" would be wrong. +CREATE EXTENSION IF NOT EXISTS vector; + +CREATE TABLE IF NOT EXISTS molecule ( + inchikey_skel text PRIMARY KEY, -- stereo-agnostic connectivity skeleton + smiles text NOT NULL, + name text, -- never null in practice: falls back to formula + mw real, + logp real, + tpsa real, + food_listed boolean DEFAULT false, -- open-gov register listing, NOT a safety clearance + taste_documented text +); + +CREATE TABLE IF NOT EXISTS molecule_profile ( + inchikey_skel text PRIMARY KEY REFERENCES molecule(inchikey_skel) ON DELETE CASCADE, + profile vector(177) NOT NULL, + aromas text[] -- heads clearing their own calibrated threshold +); + +-- Cosine distance: the profile is a direction in flavour space, and two molecules with the same +-- balance of notes at different intensities should still read as neighbours. +CREATE INDEX IF NOT EXISTS molecule_profile_cos + ON molecule_profile USING hnsw (profile vector_cosine_ops); + +CREATE INDEX IF NOT EXISTS molecule_food_listed ON molecule (food_listed); diff --git a/training/app.py b/training/app.py index 70c23a7..2acfff9 100644 --- a/training/app.py +++ b/training/app.py @@ -164,13 +164,13 @@ def _load_name2smiles(): idx = {} with contextlib.suppress(Exception): # table absent / no pandas; live lookup still covers it import pandas as pd - df = pd.read_parquet("master_enrichment.parquet") + df = pd.read_parquet(P.artifact("master_enrichment.parquet")) for nm, smi in zip(df["name"], df["smiles"]): if isinstance(nm, str) and isinstance(smi, str) and nm.strip() and smi.strip(): idx.setdefault(nm.strip().lower(), smi) with contextlib.suppress(Exception): # no suggest file; fine import csv - with open("flavor_volatiles.csv", encoding="utf-8") as fh: + with open(P.artifact("flavor_volatiles.csv"), encoding="utf-8") as fh: for r in csv.DictReader(fh): if r.get("name") and r.get("smiles"): idx.setdefault(r["name"].strip().lower(), r["smiles"]) @@ -220,7 +220,7 @@ def _load_name_table(): written name columns; live PubChem stays the fallback for anything not in the table.""" try: import pandas as pd - df = pd.read_parquet("properties.parquet") + df = pd.read_parquet(P.artifact("properties.parquet")) if "common_name" not in df.columns: return {} out = {} @@ -238,7 +238,7 @@ def _merge_iupac_backfill(table): the main properties crawl missed (skeleton -> keep any common name, add the IUPAC).""" try: import pandas as pd - bf = pd.read_parquet("iupac_backfill.parquet") + bf = pd.read_parquet(P.artifact("iupac_backfill.parquet")) except Exception: # noqa: BLE001 — backfill not built; nothing to merge return table for skel, u in zip(bf["inchikey_skel"], bf["iupac_name"]): @@ -306,7 +306,7 @@ def _load_spectra(): public-domain PubChem availability metadata. Empty until the crawl has run.""" try: import pandas as pd - df = pd.read_parquet("spectra.parquet") + df = pd.read_parquet(P.artifact("spectra.parquet")) labels = [("has_ms", "MS"), ("has_ir", "IR"), ("has_nmr", "NMR"), ("has_uv", "UV"), ("has_raman", "Raman")] out = {} @@ -1403,7 +1403,7 @@ def _precompute_top_lists(): aroma heads over the odor corpus. Model-derived: honest 'what the tool predicts'.""" import pandas as pd with contextlib.suppress(Exception): # no taste data; skip taste lists - tm = pd.read_parquet("taste_master.parquet") + tm = pd.read_parquet(P.artifact("taste_master.parquet")) for taste, clf in P._CLASSIFIERS.items(): ranked = _rank(tm["smiles"], lambda X, c=clf: c.predict_proba(X)[:, 1]) _TOP_LISTS[f"taste:{taste}"] = {"label": f"Top {taste}", "items": _named_top(ranked)} @@ -1418,7 +1418,7 @@ def _precompute_top_lists(): # skews industrial, so ranking by a head surfaces confident-but-odd picks (cyanide under # "almond"). Documented examples are real, recognizable, and honest ("documented citrus"). from build_aroma_dataset import tag as _odor_tag - od = pd.read_parquet("odor_notes.parquet") + od = pd.read_parquet(P.artifact("odor_notes.parquet")) by_desc = {} for _, r in od.iterrows(): nm, odor = r.get("name"), r.get("odor") @@ -1456,7 +1456,7 @@ def _load_flavor_map(): + 3D (x3,y3,z3) coordinates normalized to 0..1 with names — an interactive scatter / cloud.""" try: import pandas as pd - df = pd.read_parquet("flavor_map.parquet") + df = pd.read_parquet(P.artifact("flavor_map.parquet")) # UMAP occasionally emits NaN coords for a few near-duplicate rows — drop them so the # JSON stays valid (NaN isn't JSON-compliant) and the scatter has no phantom points. df = df.dropna(subset=[c for c in ("x", "y", "x3", "y3", "z3") if c in df.columns]).reset_index(drop=True) @@ -1570,7 +1570,7 @@ def _precompute_design(): import numpy as np import pandas as pd from build_aroma_dataset import tag as _odor_tag - od = pd.read_parquet("odor_notes.parquet") + od = pd.read_parquet(P.artifact("odor_notes.parquet")) rows = [] # (smiles, name, mol_skeleton, {documented tags}) for smi, nm, odor in zip(od["smiles"], od.get("name", [None] * len(od)), od["odor"]): mol = Chem.MolFromSmiles(str(smi)) if isinstance(smi, str) else None @@ -1880,7 +1880,7 @@ def _load_enrichment(): """Rows from master_enrichment.parquet with taste collapsed to a display string.""" try: import pandas as pd - df = pd.read_parquet("master_enrichment.parquet") + df = pd.read_parquet(P.artifact("master_enrichment.parquet")) except Exception: # noqa: BLE001 — not built yet return [] def _s(v): # NaN (a truthy float) -> "" ; keep real strings @@ -2069,7 +2069,7 @@ def _load_odor_table(): build_odor_notes.py has run; tolerant of older tables without the threshold columns.""" try: import pandas as pd - df = pd.read_parquet("odor_notes.parquet") + df = pd.read_parquet(P.artifact("odor_notes.parquet")) def col(name): return df[name] if name in df.columns else [None] * len(df) @@ -2106,13 +2106,13 @@ def _load_documented_full(): out = {} with contextlib.suppress(Exception): import pandas as pd - od = pd.read_parquet("odor_notes.parquet") + od = pd.read_parquet(P.artifact("odor_notes.parquet")) for ik, odor in zip(od["inchikey"], od["odor"]): if isinstance(ik, str) and isinstance(odor, str) and odor.strip(): out.setdefault(ik, {})["odor"] = odor.strip().split("\n")[0][:160] with contextlib.suppress(Exception): import pandas as pd - tn = pd.read_parquet("taste_notes.parquet") + tn = pd.read_parquet(P.artifact("taste_notes.parquet")) for ik, taste in zip(tn["inchikey"], tn["taste"]): if isinstance(ik, str) and isinstance(taste, str) and taste.strip(): out.setdefault(ik, {})["taste"] = taste.strip().split("\n")[0][:160] diff --git a/training/predict.py b/training/predict.py index 48bba8b..b2f1de0 100644 --- a/training/predict.py +++ b/training/predict.py @@ -62,9 +62,21 @@ rdMolDescriptors, ) +# Where the trained artifacts live. Defaults to the working directory, which is how the systemd +# deployment has always run (code and models share one directory). Setting FLAVORMANCER_HOME lets +# a container bake the CODE into the image while MOUNTING the ~1 GB of models and parquet tables — +# without it, any bind mount that reached the artifacts would also shadow app.py. +HOME = Path(os.environ.get("FLAVORMANCER_HOME") or ".") + + +def artifact(name): + """Resolve one trained artifact (model directory, parquet or csv) under FLAVORMANCER_HOME.""" + return HOME / name + + FP_BITS, FP_RADIUS = 2048, 2 _MORGAN = rdFingerprintGenerator.GetMorganGenerator(radius=FP_RADIUS, fpSize=FP_BITS) -TASTE = Path("taste_models") +TASTE = artifact("taste_models") ACID_SMARTS = { # Match both protonated (-OH) and deprotonated (-O-) forms — sour compounds are @@ -209,13 +221,13 @@ def _load_rf(path): _TASTE_META = {} # taste -> {auroc, ...} from taste_models/manifest.json (held-out score) _TOX_MODELS = {} # Tox21 caution-only assay heads (INDICATIVE, never a determination) _TOX_META = {} # assay -> {auroc, n_pos, ...} from tox_models/manifest.json (held-out CV) -_TOX_DIR = Path("tox_models") +_TOX_DIR = artifact("tox_models") _AROMA_MODELS = {} # HSDB odor-descriptor heads (presence/absence; NOT intensity) _AROMA_META = {} -_AROMA_DIR = Path("aroma_models") +_AROMA_DIR = artifact("aroma_models") _MOUTHFEEL_MODELS = {} # trigeminal/chemesthesis heads (warming/astringent/tingling), own modality _MOUTHFEEL_META = {} -_MOUTHFEEL_DIR = Path("mouthfeel_models") +_MOUTHFEEL_DIR = artifact("mouthfeel_models") MODELS_READY = _threading.Event() # set once every head is loaded; the app gates requests on this _INFER_POOL = None # shared thread pool for fanning a novel-molecule read across cores (lazy) @@ -321,12 +333,12 @@ def _load_all_models(): # is how the salty/sour data works as a FLAG without a model — if a queried # molecule is in our labeled set, we report the verified fact instead of a guess. _KNOWN = {} # inchikey -> {taste: 1} -_MASTER = Path("taste_master.parquet") +_MASTER = artifact("taste_master.parquet") # The neighbor / substitute reference set: the FULL molecule universe (every structure we know, # ~8.8k) so structural neighbors and profile substitutes can surface ANY molecule — e.g. ethyl # vanillin as the top vanillin substitute — not just the taste-labelled subset. Falls back to # taste_master when the enrichment table hasn't been built yet. -_UNIVERSE = Path("master_enrichment.parquet") +_UNIVERSE = artifact("master_enrichment.parquet") if _MASTER.exists(): import pandas as pd _m = pd.read_parquet(_MASTER) @@ -342,7 +354,7 @@ def _load_all_models(): # 'inchikey' column and we cross-check against it; absent the file we say so # honestly rather than guessing. _GRAS = set() -_GRAS_FILE = Path("gras_reference.parquet") +_GRAS_FILE = artifact("gras_reference.parquet") if _GRAS_FILE.exists(): import pandas as pd _g = pd.read_parquet(_GRAS_FILE) @@ -377,7 +389,7 @@ def _foodsafe_label(fl, cfr): return f"{term} — {' & '.join(refs)}{tag}" if refs else term -_FOODSAFE_FILE = Path("food_safe_supplement.csv") +_FOODSAFE_FILE = artifact("food_safe_supplement.csv") _FOODSAFE_BASIS = {} # skeleton -> specific open-gov label (term + refs + jurisdiction) if _FOODSAFE_FILE.exists(): import pandas as pd @@ -400,7 +412,7 @@ def _clean(v): # permitted); every AUTHORISED row is a food-cleared flavouring cited by its FL number. Union into # the food-use reference so the whole authorised list reads food-listed with a specific citation. # The FILE is a private data asset (gitignored); this LOADER is open framework. -_GB_FILE = Path("gb_union_list.csv") +_GB_FILE = artifact("gb_union_list.csv") if _GB_FILE.exists(): import pandas as pd _gb = pd.read_csv(_GB_FILE, dtype=str, keep_default_na=False) @@ -1336,7 +1348,7 @@ def _build_sub_index(): # Fast path: load the precomputed profile index (build_profile_index.py). The 178-dim # inference over ~8.8k molecules is slow (~3 min); the cache makes startup instant. We only # rebuild the cheap Morgan fingerprints from SMILES on load. - cache = Path("profile_index.npz") + cache = artifact("profile_index.npz") if cache.exists(): z = np.load(cache, allow_pickle=True) smis = [str(s) for s in z["smiles"]]