Resolve names from master_enrichment, the one base table - #276
Merged
Merged
Conversation
…ble (#224) app.py built its own name lookup from properties.parquet and then merged iupac_backfill.parquet itself. build_enrichment.py had already done that work — and better — so the two diverged in three ways, all silent: 1. The enrichment build resolves a name curated-list-first, then PubChem common name, then IUPAC name, then a molecular-formula fallback, un-inverting CAS-style ordering along the way. None of that reached app.py's table. 2. The merge read only `iupac_name` from the backfill and ignored the `common_name` column, so every PubChem Title the crawler recovered was thrown away on arrival. 3. Nothing carried the formula fallback, so the molecules that only have one resolved to nothing. Measured before the change: 754 molecules had a name in master_enrichment that app._NAME_TABLE did not know — and not obscure ones. cedrol, fenchyl alcohol, hydroxycitronellal and musk ketone all rendered by name in the grid and came back empty from the UI's own lookup. After: 0. Sourcing the resolved name straight off the base table makes the grid and the lookup agree by construction instead of by coincidence, and deletes a whole merge function rather than adding one. That is what #224 is actually asking for, applied where the duplication genuinely was. Note what is deliberately NOT folded in. profile_index.npz is an 8,850 x 183 matrix — that 183 is the FLAVOUR-PROFILE vector (6 taste + 172 aroma + 5 mouthfeel), not the head count, which is 195. The 12 Tox21 heads are excluded on purpose: safety is not a flavour-match dimension and must never steer "what tastes similar". flavor_map.parquet is a UMAP embedding. Different shapes serving different questions; forcing either into a row-oriented molecule table would be worse, not tidier. odor_notes/taste_notes are documented-text SOURCES the enrichment build consumes — upstream of the base table, not competitors to it. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>
Prompted by a fair question — "don't we have more heads now?" — which turned out to be pointing at a genuinely ambiguous line. Two different numbers had been drifting: 195 trained heads 6 taste + 172 aroma + 5 mouthfeel + 12 safety 183 profile dimensions 6 taste + 172 aroma + 5 mouthfeel (tox EXCLUDED, deliberately) Tox is kept out of the flavour-profile vector on purpose: safety is not a flavour-match dimension, and if it were in there two molecules could rank as "similar tasting" partly because they share a liver-toxicity signature. That is wrong in a formulation tool and dangerous in a food one. The stale numbers were spread across the schema, six docs and three source files — 175/177/178-dim profiles and 166/167-head rosters, each correct at some point in the last two days. The pgvector column was the worst of them: vector(177) would have rejected every insert of a 183-dim profile at runtime, and nothing would have caught it until the first real docker compose up. Also corrects a comment in predict.py that was not merely stale but actively wrong. It claimed the model load "is fanned out across cores (joblib.load releases the GIL)". Both halves are false — unpickling is GIL-bound, which is why threads measured ~2.5x SLOWER, and the process pool that IS faster deadlocks because loading runs during module import. Anyone optimising from that comment would have re-walked the exact path #225 documents as a dead end. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #224.
app.pybuilt its own name lookup fromproperties.parquetand mergediupac_backfill.parquetitself.build_enrichment.pyhad already done that work — and better — so the two diverged silently in three ways:app.py's table.iupac_namefrom the backfill and ignored thecommon_namecolumn — so every PubChem Title the crawler recovered was thrown away on arrival.Measured
754 molecules had a name in
master_enrichmentthatapp._NAME_TABLEdidn't know — and not obscure ones:All four render by name in the grid and came back empty from the UI's own lookup. After this change: 0.
What this actually does
Sourcing the resolved name straight off the base table makes the grid and the lookup agree by construction rather than by coincidence — and deletes a merge function rather than adding one. That's what #224 asks for, applied where the duplication genuinely was.
What is deliberately NOT folded in
profile_index.npzis an 8,850 × 183 float matrix;flavor_map.parquetis a UMAP embedding. Different shapes serving different questions — forcing them into a row-oriented molecule table would be worse, not tidier.odor_notes/taste_notesare documented-text sources the enrichment build consumes. They're upstream of the base table, not competitors to it.ruff clean; suite 54 passed