Chip legend, measured-property crawler, and the data-pipeline doc - #273
Merged
Merged
Conversation
…oc (#229, #26, #25) CHIP LEGEND (#229). Four dimensions with four different meanings sit side by side in the studios, and after the chip-system work in #269 the dot colour was the only thing distinguishing them — which works only if you already know the code. A key now states it once, up front: what each family means, and what solid / outline / hatched marking indicates (documented, predicted, indicative). The hatched swatch matters most: it is the one that says "right less than half the time — a hint, not an answer". MEASURED PROPERTIES (#209, in progress). 73% of the enrichment table had no measured boiling point, and the assumption was that the data did not exist. It does. build_properties.py reads PubChem's property table, which only ever carries COMPUTED values; measured boiling and melting points live in PUG-View under Experimental Properties, an endpoint this codebase had never called. A sample of molecules with no BP found 8 of 8 had a PubChem record. build_measured_properties.py crawls that endpoint. Parsing is the real work — values are free text written by whoever took the measurement ("246 °C", "115-116 °C at 12 mm Hg", "410 °F") — so it converts Fahrenheit, takes the midpoint of a range, and DISCARDS anything measured at reduced pressure. A boiling point at 12 mmHg is not comparable to one at atmospheric, and silently mixing them would corrupt the volatility ordering the formulation studio depends on. properties.parquet already carries boiling_point_pressure_mmhg for exactly this reason. Smoke test resolved 11 of 12 with values that check out (pyrrolidine 86.6 °C). The full crawl is ~7,200 molecules at roughly 57% hit rate and takes hours; it checkpoints every 100 so it can be folded in whenever it finishes. _by_skel() would have silently discarded all of it. It required an `inchikey` column, but both backfill tables are keyed by `inchikey_skel` — so it returned an empty dict, with no error, and every backfilled name and boiling point would simply never have appeared. It now accepts either. DATA-PIPELINE.md (#26, #25). What every build script reads, the REAL column names in each table read off the built artifacts rather than copied from the loader's hopes, and a clean-machine first run in dependency order with timings and an end-to-end verification that checks a prediction rather than just that the server started. Two claims in the first draft were wrong and are corrected: the verification example printed `.aroma.top[0]`, which is the highest-scoring CONFIDENT head and not the one you searched for; and I had written that `n_sources` breaks label conflicts by weight of evidence. It does not — nothing reads it, and conflicts resolve POSITIVE-WINS: one source calling a molecule sweet makes it sweet. That is the right default for sparse partially-annotated sources, but it means a single mislabelled source can assert a taste the others deny, which is worth knowing and is now written down. Also resolves the [OBTAIN]/[VERIFY] markers: flavordb_taste.csv is permanently excluded (NonCommercial), umami_list.csv is an unused nice-to-have, sweeteners_db.csv is present and drives the intensity regressor. Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>
rvnminers-A-and-N
added
area:data
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #229. Closes #26. Closes #25.
Chip legend (#229)
Four dimensions with four different meanings sit side by side in the studios, and after #269 the dot colour was the only thing distinguishing them — which works only if you already know the code. A key now states it once: what each family means, and what solid / outline / hatched indicates.
The hatched swatch is the one that matters: right less than half the time — a hint, not an answer.
Measured properties (#209, in progress)
73% of the enrichment table had no measured boiling point, and the assumption was the data didn't exist. It does. reads PubChem's property table, which only carries computed values; measured BP/MP live in PUG-View under Experimental Properties — an endpoint this codebase had never called. Sampling molecules with no BP found 8 of 8 had a PubChem record.
crawls it. Parsing is the real work — free text written by whoever took the measurement (
246 °C,115-116 °C at 12 mm Hg,410 °F) — so it converts Fahrenheit, takes range midpoints, and discards reduced-pressure values. A BP at 12 mmHg isn't comparable to atmospheric, and mixing them would corrupt the volatility ordering the formulation studio depends on. ( already carries for exactly this reason.)Smoke test: 11 of 12 resolved, values check out (pyrrolidine 86.6 °C ✓). Full crawl is ~7,200 molecules at ~57% hit rate, several hours, checkpointing every 100 — folded in at the next rebuild. #209 stays open until it lands.
** would have silently discarded all of it.** It required an column, but both backfill tables key on — returning an empty dict, no error, so every backfilled name and boiling point would never have appeared. Now accepts either.
(#26, #25)
Every build script's inputs, the real column names read off the built artifacts rather than copied from the loader's hopes, and a clean-machine first run in dependency order with timings — verified end-to-end against the running app.
Two claims in my first draft were wrong and are corrected:
/ markers resolved: permanently excluded (NonCommercial), unused nice-to-have, present and driving the intensity regressor.