Merge pull request #2 from rastagan-git/agent/multi-source-aroma-skill

feat: package the crawler as a multi-source aroma data skill

Author: rastagan-git

.agents/skills/curate-flavor-data/SKILL.md

@@ -0,0 +1,45 @@
+---
+name: curate-flavor-data
+description: Validate, normalize, enrich, and export flavor, odor, aroma, and chemical-sensory datasets with source-level provenance. Use for XLSX, CSV, or TSV workbooks containing CAS numbers, compound names, calculated retention indices, sensory descriptors, odor thresholds, or olfactory-receptor evidence; for selecting among NIST, PubChem, Pyrfume, M2OR, MFFI, and the permission-gated ChemicalBook compatibility connector; and for verifying analysis-ready outputs. Do not use to bypass access controls, relicense source data, or make biomedical or machine-learning claims unsupported by the exported evidence.
+---
+
+# Curate Flavor Data
+
+Build traceable compound tables through the repository's `flavor-data` CLI. Keep the skill as an orchestration layer; modify provider behavior in the Python package, not here.
+
+## Workflow
+
+1. Inspect the input without modifying it.
+   - Run `python .agents/skills/curate-flavor-data/scripts/inspect_workbook.py INPUT` from the repository root.
+   - Confirm the row count, exact column names, identifier quality, duplicates, and formula-like cells.
+2. Choose the smallest provider set that supplies the requested fields.
+   - Read [references/provider-matrix.md](references/provider-matrix.md) before any network or browser operation.
+   - Prefer PubChem for canonical identity and sourced odor annotations.
+   - Use NIST only for the existing retention-index or name-resolution workflows.
+   - Use Pyrfume only for explicitly selected archives after reviewing each manifest note.
+   - Use M2OR only when receptor bioassay evidence is relevant; label species and assay scope.
+3. Preview the operation.
+   - State the input, new output path, selected provider, expected columns, approximate request count, cache behavior, and material access caveats.
+   - Write a sibling output by default. Do not pass `--force` or overwrite the input unless the user explicitly requests that exact replacement.
+4. Run one focused command.
+   - Identity and odor metadata: `flavor-data pubchem INPUT --identifier-column "CAS Number"`
+   - Retention indices: `flavor-data nist-ri INPUT --cas-column "CAS Number" --calculated-ri-column "Calculated RI"`
+   - Names to CAS: `flavor-data resolve-cas INPUT --name-column "Name"`
+   - Curated descriptors: `flavor-data pyrfume INPUT --archives aromadb,superscent`
+   - Receptor evidence: `flavor-data m2or INPUT --cas-column "CAS Number"`
+   - Source inventory: `flavor-data sources`
+5. Verify the result.
+   - Re-run the inspection script on the output.
+   - Confirm identical row order and row count, expected new fields, typed status counts, source URL, retrieval time, version, and license/access fields.
+   - Treat `http_error`, `network_error`, `parse_error`, `missing_data`, `data_error`, `partial`, `blocked`, and `skipped` separately from `not_found`.
+   - Consult [references/output-schema.md](references/output-schema.md) when reconciling columns or statuses.
+6. Report the output path, provider versions, status counts, partial failures, and any source terms the user must still review.
+
+## Guardrails
+
+- Keep NIST's interval at five seconds or slower and retain caching.
+- Never automate CAPTCHA solving. Keep browser sources visible when user intervention may be required.
+- Do not run ChemicalBook automation unless the user confirms documented permission; its current robots policy excludes the legacy routes.
+- Do not describe the toolkit as AI-powered. Say that structured exports can support downstream statistics, cheminformatics, or machine-learning experiments.
+- Do not bundle or republish downloaded Pyrfume or M2OR data in the repository.
+- Preserve remote text as literal spreadsheet cells to prevent formula execution.

.agents/skills/curate-flavor-data/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Curate Flavor Data"
+  short_description: "Curate traceable odorant and flavor datasets"
+  default_prompt: "Use $curate-flavor-data to inspect and enrich this odorant workbook with source provenance."

.agents/skills/curate-flavor-data/references/output-schema.md

@@ -0,0 +1,40 @@
+# Output and validation contract
+
+## Identity
+
+- Preserve the original input identifier.
+- Use PubChem CID and InChIKey as canonical joins when available.
+- Keep CAS as an external identifier and validate its checksum before exact-match providers.
+- Never silently pick an ambiguous name result.
+
+## Status values
+
+- `ok`: a provider returned a parsed record.
+- `not_found`: the provider responded successfully but no exact record matched.
+- `ambiguous`: more than one defensible identity matched.
+- `invalid_input`: the identifier is empty, malformed, or fails validation.
+- `network_error`: transport or remote-service failure; safe to retry later.
+- `http_error`: the provider returned a non-success HTTP response that was not a clean not-found result.
+- `parse_error`: the response arrived but its structure could not be interpreted; investigate a selector/schema change.
+- `missing_data`: a required local or cached snapshot is unavailable and downloading is disabled or failed before parsing.
+- `data_error`: every selected archive failed to load; inspect the per-archive diagnostic message.
+- `partial`: at least one selected source succeeded and at least one failed, or a provider returned usable data with warnings.
+- `blocked`: access policy or missing permission prevented the request.
+- `skipped`: a user deliberately skipped an interactive record.
+
+Do not merge an access, transport, HTTP, snapshot, parse, or partial failure into `not_found`.
+
+## Provenance
+
+Keep provider status, source URL, retrieval time, cache-hit flag, pinned version or snapshot, license URL, and message columns. For PubChem odor text, also keep contributor source names, URLs, and license URLs.
+
+## Workbook QA
+
+After every run, confirm:
+
+1. Output row count and row order equal the input.
+2. Original columns remain present.
+3. Requested output columns exist.
+4. Every processed row has a typed status.
+5. Remote strings beginning with `=`, `+`, `-`, or `@` are stored as literal text.
+6. Partial outputs are reported separately if a run is interrupted.

.agents/skills/curate-flavor-data/references/provider-matrix.md

@@ -0,0 +1,14 @@
+# Provider matrix
+
+Review these notes before choosing or running a provider. Access policies can change; re-check the linked official page when a live run or redistribution decision matters.
+
+| Provider | Best use | Access behavior | Rights and scientific caveat |
+| --- | --- | --- | --- |
+| [PubChem PUG REST](https://pubchem.ncbi.nlm.nih.gov/docs/pug-rest) and [PUG-View](https://pubchem.ncbi.nlm.nih.gov/docs/pug-view) | CID, InChIKey, structures, formula, properties, sourced odor annotations | Live API, cached locally, below the official five-requests-per-second ceiling, bounded retries | PubChem aggregates contributor records. Preserve contributor source, URL, and license URL for annotation text. |
+| [NIST Chemistry WebBook](https://webbook.nist.gov/chemistry/) | Existing GC retention-index and name-to-CAS workflows | Cached HTML, at least five seconds between uncached requests per [robots.txt](https://webbook.nist.gov/robots.txt) | NIST SRD compilation rights apply. Fetch on demand and cite; do not redistribute a bulk scrape. |
+| [Pyrfume Public Data Archive](https://github.com/pyrfume/pyrfume-data) | Curated odor descriptors or collection membership keyed by PubChem CID | Pinned GitHub snapshot; explicit archive allowlist; files cached locally | The repository code is MIT, but manifests record upstream rights and sometimes copyright caveats. Do not treat every archive as MIT-licensed data. |
+| [M2OR](https://github.com/chemosim-lab/M2OR) | Molecule-olfactory-receptor pairs, species, responsive assays, and study DOI | Optional pinned CSV download (about 43 MB), cached locally | Dataset repository is Apache-2.0. Results are assay evidence, not human odor perception or clinical prediction. |
+| [MFFI](https://mffi.sjtu.edu.cn/database/search) | Chinese/English names, sensory characteristics, and water thresholds | Interactive Selenium browser, conservative pacing | No public API, rate policy, or reuse license was found. Robots allowance is not a data license; use conservatively and cite. |
+| [ChemicalBook](https://www.chemicalbook.com/) | Original odor/threshold/type compatibility path only | Disabled until explicit permission confirmation; visible manual browser | Current [robots.txt](https://www.chemicalbook.com/robots.txt) excludes the search and product-property routes. Never bypass CAPTCHA or imply permission. |
+
+Avoid automated extraction from FlavorDB2, Flavornet, Good Scents, VCF, or other attractive-but-undocumented sites unless an official API, bulk download, or written permission clearly covers the intended use.

.agents/skills/curate-flavor-data/scripts/inspect_workbook.py

@@ -0,0 +1,70 @@
+"""Inspect a flavor-data workbook without modifying it."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[4]
+if str(REPO_ROOT) not in sys.path:
+    sys.path.insert(0, str(REPO_ROOT))
+
+from flavor_data_crawler.excel_io import FORMULA_PREFIXES, read_table  # noqa: E402
+from flavor_data_crawler.identifiers import is_valid_cas, normalize_cas  # noqa: E402
+
+
+def inspect(path: Path, cas_column: str | None = None) -> dict[str, object]:
+    frame = read_table(path)
+    detected_cas = cas_column or next(
+        (name for name in ("CAS Number", "CAS", "cas", "cas_number") if name in frame.columns),
+        None,
+    )
+    formula_like = 0
+    for column in frame.columns:
+        formula_like += int(
+            frame[column]
+            .map(lambda value: isinstance(value, str) and value.startswith(FORMULA_PREFIXES))
+            .sum()
+        )
+    report: dict[str, object] = {
+        "path": str(path.resolve()),
+        "rows": len(frame),
+        "columns": [str(column) for column in frame.columns],
+        "duplicate_rows": int(frame.duplicated().sum()),
+        "formula_like_cells": formula_like,
+        "missing_by_column": {
+            str(column): int(frame[column].isna().sum() + frame[column].eq("").sum())
+            for column in frame.columns
+        },
+    }
+    if detected_cas:
+        normalized = frame[detected_cas].map(normalize_cas)
+        valid_mask = normalized.map(is_valid_cas)
+        nonempty_mask = normalized.ne("")
+        invalid = normalized[nonempty_mask & ~valid_mask]
+        report["cas"] = {
+            "column": detected_cas,
+            "nonempty": int(nonempty_mask.sum()),
+            "valid": int(valid_mask.sum()),
+            "invalid": int((nonempty_mask & ~valid_mask).sum()),
+            "invalid_examples": list(dict.fromkeys(invalid.astype(str)))[:10],
+            "duplicates": int(normalized[nonempty_mask].duplicated().sum()),
+        }
+    else:
+        report["cas"] = {"column": None, "note": "No common CAS column name detected"}
+    return report
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("input", type=Path)
+    parser.add_argument("--cas-column")
+    args = parser.parse_args()
+    print(json.dumps(inspect(args.input, args.cas_column), ensure_ascii=False, indent=2))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.github/workflows/ci.yml

@@ -0,0 +1,42 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: ${{ matrix.os }} / Python ${{ matrix.python-version }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        python-version: ["3.11", "3.13"]
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install package and development dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -e ".[dev]"
+
+      - name: Lint
+        run: python -m ruff check .
+
+      - name: Check formatting
+        run: python -m ruff format --check .
+
+      - name: Test
+        run: python -m pytest

.gitignore

@@ -0,0 +1,13 @@
+.venv/
+venv/
+myenv/
+__pycache__/
+*.py[cod]
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+build/
+dist/
+.flavor-data-cache/