Quarry Materials Extraction

Given a coordinate + radius, discover quarries in that area and return a structured, grounded record per quarry answering "what quarries exist there, and are they still operating?" — from messy, stale, contradictory public web sources.

Design principle throughout: confident extraction from a stale source is worse than no extraction. Every field is grounded in a verbatim quote or it abstains.

Run it

docker compose up --build          # api + ui together → open http://localhost:8080
# UI needs no API key to browse; extraction needs one — put it in .env (see .env.example):
#   ANTHROPIC_API_KEY=sk-ant-...
# Optional, for the offline satellite demo: QUARRY_SATELLITE_MOCK=true docker compose up --build

…or the local dev toolchain (uv):

make bootstrap        # uv sync + create .env from .env.example
make serve  &  make ui               # API :8000 + React dev UI :5173  (open :5173)
make extract LAT=45.7640 LON=4.8357 RADIUS_KM=40   # CLI → schema-valid JSON on stdout
make test             # 76 offline regression tests
make eval             # offline leave-one-out evaluation of the status scorer

The stack is fully in-memory / in-process — no Postgres, Redis, or queue to run.

---

Architecture — components & data flow

        ┌──────────────────────┐        REST + SSE       ┌──────────────────────┐
        │  React UI (Vite)      │◀───────────────────────│  FastAPI service      │
        │  nginx :8080 → /api ──┼────────────────────────▶  (:8000)              │
        │  run · table · drawer │                        │  in-memory store +    │
        └──────────────────────┘                         │  asyncio job "queue"  │
                                                          └──────────┬───────────┘
                                                                     │ run_extraction(coord, radius)
                                     per site, bounded-parallel      ▼
   ┌───────────────────────────────────────────────────────────────────────────────────────┐
   │  pipeline._process_site                                                                  │
   │                                                                                          │
   │  discover ──▶ find URLs ──▶ polite crawl ──▶ LLM extract ──▶ enrich signals ──▶ features │
   │  (Overpass)   (web_search)  robots·SSRF·      (Claude,        Géorisques ICPE ·  (27-dim) │
   │                             Retry-After·      grounded per-   satellite · TLS/MX          │
   │                             jitter·relevance  field quotes)                     ┌─────────▼──────┐
   │                             gate)                                               │ RandomForest   │
   │                                                                                 │ → P(active)    │
   │                                                                                 └───────┬────────┘
   │  QuarryRecord ◀── assemble ◀── decide_status (threshold + ground in a dated quote, else abstain)  │
   └───────────────────────────────────────────────────────────────────────────────────────┘
                                    │
                     validates against schema/effective.schema.json

• Deterministic orchestration, LLM only for extraction. The pipeline (not the LLM) does discovery, crawling, and scoring; web_search is used only to find URLs, then our own polite crawler fetches them — so grounding offsets index stored text and cost is predictable.
• Two evaluation tracks. Activity (operational_status) is a fitted RandomForest; every other field is LLM extraction gated by grounding + reconciliation (see Trade-offs).
• Ground-first (anti-hallucination). The LLM returns a verbatim quote per value; we locate its char offsets in the stored source (str.find → whitespace-map → high-cutoff rapidfuzz) and reject any quote we can't find → that field abstains. Non-web signals (regulatory / satellite / liveness) are registered as synthetic sources so every piece of evidence — including operational_status — carries real offsets.
• Parallel + polite. Sites run concurrently (bounded semaphore); each site's URLs fetch concurrently; the crawler honors robots.txt, sets a real User-Agent, respects Retry-After, adds jitter, validates URLs (SSRF) and never follows redirects into private networks.

The UI is an evaluation console: a run bar (optional USD budget shown spent-vs-budget) + an SSE progress strip; a results table backed by /api/sites (status/search filters, pagination, sort); a summary drawer (verdict + P(active) + a plain-language LLM "why" with the grounded quote beneath, badged 🛰 when satellite is the deciding signal, + key facts); and a "View details" page — every schema field with where it came from (quote + offsets + source tier), full provenance, the 27-feature vector with a description per value, metrics, run metadata, and "Export JSON" (the exact schema record).

---

Trade-offs

The interesting decisions, and what we gave up for them:

• We model activity only; the rest is grounded extraction. The brief's real question is "are they still operating?", so the one learned decision is operational_status: features → RandomForest → calibrated P(active) → decide_status (threshold + ground the label in a dated proof-of-life quote, else abstain). The descriptive fields (name, type, materials, certifications, description, location) are not modelled — they're extraction. We spent the ML/calibration/abstention machinery where the question is, and kept the rest simple.
• The other fields carry the LLM's self-reported confidence, not a trained score. They come from the extraction agent → grounding (the quote must be locatable, or the field abstains) → reconciliation (trust-tier winner, recorded in provenance.reconciliations). Trade-off: those confidences aren't calibrated the way P(active) is — but grounding + first-class abstention keep them honest (a value needs a real quote; ungroundable ⇒ dropped). We chose verifiability over a second calibrated model.
• Extraction is agentic — tools on by default. The extraction agent can call fetch_page (pull an extra page when the pre-fetched sources are thin) and geocode (resolve a place name), bounded by agent_request_limit (model requests) + max_extra_fetches. Trade-off: better recall and self-directed evidence-gathering vs. higher, less-predictable per-site cost/latency than one structured call. Set QUARRY_ENABLE_AGENT_TOOLS=false for the cheap, deterministic single-call path (what the test suite uses). Tool-fetched pages keep their own source_id and are folded into the provenance registry so their quotes still ground.
• Satellite is an offline mock by default, and deliberately a weak corroborator. The real Sentinel-2 path needs the GDAL-heavy geo extra + network, and high-res vehicle detection is a paid tile provider — too heavy for a demo. So QUARRY_SATELLITE_MOCK=1 gives a deterministic, no-network stand-in. Crucially a synthetic signal must never manufacture confidence: the mock never sets the strong scorer features and stays silent unless the other evidence already leans active, then only adds one low-weight, [mock]-labelled corroboration. (Earlier it set confirms_quarry=1 in the neutral case and, since that's a near-perfect Active predictor in the tiny training set, single-handedly produced P(active)≈0.98 on sites with no real evidence — fixed.) Status grounding also prefers a real dated quote over the mock, so an Active verdict is never receipted by the synthetic signal when real evidence exists. The real signal is a drop-in when geo is installed.
• Training data is small and proxy-labelled. 24 hand-adjudicated Lyon-area rows (including a curated registry-conflict slice — "registry says open, physically closed") + a committed ~321-row France-wide snapshot auto-labelled from Géorisques état / OSM lifecycle. Trade-offs: small n; France proxy labels (title validity ≠ verified physical status); Géorisques skews to operating sites so Active is capped ~2:1 for balance; and the registry features double as a label source → leakage risk, which make eval measures via a registry-masked ablation. Curated + conflict rows are up-weighted 3× so the volume of easy registry-agreement rows can't drown the hard lessons.
• What we measure is self-consistency, not field accuracy — and an honest field number is expensive. The bulk labels are derived from the same features the RF reads (registry état, OSM lifecycle), so cross-validation on the training set is near-perfect by construction — it shows the model recovers the labelling rule, not that it's right on the ground. The trustworthy metric is the curated leave-one-out (make eval, n=20). We tried to get a real field number with an independent test set — make testset (quarry build-testset) samples held-out named quarries and labels each with a Claude web_search verdict that sees only the name + coordinates, never the RF's features. It proved best-effort, not a committed metric: each verdict is a ~2-minute web_search (credit-heavy), it returns Unknown for the many quarries with no real web presence, and it's sensitive to search depth and name collisions (the famous historic Villebois quarries vs. the active Perrin operation there). A clean n≈40 field set didn't materialise within budget, so we ship the infra + the honest caveat rather than an inflated number. A real field metric needs manual/expert adjudication or high-res satellite review (see next steps).
• Off-topic web results are dropped, not shown. web_search for a weak/absent name drifts (a page about a singer once slipped in — in French "carrière" also means "career"), so a page that isn't the operator's own domain must contain unambiguous quarrying vocabulary (granulats, gravière, "exploitation de carrière"…) or it's never registered as a source. Trade-off: precision over recall on sources — a legitimate but oddly-worded page could be dropped.
• Extraction favours recall + folds in the page title. Precision-first boilerplate stripping dropped the <title>/headings where operator names live (one page kept 530 of 27,936 chars, so the name never grounded). We switched to recall + prepend the title/meta. Trade-off: noisier input text, mitigated by the relevance gate + grounding.
• In-memory, and cost is reported not enforced. No Postgres/Redis/queue — trivial to run, not durable across restarts. max_usd_cost is accepted and shown as spent-vs-budget but not hard-enforced (the old guard was dead code; a real one needs mid-run projection with cooperative cancellation of in-flight sites).
• Schema: we ship the verbatim spec and an effective schema. The spec's groundedEnum constrains value to ["Quarry", null] and operational_status references it — read literally a status could only be Quarry/null (an oversight). Records validate against schema/effective.schema.json (status ∈ Active/Inactive/Unknown/null); the CLI warns when a record would deviate from the verbatim schema/quarry.schema.json.
• Hardcoded, France-tuned heuristics & keyword lists. Several decisions are rule-based rather than learned — fast to build and inspectable, but brittle outside France/French and requiring manual upkeep. The notable ones (all easy to find and extend):
  • Quarry-relevance vocabulary — _QUARRY_TERMS regex (crawl/extract.py): granulats, gravière, sablière, ballastière, concassé, "exploitation de carrière", quarry, aggregates… (bare "carrière" excluded — it also means career in French).
  • Proof-of-life keywords — _ACTIVE_KW / _INACTIVE_KW (features/build.py): "en activité / en exploitation / producteur…" vs "ancienne carrière / fermée / réaménagée…".
  • Trust-tier domain lists — DIRECTORY_DOMAINS / NEWS_DOMAINS / NEWS_PATH_HINTS (crawl/extract.py) (pagesjaunes, societe.fr, kompass, …); the web_search prompt also names those directories.
  • Decision thresholds & signal weights — status_active_threshold=0.55 / status_inactive_threshold=0.45 (config.py); proof-of-life weights (web 0.6-0.7, ICPE, satellite mock 0.2), NO_SIGNAL_AGE_DAYS sentinel, real-Sentinel NDVI cut 0.35 (imagery/satellite.py).
  • Training heuristics — France REGION_CENTERS, Géorisques carriere/rubrique-2510 filter, Active-class 2:1 cap, curated/conflict 3× up-weights (scoring/dataset.py, scoring/training.py); default Lyon coordinate (config.py).
  • Models — extraction_model=claude-sonnet-5, explanation_model=claude-haiku-4-5 (config.py); the extraction SYSTEM_PROMPT (agent/extractor.py).

---

Proof of life

The dominant scorer signal is the recency + polarity of dated activity/closure evidence:

• French regulatory: Géorisques ICPE (en fonctionnement vs cessation + dates) — a positive and negative oracle.
• Web (dated): news dates, "mis à jour 2024", copyright year, HTTP Last-Modified, sitemap lastmod.
• Satellite (mock by default; real needs the geo extra): a weak corroboration only when other evidence already leans active — [mock]-labelled, never a decider (see Trade-offs).
• Infra liveness: TLS notBefore / MX — restricted to the operator's own domain.
• Phone: lookup-only (validity) — no automated calls (FR ePrivacy/GDPR/Bloctel).

---

API surface

Method	Path	Purpose
POST	/api/jobs	{latitude, longitude, radius_km, max_usd_cost?} → {job_id}. max_usd_cost is advisory (shown spent-vs-budget; reported, not hard-enforced). Optional extension: max_sites? caps the random per-job sample (default QUARRY_MAX_SITES_PER_JOB).
GET	/api/jobs/:id	state, progress, results summary
GET	/api/jobs/:id/events	SSE live per-stage progress (extension)
GET	/api/sites	list with q, status filters + pagination (drives the console table)
GET	/api/sites/:id	full grounded record + provenance (+ a ui sidecar: features, score, explanation, osm_name — stripped on Export)
GET	/api/health	queue depth, worker count, recent error rate, model, scorer

---

Scoring & training data

data/ground_truth/fr_quarries.csv — 24 hand-adjudicated Lyon-area rows (Active / Inactive / Unknown) incl. the registry-conflict slice. make dataset mines a France-wide snapshot into fr_quarries_bulk.csv (~321 rows, auto-labelled by ICPE état / OSM lifecycle: operating → Active, disused/abandoned → Inactive, registry-valid-but-OSM-disused → real conflict rows). Both CSVs are committed and merged at fit time (curated + conflict rows up-weighted); regeneration is a network one-off (Overpass rate-limits → retries with backoff), so training + tests stay offline. QUARRY_SCORER_USE_REGISTRY_FEATURES=false masks the registry features to measure label leakage.

---

Evaluation

make eval (quarry eval) runs a fully-offline leave-one-out cross-validation of the RandomForest on the bundled ground truth and prints a metrics table + a plain-English reading:

• accuracy + macro-F1 at the decision thresholds, and the abstention rate;
• calibration — Brier, 10-bin ECE, log-loss (are the probabilities honest?);
• the registry-conflict hard slice (registry disagrees with reality);
• a registry-masked ablation — how much the model leans on Géorisques (also a label source);
• an abstention diagnostic on the Unknown rows.

It ends with caveats (tiny n, France proxy labels) — treat the numbers as directional. tests/test_eval.py runs the same eval in the offline suite, so the deliverable can't rot.

Note these are self-consistency metrics: the bulk labels are derived from the RF's own features, so they show the model recovers the labelling rule, not field accuracy. For an independent check, make testset (quarry build-testset) builds a held-out set labelled by a name+coords-only web_search oracle and prints RF-vs-oracle accuracy — but it's best-effort (slow/credit-heavy web_search, Unknown for quarries without a web footprint), not a committed number. See Trade-offs / Known limitations.

---

Repository layout

src/quarry/
  models.py schema_validate.py signals.py config.py dates.py pipeline.py cli.py eval.py
  discovery/{overpass,websearch}.py     crawl/{ssrf,robots,fetcher,extract}.py
  agent/{extractor,grounding,reconcile,explain}.py
  sources/{regulatory,liveness}.py      imagery/{satellite,vision,mock}.py
  features/build.py   scoring/{baseline,factory,training,dataset}.py   metrics/cost.py
  service/{api,runner,queue,store}.py   (in-memory, in-process)
schema/{quarry,effective}.schema.json   data/ground_truth/{fr_quarries,fr_quarries_bulk}.csv
tests/ (76 tests)   ui/ (React + Vite: run bar · results table · detail drawer + page)
Makefile  docker-compose.yml (api + ui)  Dockerfile.backend  ui/Dockerfile  .env.example

---

Verification

make test        # 76 offline tests: schema, grounding, SSRF, robots, reconcile, RF scoring,
                 #   eval, satellite mock, web-relevance gate, discovery, pipeline, API
make eval        # offline LOO scorer metrics (accuracy, calibration, ablation)
docker compose up --build   # packaged api + ui on :8080

Without an API key the pipeline still runs (discovery + crawl + Géorisques + RF scoring); the LLM-extracted fields abstain. With ANTHROPIC_API_KEY set, the agent scrapes + grounds and web_search finds operator/directory/news URLs. make test is fully offline.

---

Known limitations & next steps

• In-memory store/runner — results aren't durable across restarts (deliberate; the earlier Postgres/Redis/arq design is in git history if persistence is needed).
• Discovery is best-effort. Happy path: a quarry with an operator or directory page → web_search finds it → the crawler + agent extract the full record (e.g. Granulats Vicat Niévroz → materials [Granulats, sables, cailloux et graviers]; Bonnefoy Béton → [calcaire, grave naturelle, ardoise, …]). Thin path: a small quarry with no findable web presence extracts only registry + satellite, and the descriptive fields honestly abstain — that's the design, not a bug. (web_search itself was silently timing out under the shared 40 s LLM budget, yielding zero URLs; it now has its own web_search_timeout_s, which is what unblocked the happy path above.) Next: use OSM operator/contact:website tags directly, and follow in-page links to the operator's materials/certifications sub-pages.
• Overpass public endpoints are rate-limited/flaky — retry if discovery errors; next: cache/self-host.
• Training set is small and France-proxy-labelled, and the reported numbers are self-consistency, not field accuracy — the bulk labels are derived from the RF's own features, so training-set CV is near-perfect by construction. Attempted field check: make testset builds an independent held-out set labelled by a name+coords-only web_search oracle. Happy path: a well-documented commercial quarry → the oracle finds operator/news pages and returns Active/Inactive with a quote. Thin path: obscure or unnamed quarries have no web footprint → the oracle honestly returns Unknown; combined with ~2-min/credit-heavy calls, it didn't yield a committed n≈40 field metric — so it stays best-effort, not a shipped number. Next: manual/expert verification, a verified-status set (e.g. US MSHA), or high-res satellite adjudication. make eval reports the current (small-n, curated leave-one-out) numbers.
• Satellite real signal is optional/heavy (geo extra; Sentinel-2 ~10 m can't resolve vehicles; high-res is paid) — hence the offline weak-corroborator mock for demos/CI.
• Cost is reported per record, not enforced.