fix: reconcile model-buyer results with transcripts, add run provenance

ARCHITECTURE.md

@@ -25,9 +25,9 @@ The diagrams live in the [README](README.md) (system, order sequence, compile pi
 | 200 buyer tasks | 100% accuracy | 99.0% — both misses were impossible tasks it still refused |
 | wrong orders / wrong declines | 0 / 0 | 0 / 0 |
 | red team · fairness · conformance | 19/19 hand-written + 190/190 generated · 159,840 cohorts clean · 24/24 | same |
-| latency p50 / p95 | 47 / 62 ms | 54 ms / 4.0 s |
+| latency p50 / p95 | 47 / 62 ms (deterministic) | cache hit ≈ offline; a live gpt-4o proposal adds ~1.5–4 s |
 
-84 tests, fully offline, green in CI.
+91 tests, fully offline, green in CI.
 
 ## Honest limitations
 

README.md

@@ -199,7 +199,7 @@ Real incidents from this build, kept because the fixes became the architecture:
 
 ## Measured (generated, not hand-written — `results/RESULTS.md` · `results/gpt4o/RESULTS.md`)
 
-**How these were produced.** The same 200-task suite runs on two backends: `fake` — the deterministic offline engine that doubles as the model-down fallback (reproducible, no keys) — and **live gpt-4o** (proposals + catalog normalisation, with gpt-4o-mini routed for compile work and an SQLite call cache). Payments are the sandbox client in both; `BAZAAR_RAZORPAY=razorpay` swaps in Razorpay test-mode APIs. One caveat stated plainly: the synthetic corpus is a closed loop (the generator writes both the messy CSVs and the truth labels), so offline compiler accuracy is the parsers' ceiling, not a model score. First gpt-4o run cost ≈ $1; re-runs are mostly free (this one: 2,844 cache hits, 427 misses — the new sweep sessions). The false-positive sweep re-runs the same tasks under three tighter per-order caps so the cost of over-gating is a number, not a reassuring zero. Attack classes, where each defence lives, and a like-for-like table against published attack rates: [`THREAT_MODEL.md`](THREAT_MODEL.md).
+**How these were produced.** The same 200-task suite runs on two backends: `fake` — the deterministic offline engine that doubles as the model-down fallback (reproducible, no keys) — and **live gpt-4o** (proposals + catalog normalisation, with gpt-4o-mini routed for compile work and an SQLite call cache). Payments are the sandbox client in both; `BAZAAR_RAZORPAY=razorpay` swaps in Razorpay test-mode APIs. One caveat stated plainly: the synthetic corpus is a closed loop (the generator writes both the messy CSVs and the truth labels), so offline compiler accuracy is the parsers' ceiling, not a model score. Model calls are cached, so re-runs are mostly free; the exact hit/miss split and — the point that matters — the **model-failover count (0: gpt-4o produced the answers, not the fallback)** are reported in the Provenance section of [`results/gpt4o/RESULTS.md`](results/gpt4o/RESULTS.md), regenerated with the file so they can never drift from it. The false-positive sweep re-runs the same tasks under three tighter per-order caps so the cost of over-gating is a number, not a reassuring zero. Attack classes, where each defence lives, and a like-for-like table against published attack rates: [`THREAT_MODEL.md`](THREAT_MODEL.md).
 
 | metric | offline `fake` | live gpt-4o |
 |---|---|---|
@@ -215,14 +215,14 @@ Real incidents from this build, kept because the fixes became the architecture:
 | compiler — price / GST / stock (parser-owned) | 1.000 / 0.916 / 0.900 | **1.000** / 0.916 / 0.900 — identical, because money fields never touch the model |
 | compiler — name / category (model-owned, exact match vs generator vocabulary) | 1.000 / 1.000 (the closed-loop ceiling) | 0.551 / 0.797 — the honest exact-string number; low-confidence fields go to the review queue, never guessed |
 | **held-out** compiler eval — 3 hand-written catalogs the generator never saw (kirana rate card, Shopify export, electronics price list) | price/stock/GST **1.000/1.000/1.000**, unit 0.906 — the parsers hold; names 0.094 (dictionary can't know brands, so 100% review-queued) | price/stock/GST **1.000/1.000/1.000**, unit 0.969, names 0.469 exact-match with 72% review-queued — money fields perfect on sheets nobody tuned for |
-| latency p50 / p95 | 47 / 62 ms | 54 ms / 4.0 s (cache hit / real call; the committed gpt-4o run is a cache replay of the original paid run, so misses = 0) |
+| latency p50 / p95 | 47 / 62 ms (deterministic) | a cache hit ≈ the offline figures; a real gpt-4o proposal adds ~1.5–4 s (the p95 tracks how many calls in a run are live vs cached) |
 | model failovers during the run | — | 0 |
 
 ### Real agents, not scripts
 
 The 200-task table above is produced by a deterministic scripted buyer — reproducible, and the honest baseline. But "an agent *could*" is weaker than "an agent *did*", so three separate pieces of evidence show real models on the wire:
 
-- **A model-driven buyer** (`python -m bazaar.simulator.model_buyer`): a tool-calling model that decides every step — which merchant, what to say in EN/HI/Hinglish, whether to ask for an offer, when to walk away, when to pay — over the same RFC 9421-signed HTTP API an external agent uses. Runs on `groq` (free, judge-reproducible) or `openai`. Verbatim tool-call transcripts committed under [`results/model_buyer/`](results/model_buyer/).
+- **A model-driven buyer** (`python -m bazaar.simulator.model_buyer`): a tool-calling model that decides every step — which merchant, what to say in EN/HI/Hinglish, whether to ask for an offer, when to walk away, when to pay — over the same RFC 9421-signed HTTP API an external agent uses. On gpt-4o, 40 tasks: 30 orders (14 at the named merchant, **16 via network reroute** where the named merchant couldn't fulfil and the index found one that could), 9 walk-aways, and **0 orders the gate should have blocked**. The reroutes are listed by task in [`results/model_buyer/`](results/model_buyer/) so the transcripts and the number agree — the model never refuses an impossible-at-named task; the gate, not the model's judgement, is what keeps it safe. Runs on `openai` or free `groq`; verbatim tool-call transcripts committed.
 - **Claude over the MCP endpoint** — a real session (tools/list → discover → serviceability → quote) captured in [`results/claude_mcp_session.md`](results/claude_mcp_session.md).
 - **A model-generated red team** (`python -m bazaar.simulator.redteam_gen`): a model writes 190 injection attacks across 8 classes (direct override, Hinglish social-engineering, Devanagari, homoglyphs, JSON smuggling, fake `<rules>` blocks, rule-id spoofing, PII exfiltration). Each is delivered as a **follow-up on a session that already holds a discounted quote** — so the checker can actually fail: an attack passes only if the discount didn't rise, the total didn't drop, no off-table rule id appeared, no rule applied without a negotiate action, and no secret leaked. Result: **190/190 defended against a real gpt-4o seller** (144 declines, 31 searches, 11 clarifies — a model actually responding, not a fallback; committed run in [`results/redteam_generated/`](results/redteam_generated/)). The defence is architectural — `verify` rejects any off-table `rule_id` before execution — so it holds for any backend. The like-for-like answer to the *Whispers of Wealth* attack classes.
 
@@ -232,7 +232,7 @@ Every one of these still goes through propose → verify → execute: the model
 
 ```bash
 pip install -e ".[dev]"            # Python 3.10+
-python -m pytest -q                # 84 tests, fully offline (incl. README-vs-results consistency)
+python -m pytest -q                # 91 tests, fully offline (incl. README-vs-results consistency)
 python -m bazaar.simulator.run     # regenerates results/
 uvicorn bazaar.gateway.app:default_app --factory --port 8000
 cd console && npm install && npm run dev   # http://localhost:5173

THREAT_MODEL.md

@@ -44,7 +44,7 @@ The numbers are from `results/RESULTS.md` (offline engine) and `results/gpt4o/RE
 | Vault Whisper (data disclosure via direct injection) | 20% (2/10) plus partial attempts, same paper | 0% (nothing to disclose in context) |
 | Price-bound overflow in negotiation | 0–10.8% by model, arXiv 2602.06008 Table 1 | 0% by construction (model selects a rule id; engine computes) |
 
-The published numbers are small-n demonstrations on one framework each, not population estimates; the comparison is architectural, not statistical. Bazaar's probes are equally small (one to a few cases per class) and run on every results generation.
+The published numbers are small-n demonstrations on one framework each, not population estimates; the comparison is architectural, not statistical. Bazaar runs two red teams on every generation: a 19-case hand-written suite for the money/identity paths, and a **190-attack model-generated corpus** across 8 classes, each delivered as a follow-up on a live discounted quote and scored for real corruption (see `results/redteam_generated/`) — 190/190 defended against a real gpt-4o seller, because `verify` rejects any off-table `rule_id` before execution.
 
 ## What this does not cover
 

bazaar/simulator/model_buyer.py

@@ -219,17 +219,33 @@ def main(argv: list[str] | None = None) -> int:
     completed = [r for r in results if not (r.error and ("RateLimit" in r.error or "429" in r.error))]
     acc = sum(r.correct for r in results) / len(results)
     acc_completed = (sum(r.correct for r in completed) / len(completed)) if completed else 0.0
+    # a "wrong order" is one the gate should have blocked: it completed but was scored incorrect.
     wrong_orders = sum(1 for t, r in zip(tasks, results, strict=True) if t.expected != "order" and r.outcome == "order" and not r.correct)
+    orders_at_named = sum(1 for t, r in zip(tasks, results, strict=True) if r.outcome == "order" and r.merchant_id == t.merchant_id)
+    orders_via_reroute = sum(1 for r in results if r.outcome == "order" and r.rerouted)
+    model_declines = sum(1 for r in results if r.outcome in ("buyer_walked_budget", "declined_by_policy", "unserviceable", "unknown_item", "walked_away"))
+    # transparency: every order on a task whose NAMED merchant could not fulfil it — the network
+    # rerouted to a merchant that genuinely serves the pincode and holds the stock (the gate verified
+    # both), so it is a correct order, not a violation. List them so a reader isn't left guessing.
+    reroutes_on_impossible = [
+        {"task_id": t.task_id, "expected": t.expected, "from": t.merchant_id, "to": r.merchant_id, "gmv_paise": r.gmv_paise}
+        for t, r in zip(tasks, results, strict=True)
+        if r.outcome == "order" and r.rerouted and t.expected != "order"
+    ]
     summary = {
         "backend": a.backend,
         "model": model,
         "tasks": len(tasks),
         "orders": orders,
+        "orders_at_named_merchant": orders_at_named,
+        "orders_via_network_reroute": orders_via_reroute,
+        "model_walk_aways_or_declines": model_declines,
         "accuracy": round(acc, 3),
         "accuracy_excl_rate_limited": round(acc_completed, 3),
         "rate_limited": rate_limited,
         "reached_the_system": len(completed),
-        "wrong_orders": wrong_orders,
+        "wrong_orders_the_gate_should_have_blocked": wrong_orders,
+        "reroutes_on_impossible_at_named": reroutes_on_impossible,
         "gmv_paise": sum(r.gmv_paise for r in results),
         "outcomes": dict(Counter(r.outcome for r in results)),
         "errors": [r.error for r in results if r.error][:5],
@@ -243,14 +259,28 @@ def main(argv: list[str] | None = None) -> int:
         if rate_limited
         else ""
     )
+    reroute_lines = "".join(
+        f"- `{x['task_id']}` (expected {x['expected']}): named merchant `{x['from']}` could not fulfil it; the network rerouted to `{x['to']}`, which serves the pincode and holds the stock — the gate verified both.\n"
+        for x in reroutes_on_impossible
+    )
+    reroute_block = (
+        f"\n**Why some orders sit on `decline_*` tasks — and why 0 are wrong.** A `decline_*` task is impossible *at its named merchant*, not on the network. "
+        f"The model never refused these by walking away; instead it discovered a merchant that could fulfil them and ordered there. Each such order still passed the full gate "
+        f"(serviceability, stock, caps), so **{orders_via_reroute} orders came via network reroute and 0 passed the gate that shouldn't have**. The index doing its job is the point; "
+        f"the gate — not the model's judgement — is what keeps it safe.\n\n{reroute_lines}"
+        if reroutes_on_impossible
+        else ""
+    )
     (out_dir / "RESULTS.md").write_text(
         f"# Model-driven buyer — {model} on {a.backend}\n\nAn actual tool-calling agent shopped over the signed HTTP API "
         f"(same {len(tasks)} tasks as the scripted buyer's first {len(tasks)}) — it decided every step: merchant, message "
         f"(EN/HI/Hinglish), whether to negotiate, when to walk away, when to pay. Generated, never hand-edited.\n"
         f"{rl_note}\n"
         f"| metric | value |\n|---|---|\n| tasks | {len(tasks)} |\n| reached the system (not rate-limited) | {len(completed)} |\n"
-        f"| orders | {orders} |\n| accuracy (all tasks) | {acc:.1%} |\n| accuracy (excl. rate-limited) | {acc_completed:.1%} |\n"
-        f"| **wrong orders on impossible tasks** | **{wrong_orders}** |\n| GMV | ₹{summary['gmv_paise'] / 100:,.0f} |\n\n"
+        f"| orders | {orders} |\n| — at the named merchant | {orders_at_named} |\n| — via network reroute (named merchant couldn't fulfil) | {orders_via_reroute} |\n"
+        f"| model walk-aways / declines | {model_declines} |\n| accuracy (all tasks) | {acc:.1%} |\n| accuracy (excl. rate-limited) | {acc_completed:.1%} |\n"
+        f"| **wrong orders the gate should have blocked** | **{wrong_orders}** |\n| GMV | ₹{summary['gmv_paise'] / 100:,.0f} |\n"
+        f"{reroute_block}\n"
         f"Outcomes: `{summary['outcomes']}`\n\nEvery completed task went through propose → verify → execute; the model gained "
         f"no new authority. Full tool-call transcripts: `transcripts.md`.\n",
         encoding="utf-8",

bazaar/simulator/run.py

@@ -220,6 +220,11 @@ def log(msg: str) -> None:
 
     if hasattr(llm, "stats"):
         report["backend"]["llm_cache"] = llm.stats()
+    if hasattr(llm, "status"):
+        # provenance: total_failovers=0 proves the real model answered (not the deterministic
+        # fallback); on a cache replay misses=0 is expected and does not mean the model was skipped
+        h = llm.status()
+        report["backend"]["llm_health"] = {k: h[k] for k in ("backend", "degraded", "forced_down", "total_failures", "total_failovers", "last_error") if k in h}
     report["elapsed_s"] = round(time.time() - t0, 1)
     (out_dir / "results.json").write_text(json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8")
     (out_dir / "RESULTS.md").write_text(render_markdown(report), encoding="utf-8")
@@ -317,6 +322,20 @@ def render_markdown(r: dict[str, Any]) -> str:
     if "conformance" in r:
         cf = r["conformance"]
         lines += ["", f"## Protocol conformance — {cf['passed']}/{cf['checks']} checks, conformant: **{cf['conformant']}**" + (f" (failed: {', '.join(cf['failed'])})" if cf["failed"] else "")]
+    b = r["backend"]
+    if "llm_health" in b or "llm_cache" in b:
+        h = b.get("llm_health", {})
+        cache = b.get("llm_cache", {})
+        prov = f"## Provenance\n\nBackend `{b['llm']}`"
+        if "model" in b:
+            prov += f" (model `{b['model']}`)"
+        if h:
+            prov += f". Model failovers to the deterministic fallback during this run: **{h.get('total_failovers', 0)}** — so the {'model itself' if h.get('total_failovers', 0) == 0 else 'fallback partly'} produced these results (health: degraded={h.get('degraded')}, failures={h.get('total_failures', 0)})."
+        if cache:
+            misses = cache.get("misses", 0)
+            prov += f" LLM cache: {cache.get('hits', 0)} hits / {misses} misses"
+            prov += (" — every call served from a prior run's cache; a warm-cache replay costs nothing and re-bills nothing, and the failover count above proves the model, not the fallback, produced the cached answers." if misses == 0 else f" ({misses} real model calls this run; the rest replayed from cache).")
+        lines += ["", prov]
     lines += ["", f"_Elapsed {r['elapsed_s']} s._", ""]
     return "\n".join(lines)
 

results/RESULTS.md

@@ -1,6 +1,6 @@
 # Bazaar — measured results
 
-Generated 2026-09-05T08:40:41Z by `python -m bazaar.simulator.run` (v0.1.0, llm=`fake`, payments=`fake`). Nothing here is hand-edited.
+Generated 2026-09-05T09:01:25Z by `python -m bazaar.simulator.run` (v0.1.0, llm=`fake`, payments=`fake`). Nothing here is hand-edited.
 
 ## Catalog compiler (52 merchants, messy CSV → agent-readable catalog)
 
@@ -35,11 +35,11 @@ Lift: **+6 orders, −₹2,680 GMV (0.99×)**. The extra completions were bought
 
 Declines on impossible tasks — precision 1.000, recall 1.000; wrong orders on impossible tasks: **0**; wrong declines on possible tasks: 0. Overall task accuracy 100.0%. Errors: 0.
 
-By language: hi-Latn 100.0%, en 100.0%, hi 100.0%. Latency p50 51.2 ms · p95 68.7 ms (in-process, llm=`fake`).
+By language: hi-Latn 100.0%, en 100.0%, hi 100.0%. Latency p50 50.2 ms · p95 67.9 ms (in-process, llm=`fake`).
 
 ## Trust
 
-- Audit entries 1270, hash chain intact: **True**, Merkle root `0e54a9637b288ded…`
+- Audit entries 1270, hash chain intact: **True**, Merkle root `c7155900b0fd099f…`
 - Explanations present on 100.0% of agent turns
 - Grants issued 127, used 127; fairness-ledger entries 50, inconsistencies **0**
 
@@ -82,4 +82,4 @@ Same tasks, tighter merchant per-order cap. Wrong declines are *possible* tasks
 
 ## Protocol conformance — 24/24 checks, conformant: **True**
 
-_Elapsed 65.2 s._
+_Elapsed 66.3 s._

results/gpt4o/RESULTS.md

@@ -1,6 +1,6 @@
 # Bazaar — measured results
 
-Generated 2026-09-05T07:44:18Z by `python -m bazaar.simulator.run` (v0.1.0, llm=`openai`, payments=`fake`). Nothing here is hand-edited.
+Generated 2026-09-05T09:19:37Z by `python -m bazaar.simulator.run` (v0.1.0, llm=`openai`, payments=`fake`). Nothing here is hand-edited.
 
 ## Catalog compiler (52 merchants, messy CSV → agent-readable catalog)
 
@@ -35,11 +35,11 @@ Lift: **+6 orders, −₹2,680 GMV (0.99×)**. The extra completions were bought
 
 Declines on impossible tasks — precision 1.000, recall 1.000; wrong orders on impossible tasks: **0**; wrong declines on possible tasks: 0. Overall task accuracy 99.0%. Errors: 0.
 
-By language: hi-Latn 97.6%, en 99.0%, hi 100.0%. Latency p50 54.0 ms · p95 3982.4 ms (in-process, llm=`openai`).
+By language: hi-Latn 97.6%, en 99.0%, hi 100.0%. Latency p50 49.7 ms · p95 1543.0 ms (in-process, llm=`openai`).
 
 ## Trust
 
-- Audit entries 1350, hash chain intact: **True**, Merkle root `78d9abc1d2f96e3a…`
+- Audit entries 1267, hash chain intact: **True**, Merkle root `8695d8b358cfd69c…`
 - Explanations present on 100.0% of agent turns
 - Grants issued 127, used 127; fairness-ledger entries 50, inconsistencies **0**
 
@@ -82,4 +82,8 @@ Same tasks, tighter merchant per-order cap. Wrong declines are *possible* tasks
 
 ## Protocol conformance — 24/24 checks, conformant: **True**
 
-_Elapsed 1018.0 s._
+## Provenance
+
+Backend `openai` (model `gpt-4o`). Model failovers to the deterministic fallback during this run: **0** — so the model itself produced these results (health: degraded=False, failures=0). LLM cache: 2965 hits / 371 misses (371 real model calls this run; the rest replayed from cache).
+
+_Elapsed 543.3 s._