feat(demo): add honest provider-backed booth flow

CHANGELOG.md

@@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- Added a working booth prepare/present flow with real provider and cost proof.
+
+### Changed
+- Changed the internal Go `CostReporter.RecordUsage` and `Provider.CallModel` interfaces to carry operation IDs for idempotency. This breaks out-of-tree implementations.
+- Added the optional `litellm.anthropicKey` chart value. Use `litellm.existingSecret` for booth and production deployments.
+
 ## [0.2.0] - 2026-04-25
 
 Production-ready agent orchestration release with multi-agent demos, pluggable workflows, CLI onboarding, and landing page overhaul.

_staging/booth/README.md

@@ -1,28 +1,48 @@
-# Booth staging
+# Booth evidence boundaries
 
-Fallback assets for the Clawdlinux attestation demo. Use these if the live run
-hiccups. Do not present from these by default. The live demo is the demo.
+The booth flow has 6 evidence items. State the boundary before each step.
 
-## attestation-fallback.jsonl
+- **REAL:** The AgentWorkload makes one OpenAI-routed call through in-cluster LiteLLM.
+- **REAL:** Its status reports genuine nonzero tokens. Its cost metric and annotation are nonzero.
+- **REAL, SEPARATE CHECK:** A LiteLLM request checks Anthropic reachability.
+- **SIMULATION / CONFIGURATION PROOF:** A server-side dry-run shows gVisor mutation.
+- **NETWORKPOLICY OBJECT PRESENCE ONLY:** The script lists the NetworkPolicy object.
+- **PRIOR-RUN ARTIFACT:** The checked-in HMAC-signed audit fixture verifies offline.
 
-A real, hash-chained, HMAC-signed audit artifact. 6 entries simulating one
-regulated agent run: manifest emit, LLM call, an allowed egress, a denied
-egress, a human approval, and a state change to Succeeded. Produced with the
-production `pkg/audit` recorder, so it verifies with the shipped `audit-verify`.
+The current AgentWorkload does not generate the audit fixture. Audit recording
+is not wired into reconciliation yet.
+The `--present` path does not execute OPA allow or deny evaluation.
+OPA remains in the legacy default flow and target product contract.
 
-This is genuine evidence, not a mock. It verifies clean and rejects tampering.
+## Credential loading
 
-### Verify it offline (the money shot)
+`scripts/demo-booth.sh --prepare` reads provider keys from exported variables first.
+Missing keys can load from the repo-root `.env` file.
+
+Set `DEMO_ENV_FILE=/path/to/file` to use another file.
+The parser accepts only `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` definitions.
+It supports optional `export` prefixes and quoted values.
+It does not source the file or evaluate its contents.
+
+Preparation prints each variable name as `available` and reports its source.
+It never prints credential values.
+
+## Prior-run audit fixture
+
+`attestation-fallback.jsonl` is a real hash-chained, HMAC-signed artifact.
+It was produced earlier with `pkg/audit`. It contains 6 audit entries.
+
+### Verify it offline
 
 ```bash
 audit-verify --source jsonl \
   --path _staging/booth/attestation-fallback.jsonl \
   --key booth-demo-2026=bmluZXZpZ2lsLWJvb3RoLWRlbW8tYXR0ZXN0YXRpb24ta2V5LTMyYg==
 ```
 
-Expect: `PASS — chain is intact and all checkpoints match.`
+Expect a pass showing the chain is intact.
 
-### Tamper demo (prove it is real evidence)
+### Optional tamper proof
 
 ```bash
 sed 's/"actor":"policy-analyst"/"actor":"ghost-actor"/' \
@@ -31,12 +51,10 @@ audit-verify --source jsonl --path /tmp/tampered.jsonl \
   --key booth-demo-2026=bmluZXZpZ2lsLWJvb3RoLWRlbW8tYXR0ZXN0YXRpb24ta2V5LTMyYg==
 ```
 
-Expect: `FAIL at seq=4: ... entry_hash mismatch`. One altered field and the
-chain breaks. That is what an auditor wants to see.
+Expect a failure at the changed entry. The script removes its temporary file.
 
 ## Notes
 
-- The key here is a demo key, committed on purpose so the fallback is
-  self-contained. Never use this key for a real run.
-- The verify works fully offline. No cluster, no network, no external
-  dependencies. That is the air-gapped review-room story.
+- The committed key is only for this fixture.
+- Never use it for a real workload.
+- Verification works offline without the cluster.

_staging/booth/rehearsal-checklist.md

@@ -1,119 +1,75 @@
 # Booth rehearsal checklist
 
-Cold-run gate for the Clawdlinux attestation demo. Run this 5 times on the actual
-booth laptop, offline, before the summit. Assume venue wifi is dead. Tick every
-gate. If any gate fails twice, the demo is not booth-ready.
-
-Driver script: `scripts/demo-booth.sh` (560 lines). This checklist mirrors its
-phases and adds explicit pass/fail gates the script does not assert for you.
-
-## 0. One-time offline prep (do once, before rehearsals)
-
-- [ ] Laptop has `kubectl`, `helm`, and `kind` (or `k3s`) installed and on PATH.
-- [ ] Seed the kind node-image cache with the same command the demo script uses:
-      `kind create cluster --name clawdlinux-demo --wait 120s`
-      This pulls the exact `kindest/node` tag for your installed `kind` binary.
-      Without this, wifi-off `kind create cluster` fails immediately.
-- [ ] Pre-pull every image the demo needs, then point kind at the local cache so
-      no phase pulls from the network:
-      - [ ] Keep `clawdlinux-demo` running while you load images.
-      - [ ] For each image the chart and samples use: `docker pull <img>` then
-            `kind load docker-image <img> --name clawdlinux-demo`.
-      - [ ] Re-run the demo with wifi OFF. If any phase blocks on ImagePull, the
-            cache is incomplete. Fix before the summit.
-- [ ] `audit-verify` built and on PATH: `go build -o ~/bin/audit-verify ./cmd/audit-verify`.
-- [ ] Fallback artifact staged: `_staging/booth/attestation-fallback.jsonl`
-      (see `_staging/booth/README.md`). Confirm it still verifies offline.
-
-## 1. Prerequisites (script: "Checking prerequisites")
-
-- [ ] PASS gate: script prints "kubectl and helm found" and finds kind or k3s.
-- [ ] FAIL signal: "kind or k3s is required" or a missing manifest. Stop, fix PATH
-      or repo checkout.
-- [ ] Confirm present: `config/samples/agentworkload_demo_{allow,deny,swarm}.yaml`.
-
-## 2. Cluster prep (script: "Preparing Kubernetes cluster")
-
-- [ ] PASS gate: "Created kind cluster clawdlinux-demo" (or "Reusing ...") within
-      120s.
-- [ ] FAIL signal: kind create hangs past 120s. Usually a stale Docker state.
-      Run `scripts/demo-booth.sh --cleanup` and retry.
-
-## 3. Operator ready (script: "Waiting for operator pod")
-
-- [ ] PASS gate: operator pod reaches Running inside the 180s deadline.
-- [ ] FAIL signal: deadline hit. Check `kubectl -n agentic-system describe pod`.
-      Almost always an un-cached image. Go back to step 0.
-
-## 4. Deployment profile (script: "Installing ... profile")
-
-- [ ] Decide before you start: `--profile platform` (Argo and shared services)
-      or `--profile lean` (operator and governance controls only).
-      Rehearse the one you will present.
-- [ ] PASS gate: helm install completes within `HELM_TIMEOUT` (default 180s).
-- [ ] FAIL signal: helm timeout. Raise `HELM_TIMEOUT=300s` only if the laptop is
-      slow; if it is an image pull, fix the cache, do not just raise the timeout.
-
-## 5. Workload + runtime view (script: "AgentWorkload status", "Agent pod runtime view")
-
-- [ ] PASS gate: the demo AgentWorkload shows a populated status; an agent pod is
-      listed with its runtime view.
-
-## 6. Cost metric (script: "Checking cost metric")
-
-- [ ] PASS gate: a per-workload cost metric is reported (non-empty).
-- [ ] This is the FinOps talking point. If empty, say so, do not improvise.
-
-## 7. gVisor sandbox (script: "Checking gVisor RuntimeClass")
-
-- [ ] PASS gate: the labeled pod shows `runtimeClassName: gvisor`.
-
-## 8. Egress seal — the core demo (script: "Checking NetworkPolicy", "OPA allow", "OPA deny")
-
-- [ ] PASS gate (allow): `agentworkload_demo_allow.yaml` runs; egress scoped to
-      the FQDN allowlist.
-- [ ] PASS gate (deny): `agentworkload_demo_deny.yaml` is blocked at the boundary.
-- [ ] Talk track: "default deny, allowlist per workload, enforced at the network
-      layer, not in app code."
-- [ ] FAIL signal: deny case is NOT blocked. This breaks the whole pitch. Do not
-      present until fixed.
-
-## 9. Attestation artifact — the money shot
-
-- [ ] During the run, capture the audit/attestation output to the evidence dir.
-- [ ] Verify it offline in front of the prospect:
-      `audit-verify --source jsonl --path <evidence>.jsonl --key <kid>=<b64>`
-- [ ] PASS gate: "PASS — chain is intact".
-- [ ] Tamper demo (optional, strong): edit one field, re-run audit-verify, show
-      it FAIL with an entry_hash mismatch. Proves the artifact is real evidence.
-- [ ] If the live run hiccups, fall back to `_staging/booth/attestation-fallback.jsonl`
-      and run the same verify + tamper demo. Never debug live on a prospect's time.
-
-## 10. Multi-agent swarm (optional scenario, needs `--with-swarm`)
-
-- [ ] Only if presenting the swarm: run with `--with-swarm` and the platform profile.
-- [ ] PASS gate: 3-agent swarm reaches a terminal state without manual nudging.
+Run this flow 5 times on the booth Mac. The real provider steps require network
+access. Keep the prior-run verifier available when venue network access fails.
+
+Driver script: `scripts/demo-booth.sh`.
+
+## 0. One-time prep
+
+- [ ] Install `kubectl`, `helm`, `kind`, Docker, `curl`, and Go.
+- [ ] Build the current operator image using the configured demo image tag.
+- [ ] Build `bin/agentctl` with `make build-agentctl`.
+- [ ] Build `bin/audit-verify` with `make obs-build-verifier`.
+- [ ] Confirm `_staging/booth/attestation-fallback.jsonl` verifies offline.
+- [ ] Keep the repo-root `.env` ignored and readable only by the booth operator.
+
+## 1. Prepare with real credentials
+
+- [ ] Export both provider keys or define them in the repo-root `.env` file.
+- [ ] Set `DEMO_ENV_FILE` when the credential file lives elsewhere.
+- [ ] Run `scripts/demo-booth.sh --prepare`.
+- [ ] Confirm both variable names report `available` without showing values.
+- [ ] Confirm the reported source is `environment` or the expected file.
+- [ ] Confirm exported `LITELLM_MASTER_KEY` values with LF or CR are rejected.
+- [ ] Confirm context is `kind-clawdlinux-demo`.
+- [ ] Confirm cert-manager, webhook, operator, and LiteLLM become ready.
+- [ ] Confirm the script never prints key values.
+- [ ] Unset exported provider variables after preparation on shared terminals.
+
+## 2. Present the 5-7 minute flow
+
+- [ ] Run `scripts/demo-booth.sh --present`.
+- [ ] **REAL:** AgentWorkload reaches `Completed` after one OpenAI-routed call.
+- [ ] **REAL:** Routing condition shows genuine input and output token counts.
+- [ ] **REAL:** Cost annotation is greater than zero.
+- [ ] **REAL:** `clawdlinux_agent_cost_dollars` is greater than zero.
+- [ ] **REAL:** Separate Anthropic request reports provider reachability.
+- [ ] Do not claim the AgentWorkload used Anthropic.
+- [ ] **SIMULATION / CONFIGURATION PROOF:** Dry-run injects `runtimeClassName=gvisor`.
+- [ ] State that no gVisor pod was scheduled on kind/macOS.
+- [ ] **NETWORKPOLICY OBJECT PRESENCE ONLY:** Show the NetworkPolicy object.
+- [ ] Say packet enforcement requires an enforcing CNI.
+- [ ] Do not claim all runtime pods carry sandbox labels.
+- [ ] State that `--present` does not execute OPA allow or deny evaluation.
+
+## 3. Prior-run audit closer
+
+- [ ] State that the current workload did not create the fixture.
+- [ ] **PRIOR-RUN ARTIFACT:** Verify the checked-in JSONL file offline.
+- [ ] Optionally run `scripts/demo-booth.sh --present --tamper-audit`.
+- [ ] Confirm the altered temporary artifact fails verification.
+- [ ] State that audit recording is not wired into reconciliation yet.
 
 ## 11. Teardown
 
 - [ ] `scripts/demo-booth.sh --cleanup` deletes the kind cluster.
 - [ ] Confirm `kind get clusters` no longer lists `clawdlinux-demo`.
 
-## Rehearsal log (fill this in — 5 cold runs, wifi off)
+## Rehearsal log
 
-| Run | Date | Wifi off? | All gates pass? | First failure (phase) | Time to green | Notes |
-|-----|------|-----------|-----------------|-----------------------|---------------|-------|
-| 1   |      |           |                 |                       |               |       |
-| 2   |      |           |                 |                       |               |       |
-| 3   |      |           |                 |                       |               |       |
-| 4   |      |           |                 |                       |               |       |
-| 5   |      |           |                 |                       |               |       |
+| Run | Date | Network stable? | All gates pass? | First failure | Total time | Notes |
+|-----|------|-----------------|-----------------|---------------|------------|-------|
+| 1   |      |                 |                 |               |            |       |
+| 2   |      |                 |                 |               |            |       |
+| 3   |      |                 |                 |               |            |       |
+| 4   |      |                 |                 |               |            |       |
+| 5   |      |                 |                 |               |            |       |
 
-Booth-ready bar: 5 consecutive cold runs, wifi off, all gates green, under your
-demo time budget. Anything less and you are gambling on the venue.
+Booth-ready bar: 5 consecutive runs under 7 minutes with every truth label spoken.
 
 ## Optional: record a backup
 
-- [ ] `scripts/demo-booth.sh --profile platform --with-swarm --record` captures terminal output with
-      script(1). Keep one clean recording as a last-resort fallback if the laptop
-      itself dies.
+- [ ] `scripts/record-demo.sh` records the present path.
+- [ ] Set `DEMO_TAMPER_AUDIT=true` to include the tamper proof.
+- [ ] Review the recording for accidental key output before sharing it.

api/v1alpha1/agentworkload_types.go

@@ -358,6 +358,11 @@ type AgentWorkloadStatus struct {
 	// +optional
 	WorkflowArtifacts map[string]string `json:"workflowArtifacts,omitempty"`
 
+	// modelRoutingOperationID is the stable idempotency key for the current
+	// workload generation's model call.
+	// +optional
+	ModelRoutingOperationID string `json:"modelRoutingOperationID,omitempty"`
+
 	// conditions represent the current state of the AgentWorkload resource.
 	// Each condition has a unique type and reflects the status of a specific aspect of the resource.
 	//

charts/charts/litellm/templates/_helpers.tpl

@@ -17,7 +17,7 @@ app.kubernetes.io/instance: {{ .Release.Name }}
 {{- end }}
 
 {{/*
-Resolve the Secret name that holds OPENAI_API_KEY / LITELLM_MASTER_KEY.
+Resolve the Secret that holds OPENAI_API_KEY, ANTHROPIC_API_KEY, and LITELLM_MASTER_KEY.
 If .Values.existingSecret is set, use it (BYO secret -- recommended for production).
 Otherwise fall back to the chart-managed Secret (dev/demo only).
 */}}

charts/charts/litellm/templates/configmap.yaml

@@ -11,6 +11,14 @@ data:
       master_key: "os.environ/LITELLM_MASTER_KEY"
       database_url: null
     model_list:
+      - model_name: clawdlinux-openai
+        litellm_params:
+          model: openai/gpt-4o-mini
+          api_key: "os.environ/OPENAI_API_KEY"
+      - model_name: clawdlinux-anthropic
+        litellm_params:
+          model: anthropic/claude-haiku-4-5-20251001
+          api_key: "os.environ/ANTHROPIC_API_KEY"
       - model_name: gpt-4o
         litellm_params:
           model: openai/gpt-4o

charts/charts/litellm/templates/secret.yaml

@@ -5,8 +5,8 @@ Production deployments should set .Values.existingSecret to a Secret pre-created
 plaintext keys never enter `helm get values`, shell history, or the helm release Secret.
 */ -}}
 {{- if not .Values.existingSecret }}
-{{- if and (or .Values.openaiKey .Values.masterKey) (not .Values.allowPlaintextKeys) }}
-{{- fail "A plaintext Helm key value is set (litellm.openaiKey and/or litellm.masterKey). Use litellm.existingSecret to reference a pre-created Secret, or set litellm.allowPlaintextKeys=true to acknowledge the risk (dev/demo only)." }}
+{{- if and (or .Values.openaiKey .Values.anthropicKey .Values.masterKey) (not .Values.allowPlaintextKeys) }}
+{{- fail "A plaintext Helm key value is set (litellm.openaiKey, litellm.anthropicKey, and/or litellm.masterKey). Use litellm.existingSecret to reference a pre-created Secret, or set litellm.allowPlaintextKeys=true to acknowledge the risk (dev/demo only)." }}
 {{- end }}
 apiVersion: v1
 kind: Secret
@@ -18,5 +18,6 @@ metadata:
 type: Opaque
 stringData:
   OPENAI_API_KEY: {{ .Values.openaiKey | default "" | quote }}
+  ANTHROPIC_API_KEY: {{ .Values.anthropicKey | default "" | quote }}
   LITELLM_MASTER_KEY: {{ .Values.masterKey | default (randAlphaNum 32) | quote }}
 {{- end }}

charts/charts/litellm/values.yaml

@@ -14,7 +14,13 @@ resources:
     cpu: "1000m"
     memory: "512Mi"
 openaiKey: ""
+anthropicKey: ""
 masterKey: ""
+# Name of a pre-created Secret. LiteLLM reads OPENAI_API_KEY,
+# ANTHROPIC_API_KEY, and LITELLM_MASTER_KEY from it via envFrom.
+# Recommended for booth and production deployments.
+existingSecret: ""
+allowPlaintextKeys: false
 models: []
 
 # LiteLLM Routing Configuration