Repository navigation
feat(demo): add honest provider-backed booth flow - #176
Merged
Merged
Conversation
Add prepare and present modes for the Clawdlinux summit demo. The flow uses real OpenAI routing through LiteLLM, separately checks Anthropic, records genuine token cost, proves webhook mutation without claiming local runsc execution, shows NetworkPolicy presence without claiming packet enforcement, and verifies a clearly labeled prior-run HMAC audit artifact offline. Add safe ignored-.env credential parsing, Kubernetes Secret handoff, provider idempotency keys, generation-scoped routing intent, cost deduplication, cert-manager preparation, focused smoke and Helm tests, and current-versus-target demo documentation. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a “booth-ready” demo flow that can prepare a kind cluster and then present a short, provider-backed run showing real LiteLLM/OpenAI routing evidence (tokens + nonzero cost), with clearly labeled boundaries for simulated/config-only items. To support reliability and cost correctness, it also threads a stable operation ID through provider calls and cost reporting for idempotency/deduplication.
Changes:
- Introduces
--prepare/--present/--tamper-auditbooth flow, plus smoke tests validating evidence/safety claims. - Adds generation-scoped operation IDs and propagates them through routing → provider (
Idempotency-Key) → finops dedupe. - Updates LiteLLM Helm values/templates/tests + docs/checklists to match the new booth contract and evidence boundaries.
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/smoke/test_demo_booth_cli.sh | New smoke contract for booth CLI/evidence boundaries |
| tests/smoke/run_smoke.sh | Runs the new booth CLI smoke contract early |
| scripts/record-demo.sh | Records --present flow (optionally tamper audit) |
| scripts/demo-booth.sh | Adds prepare/present flow, credential parsing, evidence printing |
| pkg/llm/routing.go | Routes with operationID passed into provider calls |
| pkg/llm/routing_test.go | Updates router tests for new operationID parameter |
| pkg/llm/provider.go | Provider interface change + Idempotency-Key propagation |
| pkg/llm/provider_test.go | New test for Idempotency-Key header propagation |
| pkg/llm/mock_provider.go | Updates mock provider signature for operationID |
| pkg/finops/noop.go | Updates no-op reporter to accept/log operationID |
| pkg/finops/memory_reporter.go | Adds operationID dedupe + LiteLLM alias pricing |
| pkg/finops/memory_reporter_test.go | Tests dedupe behavior and LiteLLM alias pricing |
| pkg/finops/interface.go | CostReporter now requires operationID + dedupe semantics |
| internal/controller/agentworkload_controller.go | Persists operationID, avoids repeat paid calls, records usage synchronously |
| internal/controller/agentworkload_controller_finops_test.go | Adds idempotency/retry/once-per-generation controller tests |
| examples/research-agent.yaml | Booth AgentWorkload manifest pointing at in-cluster LiteLLM |
| docs/PITCH-SCRIPT.md | Updates pitch to distinguish current proof vs target contract |
| config/crd/bases/agentic.clawdlinux.org_agentworkloads.yaml | CRD schema adds modelRoutingOperationID in status |
| config/crd/agentworkload_crd.yaml | CRD variant adds modelRoutingOperationID field |
| cmd/main_test.go | Updates finops test for new RecordUsage signature |
| charts/values.yaml | Adds litellm.anthropicKey docs + existingSecret guidance |
| charts/values.schema.json | Extends schema for anthropicKey/existingSecret/allowPlaintextKeys |
| charts/tests/litellm_existing_secret_test.yaml | Helm unittest coverage for existingSecret + keys |
| charts/charts/litellm/values.yaml | Subchart values add anthropicKey/existingSecret settings |
| charts/charts/litellm/templates/secret.yaml | Adds ANTHROPIC_API_KEY and plaintext-key guardrails |
| charts/charts/litellm/templates/configmap.yaml | Adds clawdlinux OpenAI/Anthropic model aliases |
| charts/charts/litellm/templates/_helpers.tpl | Updates helper docs for expanded secret contents |
| CHANGELOG.md | Notes booth flow + breaking internal operationID API changes |
| api/v1alpha1/agentworkload_types.go | Adds ModelRoutingOperationID to API status struct |
| _staging/booth/rehearsal-checklist.md | Refreshes rehearsal checklist for prepare/present flow |
| _staging/booth/README.md | Documents evidence boundaries + credential loading behavior |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+255
to
+256
| if [[ "${value}" == *$'\n'* || "${value}" == *$'\r'* || "${value}" == *'\' ]]; then | ||
| die "malformed definition for ${key} in ${source_file}:${line_number}" |
| @@ -242,19 +242,22 @@ litellm: | |||
| type: ClusterIP | |||
| port: 4000 | |||
| openaiKey: "" # REQUIRED when litellm.enabled=true (dev only -- prefer existingSecret) | |||
Comment on lines
+43
to
+46
| // Process-local and unbounded by design for the OSS/demo reporter. Production | ||
| // reporters must persist, deduplicate, and expire operation IDs. | ||
| recordedOperations map[string]struct{} // key: deterministic operation ID | ||
| pricing map[string]ModelPricing // key: "provider/model" |
Comment on lines
+13
to
+29
| const operationID = "agentworkload-7-abc123" | ||
| var gotIdempotencyKey string | ||
| server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, request *http.Request) { | ||
| gotIdempotencyKey = request.Header.Get("Idempotency-Key") | ||
| w.Header().Set("Content-Type", "application/json") | ||
| _, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}],"usage":{"prompt_tokens":2,"completion_tokens":1}}`)) | ||
| })) | ||
| t.Cleanup(server.Close) | ||
|
|
||
| provider := NewOpenAICompatibleProvider("test-provider", server.URL, "test-token") | ||
| if _, err := provider.CallModel(context.Background(), operationID, "test-model", "test prompt"); err != nil { | ||
| t.Fatalf("CallModel failed: %v", err) | ||
| } | ||
|
|
||
| if gotIdempotencyKey != operationID { | ||
| t.Fatalf("Idempotency-Key = %q, want %q", gotIdempotencyKey, operationID) | ||
| } |
shreyanshjain7174
added a commit
that referenced
this pull request
Jul 22, 2026
Add prepare and present modes for the Clawdlinux summit demo. The flow uses a real OpenAI-routed call through in-cluster LiteLLM, separately checks Anthropic, records genuine token cost, proves webhook mutation without claiming local runsc execution, shows NetworkPolicy presence without claiming packet enforcement, and verifies a clearly labeled prior-run HMAC audit artifact offline. Add safe ignored-.env credential parsing, Kubernetes Secret handoff, provider idempotency keys, generation-scoped routing intent, cost deduplication, cert-manager preparation, focused smoke and Helm tests, and current-versus-target demo documentation. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a working, honest 5-7 minute Clawdlinux booth flow for a prepared Mac/kind cluster.
Real evidence in
--presentclawdlinux_agent_cost_dollarsand matchingcost-usd-todayannotation.Explicitly limited evidence
SIMULATION / CONFIGURATION PROOF. macOS kind does not executerunsc.Security
.env..envis parsed literally. It is never sourced or evaluated.litellm.existingSecret; keys do not enter Helm values or release history.Reliability changes
Idempotency-Keyupstream.Main files
scripts/demo-booth.sh:--prepare,--present,--tamper-auditexamples/research-agent.yamlBreaking internal Go API changes
Out-of-tree implementations must update:
CostReporter.RecordUsagenow receives an operation ID.Provider.CallModelnow receives an operation ID.Documented in
CHANGELOG.md.Validation
make test-go: passmake test-controller: pass with envtesthelm lint charts: passhelm unittest charts: 16/16 passtests/smoke/test_demo_booth_cli.sh: passgit diff --check: passCredential note
No live provider request was made during development or review. The credentials pasted into chat must be rotated before the real rehearsal.