GenAI Prompt Optimizer — a Claude Code skill

Cut the token/inference cost of any GenAI or agentic app by migrating it from an expensive model to a cheaper one without losing output quality — by building a deep adversarial test suite, capturing the strong model's outputs as a baseline, then iteratively tuning the prompts until a lighter model matches it.

LLM bills explode because apps reach for the biggest model "to be safe". Often a cheaper model would do — if the prompt were a little better. This skill makes that trade‑off empirical and safe: it never assumes the cheap model is good enough, it measures it against the expensive model on a deep, edge‑case‑hunting test suite, and only recommends the switch when the outputs actually collide.

Works on any repo, any language, any provider — OpenAI, Anthropic, Google Gemini, Mistral, DeepSeek, Qwen, Llama (via hosts), OpenRouter, LiteLLM, LangChain, the Vercel AI SDK, or raw HTTP.

---

The idea in one picture

        ┌─ Phase 0  Consent + scope  (warns it's token-intensive; asks before spending)
        │
        ├─ Phase 1  Discover every LLM call site  (detect_calls.py)
        │
        ├─ Phase 2  Serve the app locally + pin one model per run
        │
        ├─ Phase 3  Build a DEEP adversarial suite  (edge-cases taxonomy)
        │
   ┌────┴─ Phase 4  Baseline the ORACLE (expensive)  →  results_oracle.json  = "correct"
   │
   │    ┌─────────────────────────────────────────────────────────┐
   │    │  Phase 5  loop:                                          │
   │    │    run STUDENT (cheap) → diff the gap → fix a PROMPT     │
   │    │    rule → re-verify ≥3× → repeat until cohorts converge  │
   │    └─────────────────────────────────────────────────────────┘
   │
   └─ Phase 6  Confirm parity per cohort → report cost savings → (optionally) promote

Oracle = the current expensive model (defines "correct"; it's the ceiling, not ground truth). Student = the cheaper model you want to ship. The loop closes the gap by improving prompts, asserting on end‑state and meaning — never verbatim text.

---

What makes it trustworthy (not just "swap the model and hope")

• Deep, adversarial tests, not happy‑path. A capability × test‑type matrix (MFT / invariance / directional) across input boundaries, structured‑output traps, unicode/homoglyph attacks, prompt injection, reasoning under distractors, long‑context needles, over‑refusal contrast pairs, tool‑argument population, metamorphic invariance, and false‑premise hallucination traps. Cases are auto‑derived from the app's own prompts and JSON schemas. See references/edge-cases.md.
• A 3‑tier equivalence verdict. Deterministic checks first (schema, enum, numeric tolerance, tool‑arg equality) → a semantic‑similarity gate → reference‑based LLM‑as‑judge only for the residue, using the Factuality A–E rubric so a correct subset or paraphrase passes instead of being failed by exact‑match. Bias‑mitigated (different‑family judge, reasoning‑before‑verdict, calibration). See references/equivalence.md.
• Stochasticity discipline. Every candidate fix is confirmed across ≥3 runs; one pass or fail is treated as noise.
• Per‑cohort gating. The migration is judged on the worst cohort's regression vs the oracle, not a flattering average. Safety/refusal cohorts must show zero regression.
• Prompt‑first, overfitting‑averse fixes. Prefer a single general rule that would also help the oracle, over scenario‑specific hacks. Code is edited only for genuine bugs a prompt can't fix.
• Local and reversible. Runs against a locally served copy. It never deploys, pushes, or touches production; promotion is always the user's explicit call.

This methodology is the generalized, repo‑agnostic version of a workflow proven on a real Gemini‑lite‑vs‑Gemini‑flash migration.

---

Install

Option A — as a Claude Code plugin (recommended)

# in Claude Code
/plugin marketplace add fed3c3sa/claude-code-genai-prompt-optimizer
/plugin install genai-prompt-optimizer@genai-prompt-optimizer

(Replace fed3c3sa/... with wherever you host the repo.)

Option B — as a personal or project skill (no plugin)

# personal (all your projects)
cp -r skills/genai-prompt-optimizer ~/.claude/skills/

# or per-project (committed with the repo)
mkdir -p .claude/skills && cp -r skills/genai-prompt-optimizer .claude/skills/

Either way the skill self‑discovers from its SKILL.md. The helper scripts use only the Python 3 standard library (the judge step needs an API key for whichever judge model you choose — see below).

---

Use it

Just describe the goal in Claude Code — the skill triggers on intent, you don't have to name it:

• "This app uses Claude Opus for everything and the bill is insane — can we move to a cheaper model without wrecking quality?"
• "Optimize the prompts so gpt-5.4-mini works instead of gpt-5.4 for our extraction endpoint."
• "Benchmark Gemini Flash against Gemini Pro on our own endpoints and tell me if we can downgrade."
• "Build a comprehensive edge‑case test suite for our chat endpoint."

The skill will warn you it's token‑intensive, interview you to scope the run (target model, what must never regress, budget, volume), ask you to serve the app locally (it tells you how, based on your codebase), then run the loop and hand back a report with measured savings and a reusable regression suite.

---

Repository layout

claude-code-genai-prompt-optimizer/
├── .claude-plugin/
│   ├── plugin.json            # plugin manifest
│   └── marketplace.json       # marketplace manifest (for /plugin marketplace add)
├── skills/
│   └── genai-prompt-optimizer/
│       ├── SKILL.md           # the skill: phases, cardinal rules, resource map
│       ├── references/        # methodology, edge-cases, equivalence, model-landscape,
│       │                      #   prompt-optimization, serving, diagnosis, interview
│       ├── scripts/           # detect_calls, run_suite, assertions, judge, compare, cost
│       └── assets/            # suite + config + report templates
├── README.md
├── LICENSE                    # Apache-2.0
└── .gitignore

The helper scripts (stdlib‑only Python)

Script	Does	Phase
detect_calls.py	Find every LLM call site across SDKs; flag base_url proxies & Azure deployment names	1
run_suite.py	Send the suite to the local app; record responses, latency, tokens	4–5
assertions.py	Deterministic checks (schema, enum, numeric tolerance, tool‑arg equality, …)	4–5
judge.py	Reference‑based equivalence (Factuality A–E + per‑dimension rubric) over a configurable judge API	5
compare.py	Diff oracle vs student; classify gaps (target / both_fail / parity); per‑cohort deltas	5
cost.py	Measured‑token cost + projected monthly savings, with caching/batch caveats	4–6

Full CLI docs: skills/genai-prompt-optimizer/scripts/README.md.

---

Cost & safety notes

• It spends tokens on purpose. Baseline + every optimization iteration call the app's model endpoints, plus judge calls. The skill estimates and surfaces the cost and waits for your go‑ahead before running. Start with one endpoint.
• Bring a judge key. judge.py calls a strong, different‑family judge model via the API key named in your run config (config.judge.api_key_env). Without it, the deterministic tiers still run and Claude can judge inline using the same rubric.
• Prices in scripts/pricing.json are an approximate ~mid‑2026 snapshot. Model names and prices change constantly — re‑verify at the provider's live pricing page before quoting savings. Sometimes the bigger win is prompt caching or the batch API on the expensive model rather than downgrading at all — the skill flags this.
• No secrets are read or committed by the skill beyond the env vars your config names; everything stays in your working tree until you decide to keep it.

---

License

Apache‑2.0.

Built as a Claude Code skill following Anthropic's Agent Skills authoring guidance. Methodology draws on public work in LLM‑as‑a‑judge evaluation (OpenAI/Braintrust Factuality, promptfoo, G‑Eval), behavioral/adversarial NLP testing (CheckList, OWASP LLM Top 10, metamorphic testing, XSTest, BFCL/ToolScan), and provider prompt‑engineering guides.