Skip to content

feat: Add current OpenAI/Gemini models, newer open models, optional dynamic auto-selection, and *_DISALLOWED_MODELS - #450

Open
rpeck wants to merge 25 commits into
BeehiveInnovations:mainfrom
rpeck:add-current-models-2026-05
Open

feat: Add current OpenAI/Gemini models, newer open models, optional dynamic auto-selection, and *_DISALLOWED_MODELS#450
rpeck wants to merge 25 commits into
BeehiveInnovations:mainfrom
rpeck:add-current-models-2026-05

Conversation

@rpeck

@rpeck rpeck commented May 31, 2026

Copy link
Copy Markdown

Summary

Brings the model catalog up to date (mid-2026), adds a complementary block-list restriction, adds an
opt-in stronger auto-selection, and adds first-party (native) providers for open-model creators.
Six separable concerns:

  1. Model catalog — current OpenAI + Gemini models, plus matching OpenRouter mirrors.
  2. *_DISALLOWED_MODELS block-list — a per-provider block-list complementing the existing
    *_ALLOWED_MODELS allow-lists.
  3. DYNAMIC_MODEL_SELECTION flag — gates a "prefer the newest flagships" auto-selection and the
    bare-alias remaps behind an env var, so the default is unchanged for existing users.
  4. Native Anthropic (Claude) provider — a first-party Messages API backend so Claude models have
    a direct route instead of only OpenRouter. Dormant unless ANTHROPIC_API_KEY is set, so it is
    no change for existing users.
  5. Native direct-API providers for open-model creators — DeepSeek, Qwen, GLM/Z.ai, Kimi/Moonshot,
    MiniMax, Mistral, NVIDIA, Perplexity (all OpenAI-compatible), plus DeepSeek Vision Exp / R1 local
    distills / GLM-5.3 Flash / the Ox Alpha stealth model. Each native provider is dormant unless its
    *_API_KEY is set.
  6. Qwen3.8-Flash + OpenRouter Fusion meta-router — production Qwen3.8-Flash on both routes, plus
    the Fusion council-behind-one-slug with named -quality/-fast preset SKUs, a new auto_selectable
    opt-out flag, and cost/injection seals.

The design below is the result of working through the two review bots' comments, which initially
pulled in opposite directions. This description explains why it's shaped this way so the trade-offs
are visible.


1. Model catalog

Adds current OpenAI models (gpt-5.5, gpt-5.5-pro, gpt-5.4{,-pro,-mini,-nano}, gpt-5.3-codex,
gpt-5.2-codex, gpt-5.1, gpt-5.1-codex-max, gpt-5-pro, the *-chat-latest variants,
o1/o1-pro, gpt-4o{,-mini}, gpt-4.1-{mini,nano}) and Gemini models (gemini-3.5-flash,
gemini-3.1-pro-preview, gemini-3.1-flash-lite, gemini-3-flash-preview, gemini-2.5-flash-lite),
plus matching OpenRouter mirrors. intelligence_scores extend the existing 1–20 scale (frontier
gpt-5.5=20; near-frontier and codex models=19; prior flagship gpt-5.2=18; gemini-3.1-pro-preview=19; etc.).

Also refreshes the open-weight side of the OpenRouter catalogue, which was thin/stale (only a
lone DeepSeek R1, Llama-3-70B, and Mistral-Large). Adds a flagship + a cheaper/smaller tier for each
of: DeepSeek (V4 pro/flash), Qwen (3.7-max / 3.6-flash), Kimi (K2.6/K2.5), GLM (5.1 / 4.7-flash),
MiniMax (M3 / M2.5), Gemma 4 (31b / 26b-a4b), Mistral (Large 3 / Small), Xiaomi MiMo (v2.5 pro/v2.5),
gpt-oss (120b/20b), Llama 4 (Maverick/Scout), NVIDIA Nemotron-3 (super-120b / nano-30b) — 22 entries.
Context windows and capability flags are taken from the live OpenRouter catalogue. Note: OpenRouter
already passes through any provider/model ID, but without a catalogue entry such models fall back
to generic 32K defaults and are invisible to DEFAULT_MODEL=auto / listmodels; these entries give
them correct context budgeting and make them auto-selectable. New aliases are family-scoped and do not
repoint any existing bare alias (deepseek/mistral/llama are unchanged).

Adding models is uncontroversial. The open-model intelligence_scores were calibrated against
current (2026) HuggingFace / benchmark data (LiveCodeBench, SWE-bench, GPQA-Diamond, MMLU-Pro, AIME)
anchored to the existing scale, then peer-reviewed by two independent models; they remain a curated
ordering signal, not a precise ranking.

2. *_DISALLOWED_MODELS block-list

An opt-in per-provider block-list complementing the existing allow-lists. A block-list match takes
precedence over the allow-list; matching is alias-aware (shared with the allow-list path). A model is
usable iff it is not blocked and (no allow-list is set or it is on the allow-list). Fully
backward-compatible — no behavior change when the env vars are unset. Lets you keep a provider
otherwise open while excluding a few models (e.g. costly *-pro tiers) without enumerating every
permitted model.

3. DYNAMIC_MODEL_SELECTION — opt-in stronger auto-selection

This is the part the review feedback reshaped. The original version of this PR changed the default:
auto-mode (DEFAULT_MODEL=auto) would prefer the newest flagships (gpt-5.5,
gemini-3.1-pro-preview/gemini-3.5-flash) for everyone. Two bots flagged this, in tension:

  • chatgpt-codex (P1): defaulting auto-mode to the newest flagship is a regression risk — a key
    without access to e.g. gpt-5.5 would get a model-not-found at call time, and selection only falls
    back on restriction policy, not on an API access error. → argues for a safe default.
  • gemini-code-assist (medium ×3): the OpenAI preference lists were missing newly added models
    (pro/codex variants in EXTENDED/BALANCED; mini/nano in FAST). → argues for more complete /
    aggressive
    selection.

Resolution: gate the aggressive selection behind DYNAMIC_MODEL_SELECTION (default off).

  • Default (flag unset): restores upstream selection behavior, so there is no change for existing
    auto users — this closes the codex P1. With the default restored, gemini-code-assist's
    list-completeness concern no longer applies to the default path.
  • Flag on (DYNAMIC_MODEL_SELECTION=1): the stronger selection. The OpenAI flag-on preference
    lists now include the full current catalogue (the codex/pro/mini/nano variants gemini-code-assist
    called out), and the bare aliases (flash/pro/flash-lite) remap to the newest models.

Mechanics: a dynamic_model_selection_enabled() helper in config.py, branched inside each
provider's get_preferred_model; a dynamic_aliases field on ModelCapabilities plus an overlay in
providers/registries/base.py _build_maps for the gated alias remaps.

One important nuance — Gemini's default pick still advances (by design, not a bug)

Worth being explicit because it shows up in the test diff. OpenAI selection uses fixed preference
lists, so the flag-off path is byte-identical to upstream. Gemini selection is pool-driven: the
default find_best is upstream's reverse-alphabetical pick over the available catalogue. The flag only
swaps the ranking function (reverse-alpha → intelligence_score), not the candidate pool. Because
the newly added Gemini models are numbered higher (gemini-3.5-flash > gemini-2.5-flash,
gemini-3.1-pro-preview > gemini-3-pro-preview), reverse-alphabetical naturally selects them even
with the flag off — exactly as upstream's own algorithm advances whenever a newer model is added to the
catalogue.

So the flag-off Gemini auto-pick does move to the newest models, and the default-path Gemini test
expectations move with it (e.g. gemini-2.5-flashgemini-3.5-flash). We chose to keep upstream's
algorithm and document this
, rather than special-case Gemini to exclude catalogue models from
default auto-selection. If you'd prefer the latter, it's an easy follow-up — flag it and I'll gate the
Gemini candidate pool too.

Known limitation (follow-up filed)

Auto-selection filters candidates by restriction policy, not by whether the configured key can actually
call the chosen model. With the flag on (and, for Gemini, by default once newer models are present),
auto-mode can pick a model a key can't access, and there is no runtime API-level fallback to the
next candidate. Workaround today: constrain the pool with *_ALLOWED_MODELS / *_DISALLOWED_MODELS,
or leave the flag unset. Tracked as a follow-up (add runtime fallback on model-not-found).


4. Native Anthropic (Claude) provider

Previously Claude models were only reachable via OpenRouter (anthropic/... IDs). This adds a
first-party provider on the Anthropic Messages API (ProviderType.ANTHROPIC + ANTHROPIC_API_KEY),
so keys can call Claude directly. Because native providers rank ahead of OpenRouter in the priority
order, when an Anthropic key is configured bare aliases like opus/fable/sonnet/haiku resolve
to the direct API; users without a key still reach Fable 5 et al. through the OpenRouter entries.

  • Catalog (conf/anthropic_models.json): claude-opus-5 (score 20), claude-fable-5 (20),
    claude-sonnet-5 (18), claude-haiku-4-5 (12). Scores anchored to the existing 1–20 scale from the
    Artificial Analysis Intelligence Index (Opus 5 ≈ 63, Fable 5 ≈ 62), same methodology as the rest of
    the catalog.
  • Restrictions: honors ANTHROPIC_ALLOWED_MODELS / ANTHROPIC_DISALLOWED_MODELS, consistent with
    the other providers (and with the block-list in §2).
  • Thinking / request shaping: PAL's thinking_mode maps to Anthropic budget_tokens, clamped to
    the API's ≥1024 floor (so minimal disables thinking) and kept strictly < max_tokens; temperature
    is forced to 1.0 while thinking is enabled, as the API requires; max_tokens is always sent (the
    Messages API requires it). get_preferred_model sorts by raw intelligence_score first, because the
    effective capability rank saturates at 100 for every model scoring ≥18 (opus/fable/sonnet), which
    would otherwise let a lexical tie-break pick sonnet over the higher-scored opus.
  • Packaging: anthropic>=0.40.0 added to requirements.txt; the SDK is imported lazily inside
    the client accessor, so the module loads (and the rest of the server runs) even when the SDK isn't
    installed. Registration is gated on ANTHROPIC_API_KEY in server.configure_providers, so there is
    no behavior change unless a key is present.

5. Native direct-API providers (open-model creators) + DeepSeek / Ox Alpha

Extends the native-provider pattern from §4 to every open-model creator that runs a first-party
OpenAI-compatible API, so a creator's key gives a direct route instead of only OpenRouter. Each is a
thin subclass of the shared OpenAI-compatible base (the providers/xai.py pattern): DeepSeek, Qwen
(DashScope), Z.ai (GLM), Moonshot (Kimi), MiniMax, Mistral, NVIDIA (Nemotron), Perplexity
. New
ProviderType members + env-key map + PROVIDER_PRIORITY_ORDER (natives rank ahead of OpenRouter, so a
bare family alias like deepseek/kimi resolves to the direct API when the key is set and falls back to
OpenRouter when it isn't) + *_ALLOWED/DISALLOWED restriction wiring + key-gated registration. Each
provider is dormant unless its *_API_KEY is set
(zero change for existing users). Native conf scores
are pinned to the OpenRouter mirrors, since the same model is now reachable via two routes. Motivation:
for some creators the flagship is API-only or unhostable (GLM-5.3 weights staged; Qwen3-Max
closed/API-first; Kimi K3 / MiniMax M3 too large to self-host), so the direct API is the real route.

Also in this batch:

  • DeepSeek: native conf (deepseek-v4-pro/-flash/-reasoner + deepseek-v4-flash-vision-exp),
    the OpenRouter deepseek-v4-flash-vision-exp mirror (multimodal — supports_images set so image
    requests aren't dropped), and R1 local distills 1.5B–32B in conf/custom_models.json for
    Ollama/LM Studio (70B excluded so nothing OOMs a single box).
  • Ox Alpha: stealth/ox-alpha (anonymous stealth model, 1M ctx reasoning+vision). No AA/BenchLM
    coverage, so scored from LiveBench 2026-06-25 (Overall 69.2) → intelligence_score 15, held below
    the confirmed open flagships for the methodology mismatch + unverified-stealth caveat; provenance
    recorded in docs/open_model_scoring.md.
  • GLM-5.3 Flash (2026-08): z-ai/glm-5.3-flash (OpenRouter) + native Z.ai glm-5.3-flash, score
    16 (pinned across routes; one below the GLM-5.3 flagship's 17), supports_images true — the first
    native multimodal in the GLM-5 series (320B/18B MoE, 1M context, AA Intelligence Index 57). The bare
    glm-flash alias now resolves to it. Smoke-tested live via OpenRouter.

⚠️ PENDING-TESTING — native provider-path deferred (no keys), but every provider has a test

Unit coverage is complete and includes the cross-cutting invariants an adhd gap audit surfaced
(tests/test_native_provider_wiring.py): four-registry bijection (enum ↔ key map ↔ PROVIDER_PRIORITY_ORDER
↔ both restriction maps), natives-rank-before-OpenRouter, cross-route score pinning (native conf score
== OpenRouter mirror), per-model conf well-formedness, explicit vision flags, key-gating, and the local R1
distills. The real API call for the native providers is deferred (the author holds no keys for
them) — but not absent: tests/test_native_provider_live.py ships one skipif-no-key live smoke per
native provider (marked integration), and a census test fails if any native provider lacks one, so
"deferred" is auditable rather than silently missing.

Live-verified through OpenRouter: DeepSeek (deepseek/deepseek-v4-flashpong) and the free
Ox Alpha (stealth/ox-alpha). Note deepseek/deepseek-v4-flash-vision-exp returns an OpenRouter 404
("guardrail restrictions and data policy") until a data policy is enabled on the account — an account
setting, not a bad slug.

Still pending a key each (native direct-API path unexercised):

  • Native providers: DeepSeek (api.deepseek.com), Qwen/DashScope, GLM/Z.ai, Kimi/Moonshot, MiniMax,
    Mistral, NVIDIA, Perplexity.
  • NVIDIA NIM slugs (nvidia/nemotron-3-*) are also pending live /models validation (the conf flags this).
  • DeepSeek R1 local distills — validated against Ollama tag names; need a local CUSTOM_API_URL.

To run the deferred live tests — only the providers whose key is present run; the rest skip with a reason:

# add the key(s) to the environment or the repo-root .env, then:
.pal_venv/bin/python -m pytest tests/test_native_provider_live.py -m integration

Each present *_API_KEY (or OPENROUTER_API_KEY) fires exactly one tiny call to a cheap model for that provider.

6. Qwen3.8-Flash + OpenRouter Fusion meta-router (auto_selectable, cost/injection seals)

Qwen3.8-Flash (production). Adds qwen3.8-flash, the production serving of the Flash-Next
architecture (hosted SKU; multimodal text/image/video, 1M context). Native DashScope + a
qwen/qwen3.8-flash OpenRouter mirror, intelligence_score 16 pinned to the family's Artificial
Analysis Intelligence Index 56 (AA-56 → 16, the gemini-3.7-flash precedent; AA scored the open-weight
Flash-Next checkpoint, the hosted one differs slightly). The bare qwen-flash alias now resolves to it.
Smoke-tested live via OpenRouter.

auto_selectable flag. New ModelCapabilities.auto_selectable (default true). When false, a
model stays callable by explicit name and visible in listmodels, but is excluded from the auto-mode
ranked candidate summaries. Pure opt-out — every existing model is unaffected.

OpenRouter Fusion meta-router. openrouter/fusion is a council-behind-one-slug (a panel of up to 8
models → an analyst → a synthesis model; ~4–5× a single completion, dynamically priced). Added at
provisional score 17, auto_selectable:false (never auto-picked; used only when explicitly named). Two
named preset SKUs make the presets selectable by name:

  • openrouter/fusion-quality — panel pinned to a frontier trio (Opus + GPT + Gemini Pro).
  • openrouter/fusion-fast — the vendor general-fast preset.

Cost + injection seals. The panel config is sealed in the conf entry's _extras (a whitelisted
fusion block) and injected server-side as extra_body.plugins through a new
_augment_completion_params provider hook. The hook runs after the caller-kwargs whitelist, so a
caller-supplied plugins/extra_body (as could be smuggled via prompt-injected tool output) can never
reach the request body (injection seal). The SKU alias is rewritten to the real openrouter/fusion
wire slug; fan-out is hard-capped at 8; OpenRouter's per-call usage.cost is surfaced as a spend
receipt (cost seal). A true pre-flight dollar ceiling is not attempted, since Fusion is dynamically
priced.

Live-verified through OpenRouter (both SKUs, through PAL's own provider path so the injection is
exercised): fusion-quality → outer claude-opus-5, panel = the pinned Opus/GPT/Gemini-Pro trio
($0.077); fusion-fast → outer claude-sonnet-5, panel = Kimi/Gemini-Flash/DeepSeek ($0.018).
tests/test_fusion_router.py (9) covers the injection seal (attacker panel ignored), the fan-out
ceiling, the cost receipt, the auto-mode exclusion, and the SKU→wire-slug rewrite.

Claude Fable 5.1 + Qwen3.8-Max-0902 (2026-09). Two fast-follow catalog updates:

  • Claude Fable 5.1 — native claude-fable-5-1 + OpenRouter anthropic/claude-fable-5.1, score 20
    (tops the Artificial Analysis Intelligence Index at 66, max effort, ahead of Opus 5 at 63). Score pinned
    across routes; the bare fable alias now resolves to 5.1 (newest-wins), fable-5 still addresses Fable 5.
    Live-smoke-tested through OpenRouter.
  • Qwen3.8-Max-0902 — native DashScope snapshot qwen3.8-max-0902 (dated alias
    qwen3.8-max-2026-09-02), score 17, held at the base-Max AA tier because the gain is coding-specific
    on Code Arena WebDev (Output is getting cut off #1, 1691) with no published AA Intelligence Index. Bare qwen-max/qwen3-max now
    point at it. OpenRouter mirror deferred until it lands there; native path deferred (no key).

Alias consistency fixes

  • OpenRouter gpt-5.1: there is no openai/gpt-5.1 OpenRouter mirror, so the gpt-5.1/5.1
    aliases on the openai/gpt-5.2 entry silently resolved to 5.2 — while the OpenAI-direct catalog
    (correctly) maps gpt-5.1 to its own model. Dropped those aliases from the OR entry so the same
    alias doesn't mean two different things by provider.
  • gpt5pro/gpt5-pro: the generic shorthand pointed at gpt-5.2-pro; reassigned to gpt-5-pro
    for least-surprise (gpt5.2pro/gpt5.2-pro still address 5.2 Pro).

CI

  • Pinned black==25.1.0 in requirements-dev.txt. black 26.x's stable-style bump reformats ~10 files
    this PR never touches (the unpinned black>=23 in CI pulled 26.x and failed black --check . on
    main identically). Pinning to the last pre-26 stable makes the lint job reproducible without
    reformatting unrelated files.
  • PR title uses the Conventional Commits feat: prefix expected by the "Validate PR" check.

Testing

  • python -m pytest tests/ -m "not integration": 970 passed, 4 skipped. Lineup/alias/selection
    expectations updated for the new models; default-path selection tests assert upstream behavior
    again, and flag-on tests (DYNAMIC_MODEL_SELECTION=1) cover the stronger picks. Added tests for
    block-list alias matching (alias in block-list blocks its canonical; canonical block-list entry
    rejects a request made via alias), and for the Anthropic provider (catalog/alias/category selection
    plus generate_content request-shaping — thinking budget, temperature clamp, max_tokens). No
    assertions were weakened.
  • ruff / black (pinned) / isort clean.

Happy to split the block-list feature into its own PR if you'd prefer to review the catalog and the
selection changes separately.

rpeck added 2 commits May 31, 2026 12:02
Block-list takes precedence over allow-list; alias-aware matching is shared
between both. A model is usable iff not blocked AND (no allow-list OR on it).
Lets a provider stay open while excluding specific models. Tests + .env docs.
…ection

Catalog: gpt-5.5/-pro, gpt-5.4 family, gpt-5.3/5.2-codex, gpt-5.1, gpt-5.1-codex-max,
gpt-5-pro, *-chat-latest, o1/o1-pro, 4o/4.1 minis; gemini-3.5-flash, 3.1-pro-preview,
3.1-flash-lite, 3-flash-preview, 2.5-flash-lite; matching OpenRouter mirrors.
intelligence_scores extend the existing 1-20 scale (frontier 5.5=20, near-frontier
and codex=19, prior flagship 5.2=18, etc.); bare aliases (pro/flash/flash-lite) track newest.

Selection: OpenAI get_preferred_model prefers the newest flagships (gpt-5.5 reasoning/
balanced, gpt-5.4-mini fast); Gemini find_best is now capability/score-based instead of
lexical. Auto-mode therefore picks the current best models. Tests updated to match.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a block-list mechanism (*_DISALLOWED_MODELS) to the model restriction service, allowing specific models to be rejected per provider. It also updates the model catalogs for Gemini, OpenAI, and OpenRouter with several new model versions (such as Gemini 3.1/3.5 and GPT-5.4/5.5), refactors Gemini's fallback selection to rank models by capability rather than lexically, and updates associated tests. The reviewer feedback suggests expanding the OpenAI preference lists for EXTENDED_REASONING, FAST_RESPONSE, and BALANCED categories to include more of these newly added flagship, specialized, and mini/nano models for optimal auto-selection.

Comment thread providers/openai.py Outdated
Comment thread providers/openai.py Outdated
Comment thread providers/openai.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e19e11dd56

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread providers/openai.py Outdated
@rpeck rpeck changed the title Add current OpenAI/Gemini models, score-driven auto-selection, and *_DISALLOWED_MODELS feat: Add current OpenAI/Gemini models, optional dynamic auto-selection, and *_DISALLOWED_MODELS Jun 1, 2026
rpeck added 2 commits June 1, 2026 10:57
…LECTION

Auto-selection of the newest flagships and the bare-alias remaps (flash/pro/flash-lite
-> newest) now apply only when DYNAMIC_MODEL_SELECTION is set. Default (unset) restores
exact upstream behavior: gpt-5.2-leading OpenAI preference lists, Gemini reverse-
alphabetical ordering, and upstream bare-alias targets. This closes the model-not-found
regression for keys lacking access to the newest models (default no longer auto-picks them).

The flag-on OpenAI lists include the full current catalogue (codex + pro variants as
lower-priority fallbacks). Gemini selection filters candidates to canonical names so alias
strings cannot be returned or skew ordering. Adds a dynamic_aliases capability field plus
.env.example docs. Tests cover both flag states.
black 26.x's stable-style bump reformats files across the repo (10 untouched files fail
black --check on the unpinned black>=23.0.0). Pin to the last pre-26 stable so the lint
job is reproducible without reformatting unrelated files.
@rpeck

rpeck commented Jun 1, 2026

Copy link
Copy Markdown
Author

Thanks for the reviews! Pushed changes addressing all of it.

Title (Conventional Commits) — retitled with the feat: prefix.

Lint CI (black) — this was pre-existing black-version drift: the unpinned black>=23.0.0 pulls black-26.5.1, whose stable-style bump reformats 10 files this PR never touches (main fails identically). Pinned black==25.1.0 in requirements-dev.txt so the lint job is reproducible, without reformatting unrelated files.

chatgpt-codex-connector (P1 — auto-selecting a model the key may lack): good catch on the risk. The stronger auto-selection is now opt-in behind a new DYNAMIC_MODEL_SELECTION env var. Unset (the default), selection is the prior upstream behavior — the gpt-5.2-leading lists — so no existing user gets a model-not-found regression. (For the record, gpt-5.5 does exist in the current OpenAI API. The deeper point you raised — there's no runtime fallback when a key lacks access to the selected model — stays a known limitation only when the flag is enabled; it's documented in .env.example, and I'm happy to add an API-level fallback as a follow-up.)

gemini-code-assist (preference lists missing new models): addressed on the DYNAMIC_MODEL_SELECTION=1 path — the dynamic OpenAI lists now include the full current catalogue (the *-pro, *-codex, and mini/nano variants), with pro/codex as lower-priority fallbacks so they remain auto-selectable when a restriction policy allows only them. The default lists stay upstream.

Also gated the bare-alias remaps (flash/pro/flash-lite → newest) behind the same flag, so default alias resolution is unchanged. Tests cover both flag states; full unit suite is green and ruff/black are clean.

README gains a Model selection & restrictions subsection; docs/configuration.md gets the
block-list docs next to the allow-lists and a Dynamic Model Selection subsection with the
API-access caveat.
rpeck added 5 commits June 1, 2026 12:04
…SELECTION

The flag gates OpenAI's newest-first preference lists and the bare-alias remaps,
but Gemini selection is pool-driven (upstream's reverse-alphabetical algorithm), so
the newest-numbered model in the catalogue is naturally selected even with the flag
unset. Reword README / docs/configuration.md / .env.example to claim 'same upstream
algorithm' rather than 'same pick' for Gemini, and note the access caveat applies by
default once newer Gemini models are present. Add a clarifying comment to the
flag-off provider-selection test so the advanced default picks aren't misread as a
silent behavior change.
- OpenRouter: drop the gpt-5.1/5.1 aliases from openai/gpt-5.2 (no openai/gpt-5.1
  mirror exists, so they silently resolved to 5.2 while the OpenAI-direct catalog
  maps gpt-5.1 to its own model). Cross-provider alias meaning now consistent.
- Reassign the generic shorthand gpt5pro/gpt5-pro from gpt-5.2-pro to gpt-5-pro
  (least-surprise; gpt5.2pro/gpt5.2-pro still address 5.2 Pro). Also drop gpt5pro
  from the openai/gpt-5.2-pro OpenRouter mirror (no gpt-5-pro mirror to retarget to).
- Add tests for *_DISALLOWED_MODELS alias-aware matching: an alias in the block-list
  blocks its canonical, and a canonical block-list entry rejects requests via alias.
- Document that the flag-on bare-alias remap is native-provider (Gemini direct) only;
  OpenRouter newest models are reached by explicit or-* names.
Adds a flagship + a cheaper/smaller tier for each of: DeepSeek (V4 pro/flash),
Qwen (3.7-max / 3.6-flash), Kimi (K2.6/K2.5), GLM (5.1 / 4.7-flash), MiniMax
(M3 / M2.5), Gemma 4 (31b / 26b-a4b), Mistral (Large 3 / Small), Xiaomi MiMo
(v2.5 pro / v2.5), gpt-oss (120b/20b), Llama 4 (Maverick/Scout), and NVIDIA
Nemotron-3 (super-120b / nano-30b). 22 entries.

Context windows and capability flags (vision / extended-thinking / tools / json)
are taken from the live OpenRouter catalogue, so these models get correct context
budgeting and become eligible for DEFAULT_MODEL=auto and listmodels instead of
falling back to the 32K generic-passthrough defaults. intelligence_scores are
human-curated on the existing 1-20 scale. New aliases are family-scoped and do not
collide with or repoint any existing bare aliases (deepseek/mistral/llama unchanged).

Adds durable catalogue guard tests (families present, entries well-formed, no
duplicate aliases) that don't hard-code specific model IDs.
…3 context windows

Calibrated the 22 open-model scores against HuggingFace/benchmark data (LiveCodeBench,
SWE-bench, GPQA-Diamond, MMLU-Pro, AIME) and peer-reviewed with GPT-5.4 + Gemini-3.5-flash:

- Up: 2026 open flagships were under-scored - qwen3.7-max 15->16, kimi-k2.6 15->16,
  glm-5.1 15->16, deepseek-v4-flash 13->14, qwen3.6-flash 12->13, minimax-m3 14->15,
  minimax-m2.5 12->13, glm-4.7-flash 11->12, mimo-v2.5-pro 13->15, mimo-v2.5 11->13.
- Down: 2025-era / non-reasoning models were over-scored - llama-4-maverick 13->12,
  llama-4-scout 11->10, gpt-oss-120b 14->13, nemotron-3-super 13->12,
  mistral-large-2512 13->12 (non-reasoning MoE; benchmarks below Maverick).

Context windows corrected to the provider-served value (catalog advertised the arch max):
minimax-m3 1048576->524288, llama-4-scout 10000000->327680, nemotron-3-super 1000000->262144.

Also de-stale the conf _README note that claimed no OpenRouter model supports extended
thinking (many now do; flags are set from each model's advertised reasoning support).
…cross conf files

The field_descriptions boilerplate in azure/xai/openai/gemini model configs all carried
the same parenthetical claiming no model supports extended reasoning - now false (gpt-5.x,
gemini-3.x, grok-4, and the open-weight models all advertise reasoning). Reword to match the
corrected openrouter_models.json note; the flag is set from each provider's advertised
reasoning support. Documentation-only (_README field is not parsed by the loader).
@rpeck

rpeck commented Jun 2, 2026

Copy link
Copy Markdown
Author

How the open-model intelligence_scores were calibrated

The intelligence_score (the 1–20 signal that orders the auto-mode menu and breaks ties) for the 22 open-weight OpenRouter models was derived from current benchmark data and cross-checked by three models — Opus 4.8 + GPT-5.4 + Gemini-3.5-Flash. Both external reviewers signed off with no remaining changes. Documenting the reasoning here so the numbers aren't a black box.

Benchmark research (method & sources)

Data gathered 2026-06-01, HuggingFace-first: model cards (MMLU-Pro, GPQA-Diamond, LiveCodeBench, AIME, SWE-bench Verified & Pro). For API-only models not on HF (Qwen-Max, GLM, MiniMax, Kimi), fell back to the Artificial Analysis Intelligence Index and LMArena Elo, labeled as such. Every model ID and served context window was verified against the live OpenRouter /api/v1/models response (343 models).

Scale is integer 1–20, anchored to existing catalog entries: gpt-5.5=20, gpt-5.1-codex/gemini-3.1-pro=19, gpt-5.2=18, gpt-5-pro=17 · deepseek-r1-0528=15, grok-4=15 · mistral-large-2411=11, llama-3-70b=9. Bands: 17–20 frontier, 14–16 strong near-frontier open flagship, 11–13 solid mid, 8–10 small/older. Scores reflect benchmark standing relative to those anchors, not absolute benchmark numbers.

Reasoning by model

  • Opus 4.8 (orchestration + synthesis): commissioned the benchmark research, set the anchors, integrated both external reviews, made the final calls, and verified the disputed capability flags against OpenRouter's own supported_parameters (source data over prior intuition).
  • GPT-5.4 (benchmark-grounded): argued the 2026 open flagships were under-scored and the 2025-era / non-reasoning models over-scored. Strongest calls: mistral-large-2512 is a non-reasoning MoE (GPQA-D 48.6, LiveCodeBench 29.3) that benchmarks below Llama-4-Maverick, so its initial 13 was ~2 points high; llama-4-maverick (Apr-2025, weak coding) belongs near the mistral-large-2411=11 anchor, not with the 2026 flagships. Also flagged a stale "no OpenRouter model supports reasoning" README note. Round-2: ship it.
  • Gemini-3.5-Flash (positional / structural): pushed back on the downgrades — a flagship should outrank its cheap tier and reflect generational improvement (wanted mistral-large=13, maverick=13, scout=11, gpt-oss-120b=14, nemotron-super=13) — and flagged the awkward mistral-large == mistral-small tie. Round-2: ship it; confirmed family monotonicity is clean.

Synthesis — where the final numbers landed

  • Applied all 2026-flagship upgrades (both reviewers agreed or didn't object): qwen3.7-max / kimi-k2.6 / glm-5.1 → 16; minimax-m3 / mimo-v2.5-pro → 15; deepseek-v4-flash → 14; qwen3.6-flash / minimax-m2.5 / mimo-v2.5 → 13; glm-4.7-flash → 12.
  • Split-the-difference where the reviewers bracketed a value: mistral-large-2512 = 12 (GPT 11 / Gemini 13; also resolves the large==small tie → large 12 > small 11) and llama-4-maverick = 12 (GPT 11 / Gemini 13); llama-4-scout = 10.
  • Sided with GPT-5.4 / the benchmarks over Gemini's positional argument: gpt-oss-120b = 13 (Aug-2025, GPQA-D 80.1) and nemotron-3-super = 12 (weak LiveCodeBench 31).
  • Overrode both reviewers on capability flags after checking the source: kept supports_extended_thinking=true for the flash/small models (gemma-4, glm-4.7-flash, qwen3.6-flash, mimo-v2.5, nemotron-nano) because OpenRouter reports reasoning=true for them; kept the uncapped OR-served max_output_tokens (repo precedent: grok-4 256K/256K).

Changes that landed

  • 15 intelligence_score updates (net: 2026 open flagships up, 2025-era / non-reasoning down).
  • 3 context windows corrected to the provider-served value (catalog advertised the architectural max): minimax-m3 1048576→524288, llama-4-scout 10000000→327680, nemotron-3-super 1000000→262144.
  • De-staled the "no OpenRouter model supports extended thinking" note in the _README field descriptions of all conf files (azure/xai/openai/gemini/openrouter) — now false (gpt-5.x, gemini-3.x, grok-4 and the open-weight models all advertise reasoning).

Final scores

Score Models
16 deepseek-v4-pro, qwen3.7-max, kimi-k2.6, glm-5.1
15 minimax-m3, mimo-v2.5-pro
14 deepseek-v4-flash, kimi-k2.5
13 qwen3.6-flash, minimax-m2.5, mimo-v2.5, gpt-oss-120b
12 glm-4.7-flash, gemma-4-31b-it, mistral-large-2512, llama-4-maverick, nemotron-3-super-120b-a12b
11 gemma-4-26b-a4b-it, mistral-small-2603, gpt-oss-20b
10 llama-4-scout, nemotron-3-nano-30b-a3b

Sources: huggingface.co model cards; Artificial Analysis Intelligence Index; LMArena; OpenRouter /api/v1/models (served context + capability params). Commits 8d741df (scores + context fixes) and 8baaa7d (README de-stale).

…chmarks, council-reviewed)

Adds docs/open_model_scoring.md (linked from docs/configuration.md) documenting how the
intelligence_score for the 22 open-weight OpenRouter models was derived and reviewed, so the
numbers visible in conf/openrouter_models.json are not a black box. The scores themselves
landed in earlier commits; this records the method and the reasoning. They were cross-checked
by three models (Opus 4.8 + GPT-5.4 + Gemini-3.5-Flash); both external reviewers signed off
with no remaining changes.

== Benchmark research (method & sources) ==
Data gathered 2026-06-01, HuggingFace-first: model cards (MMLU-Pro, GPQA-Diamond, LiveCodeBench,
AIME, SWE-bench Verified & Pro). For API-only models not on HF (Qwen-Max, GLM, MiniMax, Kimi),
fell back to the Artificial Analysis Intelligence Index and LMArena Elo, labeled as such. Every
model ID and served context window was verified against the live OpenRouter /api/v1/models
response (343 models).

Scale is integer 1-20, anchored to existing entries (gpt-5.5=20, gpt-5.1-codex / gemini-3.1-pro=19,
gpt-5.2=18, gpt-5-pro=17 | deepseek-r1-0528=15, grok-4=15 | mistral-large-2411=11, llama-3-70b=9).
Bands: 17-20 frontier, 14-16 strong near-frontier open flagship, 11-13 solid mid, 8-10 small/older.
Scores reflect benchmark standing RELATIVE to those anchors, not absolute benchmark numbers.

== Reasoning by model ==
Opus 4.8 (orchestration + synthesis): commissioned the benchmark research, set the anchors,
integrated both external reviews, made the final calls, and verified the disputed capability flags
against OpenRouter's own supported_parameters (source data over prior intuition).

GPT-5.4 (benchmark-grounded): argued the 2026 open flagships were under-scored and the 2025-era /
non-reasoning models over-scored. Strongest calls: mistral-large-2512 is a non-reasoning MoE
(GPQA-D 48.6, LiveCodeBench 29.3) that benchmarks BELOW Llama-4-Maverick, so its initial 13 was
~2 points high; llama-4-maverick (Apr-2025, weak coding) belongs near the mistral-large-2411=11
anchor, not with the 2026 flagships. Also flagged a stale "no OpenRouter model supports reasoning"
README note. Round-2 verdict: ship it.

Gemini-3.5-Flash (positional / structural): pushed back on the downgrades - a flagship should
outrank its cheap tier and reflect generational improvement (wanted mistral-large=13, maverick=13,
scout=11, gpt-oss-120b=14, nemotron-super=13), and flagged the awkward mistral large==small tie.
Round-2 verdict: ship it; confirmed family monotonicity is clean.

== Synthesis (Opus 4.8): where the final numbers landed ==
- Applied all 2026-flagship upgrades (both reviewers agreed or did not object): qwen3.7-max /
  kimi-k2.6 / glm-5.1 -> 16; minimax-m3 / mimo-v2.5-pro -> 15; deepseek-v4-flash -> 14;
  qwen3.6-flash / minimax-m2.5 / mimo-v2.5 -> 13; glm-4.7-flash -> 12.
- Split-the-difference where the reviewers bracketed a value: mistral-large-2512 = 12 (GPT 11 /
  Gemini 13; also resolves the large==small tie -> large 12 > small 11) and llama-4-maverick = 12
  (GPT 11 / Gemini 13); scout = 10.
- Sided with GPT-5.4 / the benchmarks over Gemini's positional argument: gpt-oss-120b = 13
  (Aug-2025, GPQA-D 80.1) and nemotron-3-super = 12 (weak LiveCodeBench 31).
- Overrode BOTH reviewers on capability flags after checking the source: kept
  supports_extended_thinking=true for the flash/small models (gemma-4, glm-4.7-flash, qwen3.6-flash,
  mimo-v2.5, nemotron-nano) because OpenRouter reports reasoning=true for them; kept the uncapped
  OR-served max_output_tokens (repo precedent: grok-4 256K/256K).

== Final scores ==
16  deepseek/deepseek-v4-pro, qwen/qwen3.7-max, moonshotai/kimi-k2.6, z-ai/glm-5.1
15  minimax/minimax-m3, xiaomi/mimo-v2.5-pro
14  deepseek/deepseek-v4-flash, moonshotai/kimi-k2.5
13  qwen/qwen3.6-flash, minimax/minimax-m2.5, xiaomi/mimo-v2.5, openai/gpt-oss-120b
12  z-ai/glm-4.7-flash, google/gemma-4-31b-it, mistralai/mistral-large-2512,
    meta-llama/llama-4-maverick, nvidia/nemotron-3-super-120b-a12b
11  google/gemma-4-26b-a4b-it, mistralai/mistral-small-2603, openai/gpt-oss-20b
10  meta-llama/llama-4-scout, nvidia/nemotron-3-nano-30b-a3b

Sources: huggingface.co model cards; Artificial Analysis Intelligence Index; LMArena;
OpenRouter /api/v1/models (served context + capability params).
@rpeck

rpeck commented Jun 3, 2026

Copy link
Copy Markdown
Author

✅ Ready for maintainer review

All review-bot feedback has been addressed and CI is green — handing this off for human review.

What it delivers (three separable concerns):

  1. Model catalog — current OpenAI + Gemini models and matching OpenRouter mirrors, plus a refreshed open-weight OpenRouter set (DeepSeek, Qwen, Kimi, GLM, MiniMax, Gemma, Mistral, MiMo, gpt-oss, Llama 4, Nemotron — flagship + a cheap/small tier each). intelligence_scores were benchmark-calibrated and peer-reviewed (see docs/open_model_scoring.md).
  2. *_DISALLOWED_MODELS block-list — per-provider block-list complementing the existing allow-lists; block wins; alias-aware; no behavior change when unset.
  3. DYNAMIC_MODEL_SELECTION flag — gates the "prefer the newest flagships" auto-selection and bare-alias remaps behind an env var; default unset = upstream behavior, which closes the auto-default-to-inaccessible-model concern.

Status:

  • ✅ All 6 CI checks green; branch is mergeable.
  • ✅ All 4 review threads resolved (3× gemini-code-assist on the OpenAI preference lists, 1× chatgpt-codex P1) — see the replies on each for how they were addressed.
  • 📋 One intentionally-deferred follow-up: runtime fallback when an auto-selected model is inaccessible at call time (a pre-existing property of the selection path, not introduced by this PR) — tracked separately and noted under Known limitation in the description.

Happy to split the block-list feature into its own PR if you'd prefer to review the pieces separately. Thanks for taking a look!

@rpeck

rpeck commented Jun 3, 2026

Copy link
Copy Markdown
Author

@guidedways — when you have a chance, this one's ready for a look. All 6 CI checks are green, it's mergeable, and all four review-bot threads (3× gemini-code-assist + 1× chatgpt-codex P1) are addressed and resolved. The default behavior is unchanged — the newer auto-selection and alias remaps are all opt-in behind DYNAMIC_MODEL_SELECTION. No rush, and I'm happy to split the *_DISALLOWED_MODELS block-list into its own PR if that's easier to review. Thanks!

@rpeck rpeck changed the title feat: Add current OpenAI/Gemini models, optional dynamic auto-selection, and *_DISALLOWED_MODELS feat: Add current OpenAI/Gemini models, newer open models, optional dynamic auto-selection, and *_DISALLOWED_MODELS Jun 3, 2026
@rpeck

rpeck commented Jun 3, 2026

Copy link
Copy Markdown
Author

Following up on the review summary's suggestion to expand the OpenAI preference lists (EXTENDED_REASONING / FAST_RESPONSE / BALANCED) to cover the newly added flagship, specialized, and mini/nano models. This is addressed; I've resolved the three inline threads with specifics, and here's the short version for visibility:

  • With DYNAMIC_MODEL_SELECTION=1, the preference lists already include every model the review names: gpt-5.5-pro, gpt-5.4-pro, gpt-5.2-codex, gpt-5.1-codex-max (reasoning/balanced) and gpt-5-nano, gpt-4.1-mini/-nano, gpt-4o-mini (fast). Each is auto-selectable when the restriction policy allows it (find_first returns the first allowed entry), and all referenced IDs exist in the catalog (no dead entries).
  • The opt-in FAST_RESPONSE list is mini/nano-only; the large flagships the review flagged remain only in the default list.
  • The default (flag unset) lists stay byte-for-byte identical to upstream by design, so existing users see no change. That's the backward-compatibility decision the flag exists for.
  • One intentional divergence from the suggested ordering: the -pro tier sits as a lower-priority fallback rather than leading, so auto-mode doesn't default to the most expensive model on every task while keeping those models selectable when a policy allows only them.

Happy to reorder so the lists lead with the pro tier if you'd prefer that.

@adrenalin-labs

Copy link
Copy Markdown

+1 — hit this exact limitation on v9.8.2. The OpenAI key can call gpt-5.5 directly (verified in another platform using the same key), but PAL's built-in model registry rejects it, so it never shows in listmodels or dispatches. This PR is exactly the fix, and it's green across lint/tests/Docker on 3.10–3.11–3.12. Any chance of a review/merge? Happy to help test if useful.

OpenAI GPT-5.6 generation: add gpt-5.6-sol (flagship, score 20) and gpt-5.6-luna
(low-cost tier, score 17 provisional) to conf/openai_models.json; gpt-5.6-terra was
already present (19). Add matching openai/gpt-5.6-{sol,terra,luna} OpenRouter mirrors.
The flag-on (DYNAMIC_MODEL_SELECTION) OpenAI preference lists now lead with gpt-5.6-sol
and carry terra in the strong tier / luna as a fallback; the default (flag-off) lists
stay byte-for-byte upstream. Alias + flag-on selection tests updated.

Gemini score fixes (benchlm July-2026 ranking): gemini-2.5-pro 18 -> 15 (it ranks below
gemini-3.5-flash and gemini-3-pro), gemini-3.5-flash 12 -> 14 (genuinely strong). The
current Gemini catalogue is otherwise complete for what the public API serves (Gemini 3
Pro Deep Think is early-access only, not standard-API-reachable, so not added).
@rpeck

rpeck commented Jul 13, 2026

Copy link
Copy Markdown
Author

Follow-up: pushed 921cc52 adding the GPT-5.6 family (gpt-5.6-sol flagship / gpt-5.6-terra / gpt-5.6-luna) to the OpenAI catalog, matching openai/gpt-5.6-* OpenRouter mirrors, and the two Gemini score recalibrations (gemini-2.5-pro 18→15, gemini-3.5-flash 12→14) from benchlm's July ranking. The flag-on (DYNAMIC_MODEL_SELECTION) OpenAI preference lists now lead with gpt-5.6-sol; the default (flag-off) lists remain byte-for-byte upstream. 878 unit tests pass and all 6 CI checks are green on this commit.

@abossenbroek

abossenbroek commented Jul 16, 2026

Copy link
Copy Markdown

+1 on merging this work! Feels like active users need to fork if maintainers are not accepting PRs.

Keep up the work @rpeck

@jtwillis92

Copy link
Copy Markdown

I've been wanting to get gpt-5.6-sol working through pal - this would be a great addition and it looks like its ready to merge. 🥺

@jtwillis92

Copy link
Copy Markdown

Tested this branch locally and it works as expected — gpt-5.6-sol (previously rejected by the static registry) now validates and routes correctly, confirmed via metadata.model_used in the tool response.

For anyone else hitting this before it merges, you can pin your MCP config to this PR's head:

claude mcp add --scope user pal \
  --env OPENAI_API_KEY="$OPENAI_API_KEY" \
  --env DEFAULT_MODEL=auto \
  -- uvx \
  --from git+https://github.com/rpeck/pal-mcp-server.git@921cc52b8a9a1045d9488654563c2dc82a8fb528 \
  pal-mcp-server

(Pinning the SHA rather than the branch keeps the install immutable; swap back to the upstream repo once this merges.)

Worth noting the friction this fixes: several models in the current conf/openai_models.json are now deprecated upstream — gpt-5-codex and gpt-5.1-codex return deprecation errors on invocation, while newer models the API key can actually use (gpt-5.6-sol/luna/terra, gpt-5.5-pro, gpt-5.3-codex) are rejected by the registry before the request ever leaves the server. So out of the box, code-review workflows fail on both the old names and the new ones. Would be great to see this land.

nathanaelad added a commit to nathanaelad/pal-mcp-server that referenced this pull request Jul 29, 2026
…model catalogue refresh)

Merges BeehiveInnovations#450 by @rpeck, which is open,
mergeable and green on CI but unreviewed; upstream main has had no commits
since 2025-12-15, so this is carried downstream here instead of waiting.

Adds gpt-5.6-sol/terra/luna, the GPT-5.5/5.4/5.3/5.2-codex entries and
current Gemini ids to the static registries, plus opt-in
DYNAMIC_MODEL_SELECTION and *_DISALLOWED_MODELS support.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rpeck added 2 commits August 21, 2026 17:09
Google shipped gemini-3.7-flash on 2026-08-13: 1M context, thinking
(low/medium/high), multimodal, cheap flash pricing ($0.75/$3.75 intro).
Public benchmarks put it at or above the prior pro tier — GPQA-Diamond
~94.8 (vs gemini-3.1-pro's 94.3) and Artificial Analysis Intelligence
Index 56 (vs 48 for gemini-3.1-pro-preview).

intelligence_score = 16: two generational steps above gemini-3.5-flash
(14), above gemini-2.5-pro (15), but kept below the dedicated pro
reasoning models (gemini-3-pro 18 / gemini-3.1-pro 19) so auto-mode's
EXTENDED_REASONING still routes to a pro model rather than a flash whose
AA-index is inflated by cheap thinking. FAST_RESPONSE/BALANCED now
resolve to gemini-3.7-flash in both flag states (reverse-alpha flag-off,
score-rank flag-on); EXTENDED_REASONING is unchanged (pro-gated).

The "gemini-flash-latest" alias and the dynamic "flash" remap move from
gemini-3.5-flash to gemini-3.7-flash (latest = newest); 3.5-flash keeps
its version-specific flash3.5 alias. OpenRouter mirror added
(google/gemini-3.7-flash, or-gemini-flash); the 3.5 mirror's alias
becomes or-gemini-flash-3.5 to avoid a collision.

Tests: alias resolution for flash3.7/gemini-flash-latest, and the
FAST/BALANCED default-pick expectations updated across the auto-mode
suites. 878 passed, 4 skipped; ruff/black clean.
Scores calibrated from the Artificial Analysis Intelligence Index
(artificialanalysis.ai, cross-checked on benchlm.ai's aggregate),
mapped onto the catalog's 1-20 scale with the same proprietary-anchored
judgment as the existing entries.

Gemini flash family (conf/gemini_models.json):
- gemini-3.6-flash (NEW, score 15) — AA index 51.6, between 3.5-flash
  (14) and 3.7-flash (16).
- gemini-3.5-flash-lite (NEW, score 11) — AA index 37.4, the fastest
  model AA measured and well above the lite-tier median (~17); newest
  lite, so it takes over the gemini-flash-lite-latest alias and the
  dynamic flash-lite/flashlite remaps from gemini-3.1-flash-lite (now
  aliased flashlite3.1).
- Noted the whole 3.x flash family is on Google's developer free tier.
- Default FAST/BALANCED pick stays gemini-3.7-flash (16); pool selection
  and EXTENDED_REASONING (pro-gated) are unchanged.

OpenRouter frontier additions (conf/openrouter_models.json):
- anthropic/claude-fable-5 (score 20, +code-gen) — AA index 62.1, second
  only to Opus 5; premium pricing ($10/$50).
- moonshotai/kimi-k3 (17) — AA 59.7, top open-weights tier; takes the
  bare "kimi" alias from kimi-k2.6.
- z-ai/glm-5.3 (17) — AA 59.5; takes "glm"/"glm5" from glm-5.1.
- qwen/qwen3.8-max (17) — AA 58.1; takes "qwen"/"qwen-max"/"qwen3-max"
  from qwen3.7-max.
- meta/muse-spark-1.2 (16) — AA 56.8, Meta agentic reasoning model.
  Prior flagships kept at 16 with version-pinned aliases.

Tests: flash-lite alias assertions updated for the latest-lite move.
878 passed, 4 skipped; ruff clean.
PAL could only reach Claude models through OpenRouter. This adds a
first-party Anthropic provider on the Messages API so a direct route is
available with an ANTHROPIC_API_KEY, and so bare aliases like `opus` /
`fable` resolve to Anthropic-direct (native providers rank ahead of
OpenRouter in the priority order).

- ProviderType.ANTHROPIC + ANTHROPIC_API_KEY wiring in the registry
  (priority: after OpenAI, before Azure) and ANTHROPIC_ALLOWED_MODELS /
  ANTHROPIC_DISALLOWED_MODELS restriction support.
- providers/anthropic.py (RegistryBackedProviderMixin) implementing
  generate_content over anthropic.Anthropic().messages.create: system
  prompt, image blocks, usage, retries. Extended thinking maps PAL's
  thinking_mode to Anthropic budget_tokens, clamped to the >=1024 floor
  (so "minimal" disables thinking) and kept < max_tokens; temperature is
  forced to 1.0 while thinking is enabled, as the API requires. max_tokens
  is always sent (required by the Messages API). The SDK is imported
  lazily so the module loads without it installed.
- conf/anthropic_models.json: claude-opus-5 (20), claude-fable-5 (20),
  claude-sonnet-5 (18), claude-haiku-4-5 (12); scores from the Artificial
  Analysis Intelligence Index (Opus 5 63, Fable 5 62). get_preferred_model
  sorts by raw intelligence_score first because the effective capability
  rank saturates at 100 for every model scoring >=18 (opus/fable/sonnet),
  which would otherwise let a lexical tie-break pick sonnet over opus.
- server bootstrap registration, anthropic>=0.40.0 in requirements.
- Tests: catalog/alias/selection + generate_content request-shaping
  (thinking budget, temperature clamp, max_tokens). Fixed a latent env
  leak in test_auto_mode_model_listing (delete ANTHROPIC_API_KEY like the
  other non-target provider keys). 885 passed, 4 skipped; ruff/black clean.
@rpeck

rpeck commented Aug 22, 2026

Copy link
Copy Markdown
Author

@gemini-code-assist @chatgpt-codex-connector — heads up on new work pushed since your last pass, worth a re-review:

  • Native Anthropic (Claude) provider (1ddd05e) — a first-party Messages API backend (ProviderType.ANTHROPIC + ANTHROPIC_API_KEY) so Claude models have a direct route instead of only OpenRouter. Native providers rank ahead of OpenRouter, so bare aliases (opus/fable/sonnet/haiku) resolve to the direct API when a key is set; it's dormant (unregistered) unless ANTHROPIC_API_KEY is present, so there's no default-behavior change. Full write-up in the new §4 "Native Anthropic (Claude) provider" section of the PR description — covers the thinking-budget / max_tokens / temperature request-shaping, the score-first get_preferred_model (the effective capability rank saturates at 100 for every model scoring ≥18, so a raw-score sort is needed to keep opus/fable ahead of sonnet), the ANTHROPIC_ALLOWED_MODELS / ANTHROPIC_DISALLOWED_MODELS restriction hooks, and the lazy SDK import so the module loads without anthropic installed.
  • Catalog refresh — added gemini-3.7-flash / gemini-3.6-flash / gemini-3.5-flash-lite and the Aug-2026 open frontier tier on OpenRouter (Kimi K3, GLM-5.3, Qwen3.8-Max, Muse Spark 1.2, and Claude Fable 5), all calibrated against the Artificial Analysis Intelligence Index.

885 unit tests pass; ruff / black (pinned) / isort clean. Grateful for a look at §4 in particular.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ddd05ec4c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread requirements.txt Outdated
mcp>=1.0.0
google-genai>=1.19.0
openai>=1.55.2 # Minimum version for httpx 0.28.0 compatibility
anthropic>=0.40.0 # Native Anthropic (Claude) Messages API provider

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Raise the Anthropic SDK floor for extended thinking

The declared minimum anthropic==0.40.0 predates the thinking argument on messages.create, but every catalog model supports thinking and normal PAL calls use a non-minimal thinking mode, causing this provider to pass that keyword. Environments constrained to the advertised minimum therefore fail at runtime before making an API request; require an SDK version that exposes extended thinking or omit/gate the argument for older versions.

Useful? React with 👍 / 👎.

Comment thread providers/anthropic.py Outdated
Comment on lines +117 to +119
# Guarantee room for the response after the thinking budget.
if budget >= max_tokens:
max_tokens = min(capabilities.max_output_tokens or (budget + 4096), budget + 4096)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the caller's output-token limit

When max_output_tokens is smaller than the computed thinking budget, this assignment silently raises max_tokens to budget + 4096 rather than honoring the caller's maximum. For example, an Opus request capped at 1,000 tokens with medium thinking is sent with thousands of tokens instead, defeating cost and context-budget controls; reduce or disable the thinking budget when it cannot fit under the requested limit.

Useful? React with 👍 / 👎.

Comment thread server.py
Comment on lines +438 to +440
# Check for Anthropic API key
anthropic_key = get_env("ANTHROPIC_API_KEY")
if anthropic_key and anthropic_key != "your_anthropic_api_key_here":

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the Anthropic provider configuration

Users following the repository's documented setup paths cannot enable this otherwise dormant provider because neither .env.example nor docs/configuration.md exposes ANTHROPIC_API_KEY; those references also omit ANTHROPIC_ALLOWED_MODELS, ANTHROPIC_DISALLOWED_MODELS, and ANTHROPIC_MODELS_CONFIG_PATH. Add the new native provider and its configuration variables to the template and configuration reference.

AGENTS.md reference: AGENTS.md:L101-L102

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d5cad49. Added ANTHROPIC_API_KEY plus the 8 new native providers (DEEPSEEK/DASHSCOPE/ZAI/MOONSHOT/MINIMAX/MISTRAL/NVIDIA/PERPLEXITY) with base URLs to both .env.example and docs/configuration.md, and noted the per-provider *_ALLOWED_MODELS / *_DISALLOWED_MODELS / *_MODELS_CONFIG_PATH vars. The sibling P2 (Anthropic SDK floor) is fixed in the same commit: anthropic>=0.49.0, the floor that exposes the thinking arg on messages.create.

Comment thread docs/configuration.md Outdated
DYNAMIC_MODEL_SELECTION=1
```

With `DYNAMIC_MODEL_SELECTION` unset, auto-mode uses upstream's selection *algorithm* and bare aliases keep their upstream targets (`flash`→`gemini-2.5-flash`, `pro`→`gemini-3-pro-preview`). For OpenAI this means the established per-category preference lists, unchanged. For Gemini, upstream selection is reverse-alphabetical over the available models, so the newest-numbered model in the catalogue is naturally selected — adding newer Gemini models (`gemini-3.5-flash`, `gemini-3.1-pro-preview`) advances the default Gemini auto-pick accordingly, exactly as upstream's own algorithm would when models are added. Enabling the flag additionally makes OpenAI auto-selection lead with the newest flagship and remaps the bare aliases to the newest models (`flash`→`gemini-3.5-flash`, `pro`→`gemini-3.1-pro-preview`). The bare-alias remap applies to the **native** providers (Gemini direct); on OpenRouter the bare aliases keep their existing targets, so reach the newest OpenRouter models by their explicit `or-*` names. **Caveat:** with the flag on (and, for Gemini, by default once newer models are in the catalogue), auto-mode may pick a newer model your API key cannot access yet — selection falls back only on restriction policy, not on an API model-not-found at call time. Pair it with `*_ALLOWED_MODELS` / `*_DISALLOWED_MODELS` to constrain the pool.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the actual dynamic flash alias target

With DYNAMIC_MODEL_SELECTION enabled, conf/gemini_models.json assigns the flash dynamic alias to gemini-3.7-flash, but this paragraph twice promises gemini-3.5-flash. Operators relying on this reference may permit or budget for 3.5 while requests are routed to 3.7, so update the documented target to match the manifest.

AGENTS.md reference: AGENTS.md:L7-L12

Useful? React with 👍 / 👎.

@rpeck

rpeck commented Aug 22, 2026

Copy link
Copy Markdown
Author

/gemini review

1 similar comment
@rpeck

rpeck commented Aug 22, 2026

Copy link
Copy Markdown
Author

/gemini review

configuration.md described the DYNAMIC_MODEL_SELECTION bare-alias remap
and the flag-off Gemini auto-pick as `flash`→gemini-3.5-flash, but the
manifest now assigns the dynamic `flash` alias (and the newest-flash
reverse-alpha pick) to gemini-3.7-flash. Operators budgeting/allow-listing
off the doc would permit 3.5 while requests route to 3.7. Matches
conf/gemini_models.json. (Addresses codex P2.)
@rpeck

rpeck commented Aug 22, 2026

Copy link
Copy Markdown
Author

@chatgpt-codex-connector — thanks, good catch on the P2. Fixed in e9b16a0: docs/configuration.md now documents the dynamic flash alias (and the flag-off newest-flash reverse-alpha pick) as gemini-3.7-flash, matching conf/gemini_models.json. Both stale gemini-3.5-flash references in that paragraph are updated; no other docs still name 3.5 as the flash target.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

rpeck added 6 commits August 22, 2026 14:43
Two fixes from a gemini-3.5-flash review of the diff (run via PAL):

- generate_content: Anthropic counts thinking tokens against max_tokens,
  so the visible response needs headroom ON TOP of budget_tokens. The old
  code only grew max_tokens when budget >= max_tokens, which still starved
  output when a caller passed a max_output_tokens just above the budget
  (e.g. max_output_tokens=2000 with a ~21954-token high budget left ~46
  tokens for the answer). Now reserve the requested output and set
  max_tokens = budget + output_headroom, clamped to the model ceiling;
  drop thinking if both can't fit.
- get_preferred_model: the EXTENDED_REASONING filter indexed
  capability_map[m] unguarded, raising KeyError when allowed_models held a
  non-canonical name and none were canonical. Guard with `m in
  capability_map`, matching the gemini provider.

Added tests for both. 887 passed; ruff/black clean.
Every major open-model creator now runs its own OpenAI-compatible API, and
for some the flagship is API-only or unhostable (GLM-5.3 weights not yet
released; Qwen3-Max closed/API-first; Kimi K3 / MiniMax M3 too large to
self-host). PAL could only reach these via OpenRouter. This adds first-party
providers so a creator's key gives a direct route.

Each is a thin subclass of the shared OpenAI-compatible base (the xai.py
pattern): DeepSeek, Qwen (DashScope), Z.ai (GLM), Moonshot (Kimi), MiniMax,
Mistral, NVIDIA (Nemotron), Perplexity. New ProviderType members + env-key
map + PROVIDER_PRIORITY_ORDER (natives rank ahead of OpenRouter, so a bare
family alias like `deepseek`/`kimi` resolves to the direct API when the key
is set and falls back to OpenRouter when it isn't) + *_ALLOWED/DISALLOWED
restriction wiring + key-gated registration in server.configure_providers.

Each provider stays dormant unless its *_API_KEY is set (zero change for
existing users). Native conf scores are pinned to the OpenRouter mirrors.
Model IDs verified against each provider's docs (2026-08); the NVIDIA NIM
slugs are marked PENDING live /models validation. Perplexity Sonar are
search models, tagged as such.

TDD: test_deepseek_provider.py (written first, red→green) + a parametrized
test_vendor_providers.py for the other 7 (provider type, base URL, flagship
present + pinned score, alias resolution). 911 passed; ruff/black/isort clean.
…red)

- OpenRouter: deepseek/deepseek-v4-flash-vision-exp (score 13, supports_images
  true so image requests aren't silently dropped; experimental checkpoint
  scored below stable Flash).
- Local (custom_models.json / Ollama·LM Studio): DeepSeek R1 distills
  1.5B/7B/8B/14B/32B (scores 8-12, thinking on). 70B intentionally excluded
  so nothing OOMs a single local box. Catalog-only unless CUSTOM_API_URL is set.
- OpenRouter: stealth/ox-alpha (score 15) - anonymous stealth model, 1M ctx
  reasoning+vision. No AA/BenchLM coverage, so scored from LiveBench 2026-06-25
  (Overall 69.2), held below the confirmed open flagships for the methodology
  mismatch + unverified-stealth caveat. Provenance recorded in
  docs/open_model_scoring.md; provisional, revisit if it de-cloaks.
…gap audit)

An adhd gap audit of the native-provider changes found the unit tests only
checked flagship-per-provider, leaving cross-cutting invariants unproven.
Adds two files:

test_native_provider_wiring.py (route-independent, no network):
- Four-registry bijection: every new ProviderType is in PROVIDER_PRIORITY_ORDER,
  the key map, and both restriction maps (catches partial wiring).
- Positional priority: each native ranks strictly before OpenRouter.
- Cross-route score pinning: native conf score == OpenRouter mirror score for
  each mirrored model (drift detection across the two files).
- Per-model conf well-formedness for all 8 native confs (ctx/out/score/aliases,
  explicit supports_images bool, no duplicate alias within a conf).
- Vision flags explicit (vision-exp/ox-alpha True; non-vision False).
- Ox Alpha LiveBench score pin (15). Local R1 distills 1.5B-32B present/scored,
  70B excluded. Key-gating: each provider is dormant (get_provider None) with
  no key.

test_native_provider_live.py (deferred): one skipif-no-key live smoke per
native provider + an OpenRouter live test, marked `integration` so the default
suite excludes them. A census test fails if any native provider lacks a live
test, so "deferred" stays auditable. Verified: with a key, the OpenRouter live
tests pass (deepseek-v4-flash + ox-alpha); the 8 native ones skip with a reason.

955 passed (was 911), ruff/black/isort clean.
Unpinned `mcp>=1.0.0` resolved to mcp 2.1.1 in CI, which removed
`Server.list_tools`; 7 upstream test files (test_server.py,
test_conversation_memory.py, etc.) reference it and errored at collection
on Python 3.10/3.11/3.12. main has the same unpinned spec and fails the
same way. The server and those tests target the mcp 1.x API (local runs on
1.27.2), so pin to the 1.x line until the code is ported to mcp 2.x.
Same CI-reproducibility rationale as the black==25.1.0 pin already here.
- requirements.txt: raise the Anthropic SDK floor to >=0.49.0 (the version
  that exposes the `thinking` argument on messages.create). The provider
  passes `thinking` for non-minimal thinking modes, so the old 0.40.0 floor
  would fail at runtime on the advertised minimum. (codex P2)
- .env.example + docs/configuration.md: document ANTHROPIC_API_KEY and the
  8 new native providers (DEEPSEEK/DASHSCOPE/ZAI/MOONSHOT/MINIMAX/MISTRAL/
  NVIDIA/PERPLEXITY), with base URLs and a note that each is dormant unless
  keyed and ranks ahead of OpenRouter. Also notes the per-provider
  *_ALLOWED_MODELS / _DISALLOWED_MODELS / _MODELS_CONFIG_PATH vars. Users
  following the documented setup can now enable these otherwise-invisible
  providers. (codex P2, extended to cover all new natives, not just Anthropic)
@rpeck

rpeck commented Aug 28, 2026

Copy link
Copy Markdown
Author

@guidedways — this is ready for maintainer review whenever you have time. All 6 CI checks are green (lint, Validate PR, Docker, and test on 3.10/3.11/3.12) and it's mergeable.

State of the PR — five separable concerns, all opt-in with no default-behavior change:

  1. Model catalog refresh (OpenAI/Gemini + OpenRouter mirrors, benchmark-calibrated intelligence_scores).
  2. *_DISALLOWED_MODELS per-provider block-list.
  3. DYNAMIC_MODEL_SELECTION flag — default unset = upstream selection.
  4. Native Anthropic (Claude) provider — dormant unless ANTHROPIC_API_KEY is set.
  5. Native direct-API providers for 8 open-model creators (DeepSeek, Qwen, GLM/Z.ai, Kimi/Moonshot, MiniMax, Mistral, NVIDIA, Perplexity — all OpenAI-compatible), plus DeepSeek Vision Exp / R1 local distills / the Ox Alpha stealth model. Each native provider is dormant unless its *_API_KEY is set and ranks ahead of OpenRouter when present.

Bot feedback addressed: the two current chatgpt-codex P2s are fixed in d5cad49 (Anthropic SDK floor → anthropic>=0.49.0 for the thinking arg; and documenting the new providers in .env.example + docs/configuration.md); the older threads were superseded by earlier fixes. gemini-code-assist has been inactive repo-wide since mid-July, so I ran an equivalent review through a local model instead — it caught and I fixed two real bugs (see history).

CI note: I pinned mcp<2. Unpinned mcp>=1.0.0 was resolving to 2.1.1, which removed Server.list_tools and failed 7 upstream tests at collection (main fails the same way); the pin restores the 1.x API the code targets, same reproducibility rationale as the existing black==25.1.0 pin.

Testing: 955 unit tests pass. The native providers are unit-tested; their live API call is deferred behind skipif-no-key integration tests (I don't hold those keys) — the PENDING-TESTING section of the description lists them and gives a one-command way to run any you have a key for. Happy to split a piece out if that's easier to review. Thanks!

rpeck added 3 commits August 31, 2026 10:37
Zhipu shipped GLM-5.3-Flash (2026-08-26): native multimodal MoE, 320B total
/ 18B active, 1M context, AA Intelligence Index 57 (matches Claude Opus 4.8,
~7.5x cheaper than the GLM-5.3 flagship). Added both routes:
- OpenRouter z-ai/glm-5.3-flash and native Z.ai glm-5.3-flash, score 16
  (one below the glm-5.3 flagship's 17; pinned across routes), supports_images
  true (first multimodal in the GLM-5 series).
- The bare `glm-flash` alias moves to glm-5.3-flash on both routes (newest
  flash); glm-4.7-flash keeps its version-pinned alias.

Score-pin test added. Smoke-tested live via OpenRouter (round-trip OK).
956 passed; ruff/black clean.
…injection seals)

Qwen3.8-Flash:
- Production serving of the Flash-Next architecture (hosted SKU qwen3.8-flash).
- Native DashScope + OpenRouter mirror, score 16 pinned to AA Index 56
  (AA-56 -> 16, the gemini-3.7-flash precedent). Bare qwen-flash alias now
  points at it; multimodal (text/image/video), 1M context.

auto_selectable flag:
- New ModelCapabilities.auto_selectable (default true). When false, a model
  stays callable by explicit name and visible in listmodels, but is excluded
  from the auto-mode ranked candidate summaries.

OpenRouter Fusion meta-router:
- openrouter/fusion (score 17) plus named preset SKUs openrouter/fusion-quality
  (pinned Opus+GPT+Gemini-Pro panel) and openrouter/fusion-fast (vendor
  general-fast preset). All auto_selectable:false; never auto-picked.
- Cost/injection seals: panel config is sealed in conf _extras and injected
  server-side as extra_body.plugins, never from caller kwargs or prompt; the
  wire model is rewritten to the real openrouter/fusion slug; fan-out is capped
  at 8; usage.cost is surfaced as a spend receipt.

Tests: test_fusion_router (9, incl. injection/cost seals) + qwen score-pin and
vision/alias wiring. Full non-integration suite green.
Claude Fable 5.1:
- Native claude-fable-5-1 + OpenRouter anthropic/claude-fable-5.1, score 20
  (tops the Artificial Analysis Intelligence Index at 66, max effort, ahead of
  Opus 5 at 63). Score pinned across routes.
- Bare "fable" alias now resolves to 5.1 (newest-wins); "fable-5" still
  addresses Fable 5. Live-smoke-tested through OpenRouter.

Qwen3.8-Max-0902:
- Native DashScope snapshot qwen3.8-max-0902 (dated alias
  qwen3.8-max-2026-09-02), score 17. Held at the base-Max AA tier: the gain is
  coding-specific on Code Arena WebDev (#1, 1691), a separate benchmark with no
  published AA Intelligence Index. Native multimodal, 1M context.
- Bare "qwen-max"/"qwen3-max" now point at 0902; base qwen3.8-max keeps "qwen".
- OpenRouter mirror deferred until it lands there; native path deferred (no key).

Tests: Anthropic catalog/alias updated for 5.1, cross-route score-pin, and
wiring tests for both additions. Full non-integration suite green.
@jtwillis92

Copy link
Copy Markdown

The mcp<2 pin from 48027c8b only landed in requirements.txt, so it covers CI but not installs. pyproject.toml at d168bcb still declares mcp>=1.0.0, and that is what uvx --from git+https://github.com/rpeck/pal-mcp-server.git@<sha> resolves against, so a fresh install still pulls mcp 2.1.1 and crashes at import:

File ".../site-packages/server.py", line 684, in <module>
    @server.list_tools()
AttributeError: 'Server' object has no attribute 'list_tools'

The client only surfaces that as CONNECTION_CLOSED, so it reads as a config problem rather than a dependency one. Reproduced today on d168bcb and on 921cc52, ie it is not new to the recent commits.

Adding ,<2 to the pyproject.toml dependency would close it. In the meantime uvx --with "mcp<2" --from git+... pal-mcp-server works around it, which resolves mcp 1.29.1, boots clean, and returns 104 models from listmodels with gpt-5.6-sol routing correctly (model_used: gpt-5.6-sol, provider_used: openai).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants