Skip to content

Commit 1a72232

Browse files
committed
fix(catalog): advertise image input for modalities-declared sidecar rows
The runtime vision gate (isModelTextOnly) covers models listed in noVisionModels OR declared text-only via modelInputModalities (upstream fde2a95, #1024), but both catalog advertise sites only checked noVisionModels. A sidecar-covered model - and every combo built from it - stayed advertised text-only in /v1/models, so the Codex app blocked image attachments client-side before the sidecar could run ('This model does not support image inputs'). Mirror isModelTextOnly in applyProviderConfigHints and the custom-model override: a declared text-only modelInputModalities entry now advertises image on top of its configured base. Discovery-derived text-only rows stay untouched (the runtime predicate does not cover those), and declared-image rows are never duplicated. Combos inherit the fix through their hinted members; no config hand-editing needed. Update the three tests that encoded the old drift and add regressions for the hint pass, the custom-model override, and combo derivation.
1 parent e5d5886 commit 1a72232

4 files changed

Lines changed: 129 additions & 12 deletions

File tree

docs-site/src/content/docs/guides/sidecars.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -123,8 +123,11 @@ failures after response headers have started are delivered as `response.failed`
123123

124124
## Vision sidecar
125125

126-
When the routed model is listed in its provider's `noVisionModels` and a request carries an image,
127-
opencodex describes each image **before** the main call and replaces it with text. When
126+
When the routed model is listed in its provider's `noVisionModels` — or declared text-only for
127+
that model via `modelInputModalities` — and a request carries an image, opencodex describes each
128+
image **before** the main call and replaces it with text. The model catalog advertises image input
129+
for every sidecar-covered model (combos advertise image when every member is covered), so clients
130+
such as the Codex app allow attachments instead of blocking them before the sidecar runs. When
128131
`visionSidecar.model` is absent or blank, the OpenAI execution path, Dashboard, and management API
129132
use the `gpt-5.4-mini` fallback. Startup still migrates an explicitly persisted legacy
130133
`gpt-5.4-mini` value to `gpt-5.6-luna`; that migration applies to a stored value, not to an absent

src/codex/catalog/provider-fetch.ts

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -669,11 +669,16 @@ export function applyProviderConfigHints(name: string, prov: OcxProviderConfig,
669669
const configuredMaxInput = configuredMaxInputTokens(prov, model.id);
670670
const configuredAutoCompact = configuredAutoCompactTokenLimit(prov, model.id);
671671
let inputModalities = configuredInputModalities(prov, model.id);
672-
// Vision-sidecar coverage: `noVisionModels` marks models whose images the PROXY describes
673-
// (src/vision/index.ts). The catalog must still advertise image input for them — the Codex app
672+
// Vision-sidecar coverage mirrors isModelTextOnly (src/vision/index.ts): `noVisionModels` OR
673+
// a `modelInputModalities` declaration excluding "image" both mean the PROXY describes images
674+
// for this model at request time. The catalog must still advertise image input — the Codex app
674675
// gates attachments client-side on input_modalities, and a text-only entry would block images
675-
// before the sidecar ever runs ("This model does not support image inputs").
676-
if (modelInList(prov.noVisionModels, model.id)) {
676+
// before the sidecar ever runs ("This model does not support image inputs"). Discovery-derived
677+
// text-only rows stay untouched: the runtime predicate only reads these two config sources, so
678+
// it would not convert those.
679+
const sidecarCovered = modelInList(prov.noVisionModels, model.id)
680+
|| (Array.isArray(inputModalities) && inputModalities.length > 0 && !inputModalities.includes("image"));
681+
if (sidecarCovered) {
677682
const base = inputModalities ?? model.inputModalities ?? ["text"];
678683
inputModalities = base.includes("image") ? [...base] : [...base, "image"];
679684
}
@@ -2140,7 +2145,15 @@ async function gatherRoutedModelsUncached(
21402145
}
21412146
: mergedWithHardBounds;
21422147
const enrichedProvider = enrichedByName.get(cm.provider) ?? rawProvider;
2143-
if (enrichedProvider && modelInList(enrichedProvider.noVisionModels, mergedWithAutoCompact.id)) {
2148+
// Same vision-sidecar coverage rule as applyProviderConfigHints (isModelTextOnly parity):
2149+
// noVisionModels OR a text-only modelInputModalities declaration means the sidecar converts
2150+
// images at request time, so the custom row must advertise image input.
2151+
const declaredModalities = enrichedProvider
2152+
? modelRecordValue(enrichedProvider.modelInputModalities, mergedWithAutoCompact.id)
2153+
: undefined;
2154+
if (enrichedProvider
2155+
&& (modelInList(enrichedProvider.noVisionModels, mergedWithAutoCompact.id)
2156+
|| (Array.isArray(declaredModalities) && declaredModalities.length > 0 && !declaredModalities.includes("image")))) {
21442157
const current = mergedWithAutoCompact.inputModalities ?? ["text"];
21452158
if (!current.includes("image")) {
21462159
return { ...mergedWithAutoCompact, inputModalities: [...current, "image"] };

tests/catalog-vision-sidecar-modalities.test.ts

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,40 @@ describe("vision-sidecar catalog modalities", () => {
4848
expect(hinted.inputModalities).toEqual(["text", "image"]);
4949
});
5050

51+
test("modelInputModalities-declared text-only models advertise image without a noVisionModels entry", () => {
52+
// Regression: isModelTextOnly (the RUNTIME sidecar gate) treats a text-only
53+
// modelInputModalities declaration exactly like a noVisionModels entry, but the catalog hint
54+
// pass only checked noVisionModels — so sidecar-covered models stayed advertised text-only
55+
// and the Codex app blocked image paste before the sidecar could run.
56+
const prov: OcxProviderConfig = {
57+
adapter: "openai-chat",
58+
baseUrl: "https://api.example/v1",
59+
modelInputModalities: { "deepseek-chat": ["text"] },
60+
};
61+
const hinted = applyProviderConfigHints("azu-lab2", prov, { id: "deepseek-chat", provider: "azu-lab2" });
62+
expect(hinted.inputModalities).toEqual(["text", "image"]);
63+
});
64+
65+
test("modelInputModalities declaring image stays untouched (no duplication)", () => {
66+
const prov: OcxProviderConfig = {
67+
adapter: "openai-chat",
68+
baseUrl: "https://api.example/v1",
69+
modelInputModalities: { "glm-5.3": ["text", "image"] },
70+
};
71+
const hinted = applyProviderConfigHints("vdi", prov, { id: "glm-5.3", provider: "vdi" });
72+
expect(hinted.inputModalities).toEqual(["text", "image"]);
73+
});
74+
75+
test("discovery-derived text-only rows are NOT advertised image (the runtime would not convert them)", () => {
76+
// Only the two config sources the runtime predicate reads (noVisionModels,
77+
// modelInputModalities) may widen the catalog; a listing that merely reports
78+
// ["text"] without either must stay text-only.
79+
const hinted = applyProviderConfigHints("opencode-go", base, {
80+
id: "listing-text-model", provider: "opencode-go", inputModalities: ["text"],
81+
});
82+
expect(hinted.inputModalities).toEqual(["text"]);
83+
});
84+
5185
test("MiMo token-plan sends only the Pro model through the sidecar (#1927)", () => {
5286
const canonical: OcxProviderConfig = {
5387
adapter: "openai-chat",
@@ -173,6 +207,39 @@ describe("vision-sidecar custom-model override (#349/#344)", () => {
173207
globalThis.fetch = originalFetch;
174208
}
175209
});
210+
211+
test("a custom row whose modelId is declared text-only via modelInputModalities still advertises image", async () => {
212+
// Same isModelTextOnly parity as the hint pass, applied to the custom-model override path:
213+
// the registry/config text-only declaration covers the row at request time, so the catalog
214+
// must let images through to the sidecar here too.
215+
const originalFetch = globalThis.fetch;
216+
globalThis.fetch = (() => { throw new Error("fetch should not be called"); }) as typeof fetch;
217+
try {
218+
const models = await gatherRoutedModels({
219+
port: 10100,
220+
defaultProvider: "text-sidecar-provider",
221+
providers: {
222+
"text-sidecar-provider": {
223+
baseUrl: "https://text-sidecar.example/v1",
224+
adapter: "openai-chat",
225+
authMode: "key",
226+
liveModels: false,
227+
models: ["baseline-model"],
228+
modelInputModalities: { "glm-5.2": ["text"] },
229+
},
230+
},
231+
customModels: [
232+
{ id: "cm-4", provider: "text-sidecar-provider", modelId: "glm-5.2", displayName: "GLM 5.2", addedAt: "2026-01-01T00:00:00.000Z" },
233+
],
234+
});
235+
const custom = models.find(m => m.provider === "text-sidecar-provider" && m.id === "glm-5.2");
236+
expect(custom).toBeDefined();
237+
expect(custom?.inputModalities).toEqual(["text", "image"]);
238+
} finally {
239+
globalThis.fetch = originalFetch;
240+
clearModelCache("text-sidecar-provider");
241+
}
242+
});
176243
});
177244

178245
describe("vision-capable provider models feed combo modalities", () => {
@@ -258,6 +325,31 @@ describe("vision-capable provider models feed combo modalities", () => {
258325
const hinted = applyProviderConfigHints("xai", prov, { provider: "xai", id: "grok-4.5" });
259326
expect(hinted.inputModalities).toEqual(["text", "image"]);
260327
});
328+
329+
test("a combo of modelInputModalities-declared text-only members advertises image through the hints", () => {
330+
// End-to-end shape of the /planners bug: every member is sidecar-covered via
331+
// modelInputModalities (not noVisionModels). The hinted members all carry image, so the
332+
// derived combo keeps image input instead of collapsing to text-only.
333+
const memberFor = (provider: string, id: string): CatalogModel => applyProviderConfigHints(provider, {
334+
adapter: "openai-chat",
335+
baseUrl: `https://${provider}.example/v1`,
336+
modelInputModalities: { [id]: ["text"] },
337+
}, { id, provider, contextWindow: 200_000 });
338+
const memberA = memberFor("azu-lab2", "deepseek-chat");
339+
const memberB = memberFor("vdi", "glm-5.2");
340+
const derived = deriveComboCatalogModel(
341+
"planners",
342+
{
343+
targets: [
344+
{ provider: "azu-lab2", model: "deepseek-chat" },
345+
{ provider: "vdi", model: "glm-5.2" },
346+
],
347+
defaultEffort: "high",
348+
} as never,
349+
[memberA, memberB],
350+
);
351+
expect(derived?.inputModalities).toEqual(["text", "image"]);
352+
});
261353
});
262354

263355
describe("Cursor native vs sidecar vision registry", () => {

tests/codex-catalog.test.ts

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1061,7 +1061,9 @@ describe("combo catalog capability intersection", () => {
10611061
liveModels: false,
10621062
models: ["m2"],
10631063
modelContextWindows: { m2: 128_000 },
1064-
modelInputModalities: { m2: ["text"] },
1064+
// No declaration: text-only by catalog default, NOT sidecar-covered. A declared
1065+
// text-only member would widen to image (isModelTextOnly parity) and the pair
1066+
// would no longer be disjoint.
10651067
},
10661068
},
10671069
combos: {
@@ -1202,7 +1204,9 @@ describe("combo catalog capability intersection", () => {
12021204
contextWindow: 128_000,
12031205
contextCapped: false,
12041206
maxInputTokens: 100_000,
1205-
inputModalities: ["text"],
1207+
// The text-only modelInputModalities declaration is sidecar-covered at runtime
1208+
// (isModelTextOnly), so the combo advertises image input like its member does.
1209+
inputModalities: ["text", "image"],
12061210
reasoningEfforts: ["high"],
12071211
});
12081212
expect(rows.find(row => row.provider === "combo" && row.id === "nova-sol"))
@@ -1515,7 +1519,7 @@ describe("combo catalog capability intersection", () => {
15151519
liveModels: false,
15161520
models: ["m1"],
15171521
modelContextWindows: { m1: 128_000 },
1518-
// Disjoint modalities with b → empty intersection (incompatible_modalities).
1522+
// Image-only member. An image-only declaration is not sidecar-covered.
15191523
modelInputModalities: { m1: ["image"] },
15201524
},
15211525
b: {
@@ -1524,7 +1528,10 @@ describe("combo catalog capability intersection", () => {
15241528
liveModels: false,
15251529
models: ["m2"],
15261530
modelContextWindows: { m2: 128_000 },
1527-
modelInputModalities: { m2: ["audio"] },
1531+
// No modality declaration: the member is text-only by catalog default and the
1532+
// sidecar does not cover it, so text and image stay genuinely disjoint
1533+
// (incompatible_modalities). A DECLARED text-only member would be widened to
1534+
// image (isModelTextOnly parity) and no longer be disjoint.
15281535
},
15291536
},
15301537
combos: {
@@ -5185,7 +5192,9 @@ describe("Codex catalog routed normalization", () => {
51855192
expect(models.find(m => m.id === "wide-model")).toMatchObject({
51865193
contextWindow: 100_000,
51875194
maxInputTokens: 100_000,
5188-
inputModalities: ["text"],
5195+
// Declared text-only modalities are sidecar-covered at runtime (isModelTextOnly),
5196+
// so the catalog advertises image on top of the configured base.
5197+
inputModalities: ["text", "image"],
51895198
});
51905199
expect(models.find(m => m.id === "small-model")?.contextWindow).toBe(64_000);
51915200
});

0 commit comments

Comments
 (0)