Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions docs/providers/lm-studio-http.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# LM Studio HTTP compatibility

Zoo Code discovers models without `@lmstudio/sdk`. It tries these endpoints in order:

| Endpoint | Available metadata |
| --------------------------------------- | ------------------------------------------------------------------------------------------ |
| `GET /api/v1/models` (LM Studio 0.4.0+) | LLM catalog keys, vision capability, maximum context, loaded instances and runtime context |
| `GET /api/v0/models` (LM Studio 0.3.6+) | LLM/VLM IDs, vision inferred from `vlm`, maximum context |
| `GET /v1/models` | IDs with Zoo Code's existing default metadata |

HTTP 404 or 405 starts the next discovery endpoint. HTTP 200 with a body of `{"error": "<message>"}` also starts it. Authentication errors, network failures, timeouts, cancellation and other invalid bodies do not.

All three discovery requests share a 15-second deadline. Explicit model loading has a separate 10-minute timeout. A failed refresh clears cached loaded-detail flags and instance aliases for that server; caller cancellation preserves them.
Older APIs do not expose the native model-load operation; chat/JIT remains responsible for loading on those servers.

Sources: [native API changelog](https://lmstudio.ai/docs/developer/api-changelog), [legacy API reference](https://lmstudio.ai/docs/developer/rest/endpoints).

## Saved selections and instances

Saved `lmStudioModelId` values are preserved. Previous SDK discovery exposed downloaded file paths and loaded model keys, so either may be saved. A catalog miss uses default metadata and sends the original ID to chat. If chat rejects the model selection, Zoo Code asks the user to reselect a model without changing their saved setting.

Native REST catalog keys and loaded instance IDs are distinct. A separate server-scoped runtime map retains exact instance IDs as aliases; the dropdown receives only catalog keys. These aliases include custom instance names and names created before another quantization was installed. For a selected catalog key, use the instance whose ID matches the key, or the first instance in server order. For an instance alias, use that instance's context length. There is no substring or display-name mapping from legacy paths to catalog keys.

Native loading checks the selected model's `loaded_instances` first. If none exist, it posts the resolved catalog key to `/api/v1/models/load`. Concurrent callers acquire a shared operation for the normalized server URL and saved ID before discovery; discovery, loading, and cache refresh all run inside it. An unloaded model resolves only by its catalog key; instance aliases already refer to loaded models. The shared operation is cleared on success or failure. This coordinates callers within this extension process, not other clients of the server.

## Verified migration limits

A local probe on Linux with llmster **0.0.27+1**, llama.cpp CPU runtime **2.66.0**, and SmolLM2 135M established:

- Two identical REST load calls created separate instances; loading is not idempotent.
- REST `key` matched the SDK's downloaded `modelKey`, but not its file `path`.
- Installing Q4_K_M alongside Q8_0 added quantization suffixes to catalog keys; already-loaded instances retained their previous keys/IDs.
- REST loading rejected the SDK file path with HTTP 404, but `/v1/chat/completions` accepted that same path with HTTP 200 while the model was loaded.

The chat result supports preserving legacy selections; it does not establish path compatibility across all LM Studio releases, model formats, or JIT/unloaded configurations. Legacy HTTP fallbacks have focused response-contract tests, not live coverage against historical server binaries.
1 change: 0 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,6 @@
"@types/react-dom": "18.3.7",
"csstype": "3.2.3",
"zod": "4.6.7",
"@lmstudio/sdk>zod": "3.25.76",
"katex": "0.19.1"
}
}
Expand Down
42 changes: 0 additions & 42 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion renovate.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@
"/^@ai-sdk\\//",
"/^@anthropic-ai\\//",
"/^@aws-sdk\\//",
"/^@lmstudio\\//",
"/^@mistralai\\//",
"/^@openrouter\\//",
"/^@qdrant\\//"
Expand Down
70 changes: 70 additions & 0 deletions src/api/providers/__tests__/lm-studio.spec.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// npx vitest run api/providers/__tests__/lm-studio.spec.ts

import { LmStudioHandler } from "../lm-studio"
import * as lmStudioFetcher from "../fetchers/lmstudio"
import { ApiHandlerOptions } from "../../../shared/api"

// Mock the timeout config utility
Expand Down Expand Up @@ -89,6 +90,75 @@ describe("LmStudioHandler abort wiring", () => {
}
})

it("resolves saved instance metadata separately from the dropdown catalog", () => {
const aliasInfo = { contextWindow: 4096, supportsImages: false, supportsPromptCache: true }
const aliasSpy = vitest.spyOn(lmStudioFetcher, "getLMStudioModelAlias").mockReturnValue(aliasInfo)
try {
const handler = new LmStudioHandler({ ...options, lmStudioModelId: "custom-instance" })
expect(handler.getModel()).toEqual({ id: "custom-instance", info: aliasInfo })
expect(aliasSpy).toHaveBeenCalledWith("custom-instance", options.lmStudioBaseUrl)
} finally {
aliasSpy.mockRestore()
}
})

it("preserves a saved path ID when catalog metadata is unavailable", () => {
const savedId = "lmstudio-community/SmolLM2/SmolLM2-Q8_0.gguf"
const handler = new LmStudioHandler({ ...options, lmStudioModelId: savedId })
expect(handler.getModel().id).toBe(savedId)
expect(handler.getModel().info).toBeDefined()
})

describe.each(["stream", "complete"] as const)("%s model selection errors", (mode) => {
it.each([
{ status: 404, message: "Model old/path.gguf not found", code: undefined, targeted: true },
{ status: 400, message: "No models found", code: undefined, targeted: true },
{ status: 404, message: "Unavailable", code: "model_not_found", targeted: true },
{ status: 401, message: "Unauthorized", code: undefined, targeted: false },
{ status: 400, message: "Context length exceeded", code: undefined, targeted: false },
{ status: 404, message: "Endpoint not found", code: undefined, targeted: false },
])(
"handles $status $message without rewriting the saved selection",
async ({ status, message, code, targeted }) => {
const handler = new LmStudioHandler(options)
vitest.spyOn(handler, "countTokens").mockResolvedValue(1)
lastCreate().mockRejectedValue(Object.assign(new Error(message), { status, error: { code } }))
const request =
mode === "stream"
? collectStream(handler.createMessage("system", []))
: handler.completePrompt("hello")
await expect(request).rejects.toThrow(
targeted ? "Reselect a model" : "Please check the LM Studio developer logs",
)
expect(options.lmStudioModelId).toBe("llama2")
},
)

it("tells the user to load a model when the server has none loaded", async () => {
const handler = new LmStudioHandler(options)
vitest.spyOn(handler, "countTokens").mockResolvedValue(1)
lastCreate().mockRejectedValue(Object.assign(new Error("No models loaded"), { status: 400 }))
const request =
mode === "stream" ? collectStream(handler.createMessage("system", [])) : handler.completePrompt("hello")
await expect(request).rejects.toThrow("Load a model in LM Studio")
expect(options.lmStudioModelId).toBe("llama2")
})

it("passes a saved path to successful chat requests unchanged", async () => {
const savedId = "lmstudio-community/SmolLM2/SmolLM2-Q8_0.gguf"
const handler = new LmStudioHandler({ ...options, lmStudioModelId: savedId })
vitest.spyOn(handler, "countTokens").mockResolvedValue(1)
lastCreate().mockResolvedValue(
mode === "stream"
? asyncStreamFrom([{ choices: [{ delta: { content: "hello" } }] }])
: { choices: [{ message: { content: "hello" } }] },
)
if (mode === "stream") await collectStream(handler.createMessage("system", []))
else expect(await handler.completePrompt("hello")).toBe("hello")
expect(lastCreate().mock.calls[0][0]).toMatchObject({ model: savedId })
})
})

describe("createMessage", () => {
it("passes a request-local AbortSignal to the SDK and bridges the external signal", async () => {
const handler = new LmStudioHandler(options)
Expand Down
10 changes: 10 additions & 0 deletions src/api/providers/__tests__/lmstudio.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,16 @@ describe("LmStudioHandler", () => {
mockGetModelsFromCache.mockReturnValue(undefined)
})

it.each([undefined, "", "http://localhost:1234/"])("normalizes handler URL %s", async (lmStudioBaseUrl) => {
const normalized = new LmStudioHandler({ ...mockOptions, lmStudioBaseUrl })
mockGetModels.mockResolvedValue({})
await normalized.ensureModelFetched()
normalized.getModel()
const scope = { provider: providerIdentifiers.lmstudio, baseUrl: "http://localhost:1234" }
expect(mockGetModels).toHaveBeenCalledWith(scope, expect.anything())
expect(mockGetModelsFromCache).toHaveBeenCalledWith(scope, normalized)
})

describe("constructor", () => {
it("should initialize with provided options", () => {
expect(handler).toBeInstanceOf(LmStudioHandler)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,14 +1,61 @@
{
"mistralai/devstral-small-2505": {
"type": "llm",
"modelKey": "mistralai/devstral-small-2505",
"format": "safetensors",
"displayName": "Devstral Small 2505",
"path": "mistralai/devstral-small-2505",
"sizeBytes": 13277565112,
"architecture": "mistral",
"vision": false,
"trainedForToolUse": false,
"maxContextLength": 131072
}
"models": [
{
"type": "llm",
"publisher": "lmstudio-community",
"key": "smollm2-135m-instruct@q8_0",
"display_name": "SmolLM2 135M Instruct",
"architecture": "llama",
"quantization": {
"name": "Q8_0",
"bits_per_weight": 8
},
"size_bytes": 144811072,
"params_string": "135M",
"loaded_instances": [
{
"id": "smollm2-135m-instruct",
"config": {
"context_length": 1024,
"eval_batch_size": 2048,
"physical_batch_size": 512,
"parallel": 4,
"flash_attention": false,
"lazy_mode": "auto",
"context_checkpoints": 32,
"reasoning_budget_message": "",
"speculative_draft_mtp": false,
"speculative_draft_simple": false,
"speculative_draft_model": "",
"speculative_draft_max_tokens": 3,
"speculative_draft_min_tokens": 0,
"speculative_draft_min_continue_probability": 0,
"offload_kv_cache_to_gpu": true
}
}
],
"max_context_length": 8192,
"format": "gguf",
"capabilities": {
"vision": false,
"trained_for_tool_use": false
},
"description": null
},
{
"type": "embedding",
"publisher": "nomic-ai",
"key": "text-embedding-nomic-embed-text-v1.5",
"display_name": "Nomic Embed Text v1.5",
"quantization": {
"name": "Q4_K_M",
"bits_per_weight": 4
},
"size_bytes": 84106624,
"params_string": null,
"loaded_instances": [],
"max_context_length": 2048,
"format": "gguf"
}
]
}
Loading
Loading