Skip to content

Commit 086c678

Browse files
akushonkamencodelion
authored andcommitted
feat(server): add legacy /v1/completions endpoint for the inbuilt inference server
The built-in inference server only exposed the chat protocol at /v1/chat/completions, so OpenAI-legacy clients (text completions) could not use it. Extract the request_config parsing, approach/model resolution, client building and dispatch of proxy() into a shared _execute_completion() pipeline, and add POST /v1/completions on top of it: - prompt (str or list[str]) maps to user messages; the pipeline runs once per prompt and n still expands attempts into multiple choices - returns the OpenAI legacy shape: id cmpl-*, object text_completion, choices[].text/logprobs/finish_reason and usage; passthrough completion dicts (none approach / full-response plugins / batch mode) are mapped choice-by-choice - stream=true returns 400 in this first version (chat endpoint streams exactly as before); optillm_approach, max_tokens and other params pass through like on the chat endpoint - /v1/chat/completions responses are unchanged; conversation logging, batch mode and error mapping (500 JSON) behave as before Fixes #170
1 parent 4088efc commit 086c678

3 files changed

Lines changed: 467 additions & 83 deletions

File tree

‎README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -375,6 +375,21 @@ response = client.chat.completions.create(
375375
)
376376
```
377377

378+
The server also exposes the legacy text completions endpoint `POST /v1/completions` (returning `object: "text_completion"` responses), so OpenAI-legacy clients work against the inbuilt inference server too:
379+
380+
```bash
381+
curl http://localhost:8000/v1/completions \
382+
-H "Content-Type: application/json" \
383+
-H "Authorization: Bearer $OPTILLM_API_KEY" \
384+
-d '{
385+
"model": "meta-llama/Llama-3.2-1B-Instruct",
386+
"prompt": "Write a haiku about recursion.",
387+
"max_tokens": 64
388+
}'
389+
```
390+
391+
Streaming (`stream: true`) is not supported on `/v1/completions` yet; use `/v1/chat/completions` for streaming responses.
392+
378393
### Starting the optillm proxy with an external server (e.g. llama.cpp or ollama)
379394

380395
- Set the `OPENAI_API_KEY` env variable to a placeholder value

0 commit comments

Comments
 (0)