Studio: show prompt-processing progress in API monitor - #10834
Conversation
for more information, see https://pre-commit.ci
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f2eba9831
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 691c372f2b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ) | ||
| # Only a real llama.cpp prompt_progress frame means prefill is active. | ||
| # Final timing metadata must not move a decoding request back to this phase. | ||
| prompt_progress = timings.get("prompt_progress") if isinstance(timings, dict) else None |
There was a problem hiding this comment.
This only reads prompt_progress nested inside timings, which _report_live_llama_timings synthesizes for backend-owned generators. Raw llama-server proxy paths instead feed frames through _monitor_openai_sse_event and _monitor_openai_chunk (for example, streaming /v1/completions at inference.py:26952-26956), where llama.cpp's prompt_progress remains a top-level sibling and _monitor_openai_chunk forwards only data["timings"]; consequently these GGUF API requests never set running_phase and remain displayed as Running throughout prefill. Pass the top-level progress object into the monitor path as well.
Useful? React with 👍 / 👎.
| entry is None | ||
| or entry.status != "running" | ||
| or entry.kind != "request" | ||
| or entry.running_phase == "token_generation" |
There was a problem hiding this comment.
When a server-side tool loop starts a second model turn, this guard drops every new progress frame unless something first clears token_generation. The OpenAI GGUF loop only performs that reset behind _ui_events (inference.py:23079-23080), which is false for ordinary API streams, and the Anthropic server-tool loop has no corresponding reset, so their second and later prefills continue to display Token generation. Reset the phase at every internal tool boundary independently of whether a UI control card is emitted, or otherwise identify a new prefill round before applying this guard.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 691c372f2b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| prompt_progress = timings.get("prompt_progress") if isinstance(timings, dict) else None | ||
| if monitor_id and isinstance(prompt_progress, dict): |
There was a problem hiding this comment.
When a supported non-streaming GGUF /v1/chat/completions request uses n > 1, the standard chat handler sets _gguf_perf_callback to None at inference.py:22626-22630 and passes it to every generated choice. Consequently no live sample can reach this new extraction path, so long multi-choice prefills remain displayed as generic Running rather than Prompt processing. Keep request-wide final timing aggregation disabled if necessary, but provide a separate progress callback and reset it between choices.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds live inference phase reporting to the Studio API monitor.
Running GGUF requests now show:
Prompt processing · N%while llama.cpp is processing the input promptToken generationonce decoded output beginsThe existing request lifecycle status (
running,completed,error, etc.) remains unchanged.Motivation
For large prompts, the API monitor previously showed only
running, so it was impossible to tell whether a request was waiting, processing the prompt, or already generating tokens.llama.cpp already exposes live
prompt_progressinformation. This change surfaces that existing telemetry in the API monitor instead of inferring the phase from throughput.Implementation
prompt_progressevents in the request-scoped performance callback.ApiMonitorEntry.processed / total.token_generationwhen the first decoded model output arrives.No OpenAI-compatible response or SSE payloads are changed.
Compatibility
Backends that do not expose live prompt progress continue to behave as before and fall back to the generic running state.
The percentage is currently available for llama.cpp/GGUF requests where
prompt_progressis provided.Validation
npm run build— passedPrompt processing · N%Token generation