Skip to content

Commit e575ca7

Browse files
VAP15-76 docs: define each per-turn latency field and link it from sims, Traffic and logs
Add a "Per-turn latency on a call" section to Understanding latency that says where each turnLatencies field starts and ends, and fix the stage order (transcription runs before endpointing). Link it from the simulation latency limits, Traffic tab, performance metrics, call logs and glossary pages, and note why the stages don't add up to the turn total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent aec982c commit e575ca7

6 files changed

Lines changed: 40 additions & 13 deletions

File tree

‎fern/assistants/model-intelligence/metrics-methodology.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ Vapi reports the median response time for each component based on production cal
3030
The displayed total is simply the sum of the three component medians, so treat it as a rough estimate rather than an exact figure.
3131

3232
<Note>
33-
The total excludes [endpointing](/customization/speech-configuration) and transport time.
33+
The total excludes [endpointing](/customization/speech-configuration) and transport time. It isn't a call's turn latency. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for how each call measures latency.
3434
</Note>
3535

3636
See [how latency works](/assistants/model-intelligence/understanding-latency) for help understanding latency and why more capable models tend to be slower.

‎fern/assistants/model-intelligence/understanding-latency.mdx‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,15 +9,36 @@ Voice-to-voice latency is the gap between when a caller finishes speaking and wh
99

1010
## Voice-to-voice latency sequence
1111

12-
Voice-to-voice latency combines five stages that run in sequence. Each stage begins after the previous one.
12+
Voice-to-voice latency combines five stages. After the caller stops speaking, each stage begins when the previous one ends.
1313

1414
1. **Transport:** The caller's audio travels over the network for processing.
15-
2. **Endpointing:** Vapi detects that the caller has finished speaking.
16-
3. **Transcription:** The transcriber converts the caller's speech to text.
15+
2. **Transcription:** The transcriber finishes converting the caller's last words to text.
16+
3. **Endpointing:** Vapi decides that the caller has finished their turn and sends the transcript to the model.
1717
4. **Model response:** The model reads the transcript and begins generating a reply. Vapi measures this stage as time to first token.
1818
5. **Speech synthesis:** The voice provider turns the reply into audio. Vapi measures this stage as time to first audio.
1919

20-
A caller's total voice-to-voice latency is roughly the sum of all five stages.
20+
A caller's total voice-to-voice latency is roughly the sum of all five stages, plus the time the reply takes to reach the caller.
21+
22+
## Per-turn latency on a call
23+
24+
Each call records one latency entry per turn in `artifact.performanceMetrics.turnLatencies`. The call's **Latency Summary** tab, the [**Traffic** tab](/assistants/versioning/traffic-tab), and [simulation latency limits](/observability/simulations-advanced#set-latency-limits) are all built from these entries.
25+
26+
| Field | Starts when | Ends when |
27+
| --- | --- | --- |
28+
| `transcriberLatency` | The caller stops speaking, as detected by voice activity detection | The transcriber returns the words of that speech |
29+
| `endpointingLatency` | The transcriber returns the words | Vapi decides the turn is over and sends it to the model |
30+
| `modelLatency` | Vapi sends the turn's first model request | The model returns its first output token |
31+
| `voiceLatency` | The model returns its first output token | The voice provider returns the first audio of the reply |
32+
| `turnLatency` | The caller stops speaking | Vapi starts sending the reply audio to the caller |
33+
34+
Keep these points in mind when you read the numbers:
35+
36+
- **The fields don't add up to `turnLatency`.** `turnLatency` is `transcriberLatency` plus `endpointingLatency` plus the time from that decision to the reply starting. Model and voice latency happen inside that last stretch, alongside other work such as building the prompt, knowledge base lookups, and buffering the reply audio.
37+
- **Voice latency includes waiting for text.** Vapi collects enough of the model's reply to send the first request to the voice provider, as set by `voice.chunkPlan` (`minCharacters` and punctuation boundaries), so changing the chunk plan changes `voiceLatency`.
38+
- **Vapi measures on its servers.** None of the fields include network time to or from the caller, or playback on the caller's device.
39+
- **Only replies to the caller are recorded.** The first message and turns where the assistant didn't speak, for example because the caller interrupted first, have no entry.
40+
- **`0` can mean "not measured".** A stage that didn't run or wasn't measured on a turn is recorded as `0` rather than left out. For example, when a turn's first audio is a static tool message rather than model output, `modelLatency` and `voiceLatency` are `0`.
41+
- **Averages count every turn.** The call's `modelLatencyAverage`, `voiceLatencyAverage`, and other averages divide by every entry, including those zeros. Simulation latency limits ignore a `0` for `modelLatency` and `voiceLatency`, so a simulation's model or voice result can be higher than the call's average for the same call.
2142

2243
## What the displayed latency measures
2344

@@ -27,6 +48,8 @@ Vapi calculates each median from production calls, which reflect a typical call
2748

2849
The displayed total excludes [endpointing](/customization/speech-configuration) and transport time. Both contribute to the delay a caller experiences. Endpointing can account for a meaningful share of that delay. Improving endpointing can make an assistant feel more responsive even though it is not included in the displayed total.
2950

51+
The displayed total is a sum of component medians across many calls, so it isn't comparable to a call's `turnLatency`, which is measured end to end on each turn and includes endpointing. See [per-turn latency on a call](#per-turn-latency-on-a-call).
52+
3053
## Model capability and latency
3154

3255
More capable models usually respond more slowly. A model's time to first token is often the largest component of voice-to-voice latency. More capable models tend to be larger and take longer to begin generating a response.

‎fern/assistants/versioning/traffic-tab.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,7 +168,7 @@ This table lists one row per version, with these columns:
168168

169169
- **Duration**: how long calls lasted.
170170
- **Cost per minute**: the average cost of a minute of call. Expand it to see the parts: **LLM**, **STT**, **TTS**, **Vapi**, and **Transport**.
171-
- **Turn latency**: how long the assistant took to start responding after the caller finished speaking, averaged over each call. Expand it to see the **Transcriber**, **Model**, and **Voice** stages.
171+
- **Turn latency**: how long the assistant took to start responding after the caller finished speaking, averaged over each call. Expand it to see the **Transcriber**, **Model**, and **Voice** stages. The stages don't add up to the total: endpointing isn't shown, and model and voice latency happen inside the turn rather than after it. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call).
172172

173173
Use the **p50** and **p95** toggle to switch duration and latency between the typical call and the slow tail. Sort any column to rank the versions by it. Each bar shows the value against the largest value in its column, so the slowest or most expensive version is easy to spot.
174174

‎fern/glossary.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ An abbreviation used for "Text-to-speech". The process of converting raw text in
104104

105105
### Voice-to-Voice
106106

107-
"Voice-to-voice" is often a term brought up in discussing voice AI system latency — the time it takes to go from a user finishing their speech (however that endpoint is computed) → to the AI agent's first speech chunk/byte being played back on a client’s device.
107+
"Voice-to-voice" is often a term brought up in discussing voice AI system latency — the time it takes to go from a user finishing their speech (however that endpoint is computed) → to the AI agent's first speech chunk/byte being played back on a client’s device. A call's `turnLatency` measures the part of this that happens on Vapi's servers; see [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call).
108108

109109
Ideally, this process should happen in \<1s, better if closer to 500-700ms (responding too quickly can be an issue as well). Voice AI applications must closely watch this metric to ensure their applications stay responsive & usable.
110110

‎fern/observability/logs/call-logs.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,7 @@ Categories can include speech-to-text (STT), large language model (LLM), text-to
109109

110110
The **Latency Summary** tab shows where time was spent during the call. It includes the turn count, average turn latency, and a breakdown by stage. Stages include **Transcriber**, **Endpointing**, **LLM**, **Voice**, and transport in each direction.
111111

112-
Use the per-turn table to find which stage made the call feel slow. It shows each turn's total and per-stage milliseconds. The values are averages and per-turn measurements, not percentiles.
112+
Use the per-turn table to find which stage made the call feel slow. It shows each turn's total and per-stage milliseconds. The values are averages and per-turn measurements, not percentiles. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for what each stage measures.
113113

114114
For the full object behind these views, see the [call object](/api-reference/calls/get). Recordings and artifacts are also covered in [Call recording](/assistants/call-recording) and [Retrieve call artifacts](/assistants/retrieve-call-artifacts).
115115

‎fern/observability/simulations-advanced.mdx‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -323,11 +323,15 @@ curl -X PATCH "https://api.vapi.ai/eval/simulation/scenario/<scenario-id>" \
323323

324324
### Metrics and aggregations
325325

326-
| Metric | What it measures |
327-
| --- | --- |
328-
| `turn` | Total time from the end of the AI tester's speech to the start of the assistant's reply |
329-
| `model` | Time for the model to return its first token |
330-
| `voice` | Time for the voice provider to return the first audio |
326+
Each metric reads one field of the per-turn latency that every call records, measured the same way as on a live call. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for exactly where each one starts and ends.
327+
328+
| Metric | Field | What it measures |
329+
| --- | --- | --- |
330+
| `turn` | `turnLatency` | Time from the end of the AI tester's speech to the assistant starting its reply |
331+
| `model` | `modelLatency` | Time from the turn's first model request to the model's first output token |
332+
| `voice` | `voiceLatency` | Time from the model's first output token to the voice provider's first audio |
333+
334+
`model` and `voice` happen inside `turn`, so they don't add up to it. Turns that recorded `0` for `model` or `voice` didn't measure it and are left out of those metrics. The AI tester speaks with clean synthetic audio, so transcription, endpointing, and turn latency can run faster than with real callers.
331335

332336
The aggregation reduces the call's turns to one value: `mean`, `median`, `p95`, or `max`. `p95` uses the nearest-rank method, so on a call with fewer than 20 turns it equals `max`. Measured values are rounded to the nearest millisecond before they are compared with the limit.
333337

0 commit comments

Comments
 (0)