You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e575ca7
Browse filesBrowse the repository at this point in the historyBrowse files
VAP15-76 docs: define each per-turn latency field and link it from sims, Traffic and logs
Add a "Per-turn latency on a call" section to Understanding latency that says
where each turnLatencies field starts and ends, and fix the stage order
(transcription runs before endpointing). Link it from the simulation latency
limits, Traffic tab, performance metrics, call logs and glossary pages, and
note why the stages don't add up to the turn total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: fern/assistants/model-intelligence/metrics-methodology.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,7 +30,7 @@ Vapi reports the median response time for each component based on production cal
30
30
The displayed total is simply the sum of the three component medians, so treat it as a rough estimate rather than an exact figure.
31
31
32
32
<Note>
33
-
The total excludes [endpointing](/customization/speech-configuration) and transport time.
33
+
The total excludes [endpointing](/customization/speech-configuration) and transport time. It isn't a call's turn latency. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for how each call measures latency.
34
34
</Note>
35
35
36
36
See [how latency works](/assistants/model-intelligence/understanding-latency) for help understanding latency and why more capable models tend to be slower.
Copy file name to clipboardExpand all lines: fern/assistants/model-intelligence/understanding-latency.mdx
+27-4Lines changed: 27 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,15 +9,36 @@ Voice-to-voice latency is the gap between when a caller finishes speaking and wh
9
9
10
10
## Voice-to-voice latency sequence
11
11
12
-
Voice-to-voice latency combines five stages that run in sequence. Each stage begins after the previous one.
12
+
Voice-to-voice latency combines five stages. After the caller stops speaking, each stage begins when the previous one ends.
13
13
14
14
1.**Transport:** The caller's audio travels over the network for processing.
15
-
2.**Endpointing:**Vapi detects that the caller has finished speaking.
16
-
3.**Transcription:**The transcriber converts the caller's speech to text.
15
+
2.**Transcription:**The transcriber finishes converting the caller's last words to text.
16
+
3.**Endpointing:**Vapi decides that the caller has finished their turn and sends the transcript to the model.
17
17
4.**Model response:** The model reads the transcript and begins generating a reply. Vapi measures this stage as time to first token.
18
18
5.**Speech synthesis:** The voice provider turns the reply into audio. Vapi measures this stage as time to first audio.
19
19
20
-
A caller's total voice-to-voice latency is roughly the sum of all five stages.
20
+
A caller's total voice-to-voice latency is roughly the sum of all five stages, plus the time the reply takes to reach the caller.
21
+
22
+
## Per-turn latency on a call
23
+
24
+
Each call records one latency entry per turn in `artifact.performanceMetrics.turnLatencies`. The call's **Latency Summary** tab, the [**Traffic** tab](/assistants/versioning/traffic-tab), and [simulation latency limits](/observability/simulations-advanced#set-latency-limits) are all built from these entries.
25
+
26
+
| Field | Starts when | Ends when |
27
+
| --- | --- | --- |
28
+
|`transcriberLatency`| The caller stops speaking, as detected by voice activity detection | The transcriber returns the words of that speech |
29
+
|`endpointingLatency`| The transcriber returns the words | Vapi decides the turn is over and sends it to the model |
30
+
|`modelLatency`| Vapi sends the turn's first model request | The model returns its first output token |
31
+
|`voiceLatency`| The model returns its first output token | The voice provider returns the first audio of the reply |
32
+
|`turnLatency`| The caller stops speaking | Vapi starts sending the reply audio to the caller |
33
+
34
+
Keep these points in mind when you read the numbers:
35
+
36
+
-**The fields don't add up to `turnLatency`.**`turnLatency` is `transcriberLatency` plus `endpointingLatency` plus the time from that decision to the reply starting. Model and voice latency happen inside that last stretch, alongside other work such as building the prompt, knowledge base lookups, and buffering the reply audio.
37
+
-**Voice latency includes waiting for text.** Vapi collects enough of the model's reply to send the first request to the voice provider, as set by `voice.chunkPlan` (`minCharacters` and punctuation boundaries), so changing the chunk plan changes `voiceLatency`.
38
+
-**Vapi measures on its servers.** None of the fields include network time to or from the caller, or playback on the caller's device.
39
+
-**Only replies to the caller are recorded.** The first message and turns where the assistant didn't speak, for example because the caller interrupted first, have no entry.
40
+
-**`0` can mean "not measured".** A stage that didn't run or wasn't measured on a turn is recorded as `0` rather than left out. For example, when a turn's first audio is a static tool message rather than model output, `modelLatency` and `voiceLatency` are `0`.
41
+
-**Averages count every turn.** The call's `modelLatencyAverage`, `voiceLatencyAverage`, and other averages divide by every entry, including those zeros. Simulation latency limits ignore a `0` for `modelLatency` and `voiceLatency`, so a simulation's model or voice result can be higher than the call's average for the same call.
21
42
22
43
## What the displayed latency measures
23
44
@@ -27,6 +48,8 @@ Vapi calculates each median from production calls, which reflect a typical call
27
48
28
49
The displayed total excludes [endpointing](/customization/speech-configuration) and transport time. Both contribute to the delay a caller experiences. Endpointing can account for a meaningful share of that delay. Improving endpointing can make an assistant feel more responsive even though it is not included in the displayed total.
29
50
51
+
The displayed total is a sum of component medians across many calls, so it isn't comparable to a call's `turnLatency`, which is measured end to end on each turn and includes endpointing. See [per-turn latency on a call](#per-turn-latency-on-a-call).
52
+
30
53
## Model capability and latency
31
54
32
55
More capable models usually respond more slowly. A model's time to first token is often the largest component of voice-to-voice latency. More capable models tend to be larger and take longer to begin generating a response.
Copy file name to clipboardExpand all lines: fern/assistants/versioning/traffic-tab.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -168,7 +168,7 @@ This table lists one row per version, with these columns:
168
168
169
169
-**Duration**: how long calls lasted.
170
170
-**Cost per minute**: the average cost of a minute of call. Expand it to see the parts: **LLM**, **STT**, **TTS**, **Vapi**, and **Transport**.
171
-
-**Turn latency**: how long the assistant took to start responding after the caller finished speaking, averaged over each call. Expand it to see the **Transcriber**, **Model**, and **Voice** stages.
171
+
-**Turn latency**: how long the assistant took to start responding after the caller finished speaking, averaged over each call. Expand it to see the **Transcriber**, **Model**, and **Voice** stages. The stages don't add up to the total: endpointing isn't shown, and model and voice latency happen inside the turn rather than after it. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call).
172
172
173
173
Use the **p50** and **p95** toggle to switch duration and latency between the typical call and the slow tail. Sort any column to rank the versions by it. Each bar shows the value against the largest value in its column, so the slowest or most expensive version is easy to spot.
Copy file name to clipboardExpand all lines: fern/glossary.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -104,7 +104,7 @@ An abbreviation used for "Text-to-speech". The process of converting raw text in
104
104
105
105
### Voice-to-Voice
106
106
107
-
"Voice-to-voice" is often a term brought up in discussing voice AI system latency — the time it takes to go from a user finishing their speech (however that endpoint is computed) → to the AI agent's first speech chunk/byte being played back on a client’s device.
107
+
"Voice-to-voice" is often a term brought up in discussing voice AI system latency — the time it takes to go from a user finishing their speech (however that endpoint is computed) → to the AI agent's first speech chunk/byte being played back on a client’s device. A call's `turnLatency` measures the part of this that happens on Vapi's servers; see [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call).
108
108
109
109
Ideally, this process should happen in \<1s, better if closer to 500-700ms (responding too quickly can be an issue as well). Voice AI applications must closely watch this metric to ensure their applications stay responsive & usable.
Copy file name to clipboardExpand all lines: fern/observability/logs/call-logs.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -109,7 +109,7 @@ Categories can include speech-to-text (STT), large language model (LLM), text-to
109
109
110
110
The **Latency Summary** tab shows where time was spent during the call. It includes the turn count, average turn latency, and a breakdown by stage. Stages include **Transcriber**, **Endpointing**, **LLM**, **Voice**, and transport in each direction.
111
111
112
-
Use the per-turn table to find which stage made the call feel slow. It shows each turn's total and per-stage milliseconds. The values are averages and per-turn measurements, not percentiles.
112
+
Use the per-turn table to find which stage made the call feel slow. It shows each turn's total and per-stage milliseconds. The values are averages and per-turn measurements, not percentiles. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for what each stage measures.
113
113
114
114
For the full object behind these views, see the [call object](/api-reference/calls/get). Recordings and artifacts are also covered in [Call recording](/assistants/call-recording) and [Retrieve call artifacts](/assistants/retrieve-call-artifacts).
|`turn`| Total time from the end of the AI tester's speech to the start of the assistant's reply |
329
-
|`model`| Time for the model to return its first token |
330
-
|`voice`| Time for the voice provider to return the first audio |
326
+
Each metric reads one field of the per-turn latency that every call records, measured the same way as on a live call. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for exactly where each one starts and ends.
327
+
328
+
| Metric | Field | What it measures |
329
+
| --- | --- | --- |
330
+
|`turn`|`turnLatency`| Time from the end of the AI tester's speech to the assistant starting its reply |
331
+
|`model`|`modelLatency`| Time from the turn's first model request to the model's first output token |
332
+
|`voice`|`voiceLatency`| Time from the model's first output token to the voice provider's first audio |
333
+
334
+
`model` and `voice` happen inside `turn`, so they don't add up to it. Turns that recorded `0` for `model` or `voice` didn't measure it and are left out of those metrics. The AI tester speaks with clean synthetic audio, so transcription, endpointing, and turn latency can run faster than with real callers.
331
335
332
336
The aggregation reduces the call's turns to one value: `mean`, `median`, `p95`, or `max`. `p95` uses the nearest-rank method, so on a call with fewer than 20 turns it equals `max`. Measured values are rounded to the nearest millisecond before they are compared with the limit.
0 commit comments