You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bb66adc
Browse filesBrowse the repository at this point in the historyBrowse files
TEST-273 docs: document transcriber and endpointing simulation latency limits and the unmeasured flag
Add the transcriber and endpointing metrics to simulation latency limits, and
explain the per-turn unmeasured list that decides which turns count. Merge
after VapiAI/vapi#25252, which adds the metrics.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: fern/assistants/model-intelligence/understanding-latency.mdx
+8-2Lines changed: 8 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,8 +37,14 @@ Keep these points in mind when you read the numbers:
37
37
-**Model and voice latency include more than the provider.** The `modelLatency` field includes knowledge base retrieval and request routing before the provider request. The `voiceLatency` field includes the time spent collecting enough of the reply to send the first voice request. The `minCharacters` and `punctuationBoundaries` settings in `voice.chunkPlan` control that wait. The field also includes any sample-rate conversion after the provider returns audio.
38
38
-**Vapi measures on its servers.** None of the fields include network time to or from the caller, or playback on the caller's device.
39
39
-**Only replies to the caller are recorded.** Turns where the assistant didn't speak, for example because the caller interrupted first, have no entry. When the assistant speaks first, its first message has no entry either. When it waits for the caller (`firstMessageMode` set to `assistant-waits-for-user`), the first message is recorded as a reply, with `endpointingLatency` as `0`.
40
-
-**A value of `0` can mean "not measured".** A stage that didn't run or wasn't measured on a turn is recorded as `0` rather than left out. For example, when a turn's first audio is a static tool message rather than model output, `modelLatency` and `voiceLatency` are `0`. When `transcriberLatency` is `0` because it wasn't measured, `turnLatency` leaves that interval out too.
41
-
-**Averages count every turn.** The call's `modelLatencyAverage`, `voiceLatencyAverage`, and other averages divide by every entry, including those zeros. Simulation latency limits ignore a `0` for `modelLatency` and `voiceLatency`, so a simulation's model or voice result can be higher than the call's average for the same call.
40
+
-**The `unmeasured` list tells a missing value from a real `0`.** A stage that didn't run or wasn't measured on a turn is recorded as `0`, and the entry's `unmeasured` array names it. A `0` for a component that isn't listed is a real 0 ms. The array is left out when every component was measured. Entries recorded before `unmeasured` existed don't have it, so a `0` there can mean either. A component is listed when:
41
+
-`transcriber`: voice activity detection saw no speech for the turn, the sample was too stale to use, its last speech marker was already used by an earlier turn because detection missed this speech, the transcript's text was first seen before the caller started this speech, or detection still hadn't confirmed the end of speech when the reply started. That last case includes a caller who kept talking after a short pause, and its value is measured from when the speech started or from the pause they talked through, so it can be above `0`.
42
+
-`endpointing`: no caller transcript was sent to the model that turn, for example the first message or a reply to a tool result, or the transcript's text was first seen before the caller started this speech.
43
+
-`model`: the model sent no output token that turn, for example when the first audio is a static tool message or a guardrail held the reply.
44
+
-`voice`: there was no model output to time the voice from.
45
+
46
+
When `transcriberLatency` is unmeasured, `turnLatency` isn't a real measurement either: it leaves that interval out, or counts it from when the speech started.
47
+
-**Averages count every turn.** The call's `modelLatencyAverage`, `voiceLatencyAverage`, and other averages divide by every entry, including those zeros. Simulation latency limits leave out turns that list the metric in `unmeasured`, so a simulation's result can be higher than the call's average for the same call. The `artifact.performanceMetrics.turnLatencyUnmeasuredTracked` field is `true` on calls recorded with `unmeasured`, where an entry without the array had every component measured. On older calls, where it's absent, limits drop a `0` for `modelLatency` or `voiceLatency` and take no `transcriberLatency` or `endpointingLatency` values at all.
@@ -330,8 +331,10 @@ Each metric reads one field of the call's per-turn latency, measured the same wa
330
331
|`turn`|`turnLatency`| Time from the end of the AI tester's speech to the assistant starting its reply |
331
332
|`model`|`modelLatency`| Time from the start of the turn's model stage, including knowledge base retrieval, to the model's first output token |
332
333
|`voice`|`voiceLatency`| Time from the model's first output token to the first reply audio being ready, including the voice provider and any sample-rate conversion |
334
+
|`transcriber`|`transcriberLatency`| Time from the AI tester's last voice activity, normally the end of its speech, to the transcriber returning the words |
335
+
|`endpointing`|`endpointingLatency`| Time from the transcriber returning the words to Vapi deciding the AI tester's turn is over |
333
336
334
-
Model and voice latency happen inside turn latency, so the `model` and `voice`metrics don't add up to `turn`. A turn that recorded `0` for `model` or `voice`didn't measure that stage, so the metric leaves it out.
337
+
Turn latency is transcriber latency plus endpointing latency plus the time until the reply starts, and model and voice latency happen inside that last part, so the metrics don't add up to `turn`. A turn counts toward a metric unless its per-turn `unmeasured` list names that metric: a real 0 ms counts, and a turn that didn't measure the component is left out. The `turn` metric leaves out turns that list `transcriber`, because turn latency includes that interval. On calls recorded before the list existed (no `turnLatencyUnmeasuredTracked` in the call's performance metrics), a `0` for `model` or `voice`is left out, `transcriber` and `endpointing` take no turns, and `turn` counts every turn. See [`unmeasured`](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for when each component is listed.
335
338
336
339
The AI tester speaks with clean synthetic audio, so transcription, endpointing, and turn latency can run faster than with real callers.
337
340
@@ -403,7 +406,7 @@ For a transient scenario, set the same `targetOverrides` on the inline `scenario
403
406
| The assistant never asks whether the AI tester is still there | Simulations remove `customer.speech.timeout` hooks. To test them, set `targetOverrides.hooks` on the scenario. See [**Hooks and call endings**](#hooks-and-call-endings). |
404
407
| A voice simulation's assistant call ends as `customer-ended-call`| The AI tester hung up first, which matches a real caller hanging up. |
405
408
| A tool looks up the wrong caller | Set `customer.number` in the scenario's variable values. See [**Simulate a call from a specific phone number**](#simulate-a-call-from-a-specific-phone-number). |
406
-
| A latency limit fails without a measured value | No turn in the call measured that metric. Confirm the run used voice mode and that the conversation completed at least one exchange. |
409
+
| A latency limit fails without a measured value | No turn in the call measured that metric. Confirm the run used voice mode and that the conversation completed at least one exchange. For `transcriber` and `endpointing`, check each turn's `unmeasured` list in the target call's `artifact.performanceMetrics.turnLatencies`. These two limits always fail this way on calls recorded before `unmeasured` existed, which have no `turnLatencyUnmeasuredTracked`. |
407
410
| Latency limits show as skipped | Check whether the run used chat mode or the assistant under test uses GPT-Live. Both skip latency limits. |
0 commit comments