Skip to content

Commit 1295d73

Browse files
TEST-273 docs: document transcriber and endpointing simulation latency limits and the unmeasured flag
Add the transcriber and endpointing metrics to simulation latency limits, and explain the per-turn unmeasured list that decides which turns count. Merge after VapiAI/vapi#25252, which adds the metrics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent eacdb83 commit 1295d73

2 files changed

Lines changed: 15 additions & 6 deletions

File tree

‎fern/assistants/model-intelligence/understanding-latency.mdx‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,14 @@ Keep these points in mind when you read the numbers:
3737
- **Model and voice latency include more than the provider.** `modelLatency` includes knowledge base retrieval and request routing before the provider request. `voiceLatency` includes collecting enough of the reply to send the first voice request, as set by `voice.chunkPlan` (`minCharacters` and punctuation boundaries), and any sample-rate conversion after the provider returns audio.
3838
- **Vapi measures on its servers.** None of the fields include network time to or from the caller, or playback on the caller's device.
3939
- **Only replies to the caller are recorded.** Turns where the assistant didn't speak, for example because the caller interrupted first, have no entry. When the assistant speaks first, its first message has no entry either. When it waits for the caller (`firstMessageMode` set to `assistant-waits-for-user`), the first message is recorded as a reply, with `endpointingLatency` as `0`.
40-
- **`0` can mean "not measured".** A stage that didn't run or wasn't measured on a turn is recorded as `0` rather than left out. For example, when a turn's first audio is a static tool message rather than model output, `modelLatency` and `voiceLatency` are `0`. When `transcriberLatency` is `0` because it wasn't measured, `turnLatency` leaves that interval out too.
41-
- **Averages count every turn.** The call's `modelLatencyAverage`, `voiceLatencyAverage`, and other averages divide by every entry, including those zeros. Simulation latency limits ignore a `0` for `modelLatency` and `voiceLatency`, so a simulation's model or voice result can be higher than the call's average for the same call.
40+
- **`unmeasured` tells a missing value from a real `0`.** A stage that didn't run or wasn't measured on a turn is recorded as `0`, and the entry's `unmeasured` array names it. A `0` for a component that isn't listed is a real 0 ms. The array is left out when every component was measured. Entries recorded before `unmeasured` existed don't have it, so a `0` there can mean either. A component is listed when:
41+
- `transcriber`: voice activity detection saw no speech for the turn, the sample was too stale to use, or the transcript committed before voice activity detection saw the end of speech, so the value was measured from the start of speech.
42+
- `endpointing`: no caller transcript was sent to the model that turn, for example the first message or a reply to a tool result.
43+
- `model`: the model sent no output token that turn, for example when the first audio is a static tool message or a guardrail held the reply.
44+
- `voice`: there was no model output to time the voice from.
45+
46+
When `transcriberLatency` is unmeasured, `turnLatency` leaves that interval out too.
47+
- **Averages count every turn.** The call's `modelLatencyAverage`, `voiceLatencyAverage`, and other averages divide by every entry, including those zeros. Simulation latency limits leave out turns that list the metric in `unmeasured`, and a `0` for `modelLatency` or `voiceLatency` on older entries, so a simulation's result can be higher than the call's average for the same call.
4248

4349
## What the displayed latency measures
4450

‎fern/observability/simulations-advanced.mdx‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -293,7 +293,7 @@ Latency limits work alongside structured-output evaluations. A scenario still ne
293293
On the **Success criteria** tab, under **Latency expectations**, select **Add latency expectation**. Configure each limit with four fields:
294294

295295
- **Aggregation**: How the call's turns are combined: **Median**, **Mean**, **P95**, or **Max**.
296-
- **Metric**: **Turn latency**, **Model latency**, or **Voice latency**.
296+
- **Metric**: **Turn latency**, **Model latency**, **Voice latency**, **Transcriber latency**, or **Endpointing latency**.
297297
- **Limit**: The maximum in milliseconds, as a whole number from 1 to 60,000.
298298
- **Required**: Turn off to make the limit informational. An optional limit is reported but never fails the simulation.
299299

@@ -311,7 +311,8 @@ curl -X PATCH "https://api.vapi.ai/eval/simulation/scenario/<scenario-id>" \
311311
-d '{
312312
"latencyExpectations": [
313313
{ "metric": "turn", "aggregation": "median", "thresholdMs": 1200 },
314-
{ "metric": "model", "aggregation": "p95", "thresholdMs": 800, "required": false }
314+
{ "metric": "model", "aggregation": "p95", "thresholdMs": 800, "required": false },
315+
{ "metric": "endpointing", "aggregation": "median", "thresholdMs": 600 }
315316
]
316317
}'
317318
```
@@ -330,8 +331,10 @@ Each metric reads one field of the per-turn latency that every call records, mea
330331
| `turn` | `turnLatency` | Time from the end of the AI tester's speech to the assistant starting its reply |
331332
| `model` | `modelLatency` | Time from the start of the turn's model stage, including knowledge base retrieval, to the model's first output token |
332333
| `voice` | `voiceLatency` | Time from the model's first output token to the first reply audio being ready, including the voice provider and any sample-rate conversion |
334+
| `transcriber` | `transcriberLatency` | Time from the AI tester's last voice activity, normally the end of its speech, to the transcriber returning the words |
335+
| `endpointing` | `endpointingLatency` | Time from the transcriber returning the words to Vapi deciding the AI tester's turn is over |
333336

334-
`model` and `voice` happen inside `turn`, so they don't add up to it. Turns that recorded `0` for `model` or `voice` didn't measure it and are left out of those metrics. The AI tester speaks with clean synthetic audio, so transcription, endpointing, and turn latency can run faster than with real callers.
337+
`turn` is `transcriber` plus `endpointing` plus the time until the reply starts, and `model` and `voice` happen inside that last part, so the metrics don't add up to `turn`. A turn counts toward a metric unless its per-turn `unmeasured` list names that metric: a real 0 ms counts, and a turn that didn't measure the component is left out. `turn` counts every turn. See [`unmeasured`](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for when each component is listed. The AI tester speaks with clean synthetic audio, so transcription, endpointing, and turn latency can run faster than with real callers.
335338

336339
The aggregation reduces the call's turns to one value: `mean`, `median`, `p95`, or `max`. `p95` uses the nearest-rank method, so on a call with fewer than 20 turns it equals `max`. Measured values are rounded to the nearest millisecond before they are compared with the limit.
337340

@@ -401,7 +404,7 @@ For a transient scenario, set the same `targetOverrides` on the inline `scenario
401404
| The assistant never asks whether the AI tester is still there | Simulations remove `customer.speech.timeout` hooks. To test them, set `targetOverrides.hooks` on the scenario. See [**Hooks and call endings**](#hooks-and-call-endings). |
402405
| A voice simulation's assistant call ends as `customer-ended-call` | The AI tester hung up first, which matches a real caller hanging up. |
403406
| A tool looks up the wrong caller | Set `customer.number` in the scenario's variable values. See [**Simulate a call from a specific phone number**](#simulate-a-call-from-a-specific-phone-number). |
404-
| A latency limit fails without a measured value | No turn in the call measured that metric. Confirm the run used voice mode and that the conversation completed at least one exchange. |
407+
| A latency limit fails without a measured value | No turn in the call measured that metric. Confirm the run used voice mode and that the conversation completed at least one exchange. For `transcriber` and `endpointing`, check each turn's `unmeasured` list in the target call's `artifact.performanceMetrics.turnLatencies`. |
405408
| Latency limits show as skipped | Check whether the run used chat mode or the assistant under test uses GPT-Live. Both skip latency limits. |
406409

407410
## Next steps

0 commit comments

Comments
 (0)