From 4caa21dc57d263e34197174b5b1c017f59fd94df Mon Sep 17 00:00:00 2001 From: Scott Lowe Date: Fri, 30 Oct 2026 21:00:36 -0700 Subject: [PATCH] TEST-273 docs: document transcriber and endpointing simulation latency limits and the unmeasured flag Add the transcriber and endpointing metrics to simulation latency limits, and explain the per-turn unmeasured list that decides which turns count. Merge after VapiAI/vapi#25252, which adds the metrics. Co-Authored-By: Claude Opus 5.5 --- .../model-intelligence/understanding-latency.mdx | 10 ++++++++-- fern/observability/simulations-advanced.mdx | 11 +++++++---- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/fern/assistants/model-intelligence/understanding-latency.mdx b/fern/assistants/model-intelligence/understanding-latency.mdx index d1e8d423f..4e8f00c38 100644 --- a/fern/assistants/model-intelligence/understanding-latency.mdx +++ b/fern/assistants/model-intelligence/understanding-latency.mdx @@ -37,8 +37,14 @@ Keep these points in mind when you read the numbers: - **Model and voice latency include more than the provider.** The `modelLatency` field includes knowledge base retrieval and request routing before the provider request. The `voiceLatency` field includes the time spent collecting enough of the reply to send the first voice request. The `minCharacters` and `punctuationBoundaries` settings in `voice.chunkPlan` control that wait. The field also includes any sample-rate conversion after the provider returns audio. - **Vapi measures on its servers.** None of the fields include network time to or from the caller, or playback on the caller's device. - **Only replies to the caller are recorded.** Turns where the assistant didn't speak, for example because the caller interrupted first, have no entry. When the assistant speaks first, its first message has no entry either. When it waits for the caller (`firstMessageMode` set to `assistant-waits-for-user`), the first message is recorded as a reply, with `endpointingLatency` as `0`. -- **A value of `0` can mean "not measured".** A stage that didn't run or wasn't measured on a turn is recorded as `0` rather than left out. For example, when a turn's first audio is a static tool message rather than model output, `modelLatency` and `voiceLatency` are `0`. When `transcriberLatency` is `0` because it wasn't measured, `turnLatency` leaves that interval out too. -- **Averages count every turn.** The call's `modelLatencyAverage`, `voiceLatencyAverage`, and other averages divide by every entry, including those zeros. Simulation latency limits ignore a `0` for `modelLatency` and `voiceLatency`, so a simulation's model or voice result can be higher than the call's average for the same call. +- **The `unmeasured` list tells a missing value from a real `0`.** A stage that didn't run or wasn't measured on a turn is recorded as `0`, and the entry's `unmeasured` array names it. A `0` for a component that isn't listed is a real 0 ms. The array is left out when every component was measured. Entries recorded before `unmeasured` existed don't have it, so a `0` there can mean either. A component is listed when: + - `transcriber`: voice activity detection saw no speech for the turn, the sample was too stale to use, its last speech marker was already used by an earlier turn because detection missed this speech, the transcript's text was first seen before the caller started this speech, or detection still hadn't confirmed the end of speech when the reply started. That last case includes a caller who kept talking after a short pause, and its value is measured from when the speech started or from the pause they talked through, so it can be above `0`. + - `endpointing`: no caller transcript was sent to the model that turn, for example the first message or a reply to a tool result, or the transcript's text was first seen before the caller started this speech. + - `model`: the model sent no output token that turn, for example when the first audio is a static tool message or a guardrail held the reply. + - `voice`: there was no model output to time the voice from. + + When `transcriberLatency` is unmeasured, `turnLatency` isn't a real measurement either: it leaves that interval out, or counts it from when the speech started. +- **Averages count every turn.** The call's `modelLatencyAverage`, `voiceLatencyAverage`, and other averages divide by every entry, including those zeros. Simulation latency limits leave out turns that list the metric in `unmeasured`, so a simulation's result can be higher than the call's average for the same call. The `artifact.performanceMetrics.turnLatencyUnmeasuredTracked` field is `true` on calls recorded with `unmeasured`, where an entry without the array had every component measured. On older calls, where it's absent, limits drop a `0` for `modelLatency` or `voiceLatency` and take no `transcriberLatency` or `endpointingLatency` values at all. ## What the displayed latency measures diff --git a/fern/observability/simulations-advanced.mdx b/fern/observability/simulations-advanced.mdx index f55b7e242..0053e4e3a 100644 --- a/fern/observability/simulations-advanced.mdx +++ b/fern/observability/simulations-advanced.mdx @@ -293,7 +293,7 @@ Latency limits work alongside structured-output evaluations. A scenario still ne On the **Success criteria** tab, under **Latency expectations**, select **Add latency expectation**. Configure each limit with four fields: - **Aggregation**: How the call's turns are combined: **Median**, **Mean**, **P95**, or **Max**. -- **Metric**: **Turn latency**, **Model latency**, or **Voice latency**. +- **Metric**: **Turn latency**, **Model latency**, **Voice latency**, **Transcriber latency**, or **Endpointing latency**. - **Limit**: The maximum in milliseconds, as a whole number from 1 to 60,000. - **Required**: Turn off to make the limit informational. An optional limit is reported but never fails the simulation. @@ -311,7 +311,8 @@ curl -X PATCH "https://api.vapi.ai/eval/simulation/scenario/" \ -d '{ "latencyExpectations": [ { "metric": "turn", "aggregation": "median", "thresholdMs": 1200 }, - { "metric": "model", "aggregation": "p95", "thresholdMs": 800, "required": false } + { "metric": "model", "aggregation": "p95", "thresholdMs": 800, "required": false }, + { "metric": "endpointing", "aggregation": "median", "thresholdMs": 600 } ] }' ``` @@ -330,8 +331,10 @@ Each metric reads one field of the call's per-turn latency, measured the same wa | `turn` | `turnLatency` | Time from the end of the AI tester's speech to the assistant starting its reply | | `model` | `modelLatency` | Time from the start of the turn's model stage, including knowledge base retrieval, to the model's first output token | | `voice` | `voiceLatency` | Time from the model's first output token to the first reply audio being ready, including the voice provider and any sample-rate conversion | +| `transcriber` | `transcriberLatency` | Time from the AI tester's last voice activity, normally the end of its speech, to the transcriber returning the words | +| `endpointing` | `endpointingLatency` | Time from the transcriber returning the words to Vapi deciding the AI tester's turn is over | -Model and voice latency happen inside turn latency, so the `model` and `voice` metrics don't add up to `turn`. A turn that recorded `0` for `model` or `voice` didn't measure that stage, so the metric leaves it out. +Turn latency is transcriber latency plus endpointing latency plus the time until the reply starts, and model and voice latency happen inside that last part, so the metrics don't add up to `turn`. A turn counts toward a metric unless its per-turn `unmeasured` list names that metric: a real 0 ms counts, and a turn that didn't measure the component is left out. The `turn` metric leaves out turns that list `transcriber`, because turn latency includes that interval. On calls recorded before the list existed (no `turnLatencyUnmeasuredTracked` in the call's performance metrics), a `0` for `model` or `voice` is left out, `transcriber` and `endpointing` take no turns, and `turn` counts every turn. See [`unmeasured`](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for when each component is listed. The AI tester speaks with clean synthetic audio, so transcription, endpointing, and turn latency can run faster than with real callers. @@ -403,7 +406,7 @@ For a transient scenario, set the same `targetOverrides` on the inline `scenario | The assistant never asks whether the AI tester is still there | Simulations remove `customer.speech.timeout` hooks. To test them, set `targetOverrides.hooks` on the scenario. See [**Hooks and call endings**](#hooks-and-call-endings). | | A voice simulation's assistant call ends as `customer-ended-call` | The AI tester hung up first, which matches a real caller hanging up. | | A tool looks up the wrong caller | Set `customer.number` in the scenario's variable values. See [**Simulate a call from a specific phone number**](#simulate-a-call-from-a-specific-phone-number). | -| A latency limit fails without a measured value | No turn in the call measured that metric. Confirm the run used voice mode and that the conversation completed at least one exchange. | +| A latency limit fails without a measured value | No turn in the call measured that metric. Confirm the run used voice mode and that the conversation completed at least one exchange. For `transcriber` and `endpointing`, check each turn's `unmeasured` list in the target call's `artifact.performanceMetrics.turnLatencies`. These two limits always fail this way on calls recorded before `unmeasured` existed, which have no `turnLatencyUnmeasuredTracked`. | | Latency limits show as skipped | Check whether the run used chat mode or the assistant under test uses GPT-Live. Both skip latency limits. | ## Next steps