Repository navigation
Conversation
|
Before/after recordings from the live runs in the description. Each video is one caller line from a real call: the top waveform is the caller and the bottom one is the agent. Times are seconds from the start of each video. "Before" is #7287's head, and "after" is this branch with #7287 merged. 1. Default stack ( Before: VAD ends the second segment at 3.12 s. The text detector scores the stale "My name is John." as complete, and the turn commits at 3.24 s without "John Smith.", which never reaches the conversation. The agent replies "Thank you, John; please share your booking reference…". 1_text_eot_stale_commit_before.mp4After: VAD ends the second segment at 3.39 s and the STT flushes right away. The turn waits for that segment's final and commits at 3.57 s as "My name is John. John Smith.". The agent replies "Thank you, John Smith; …". 1_text_eot_stale_commit_after.mp42. Default stack: "Yes. [300 ms] But only if it's before noon." Before: the caller stops at 3.15 s. The detector scores the stale "Yes." as incomplete, so the flush waits until 4.75 s and the turn commits at 5.13 s, 1980 ms after the caller's last audio. 2_text_eot_wait_before.mp4After: the caller stops at 2.80 s. The STT flushes at end of speech (3.43 s), the detector scores the full sentence, and the turn commits at 3.57 s, 767 ms after the caller's last audio. 2_text_eot_wait_after.mp43. Before: VAD ends speech at 2.25 s, but the STT flush waits for the audio prediction (2.42 s). The turn commits at 2.58 s, 680 ms after the caller's last audio at 1.90 s. 3_audio_eot_before.mp4After: VAD ends speech at 2.07 s and the STT flushes at once, in parallel with the prediction. The turn commits at 2.24 s, 530 ms after the caller's last audio at 1.71 s. 3_audio_eot_after.mp4 |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| waiters = [ | ||
| asyncio.ensure_future(self._final_transcript_received.wait()), | ||
| asyncio.ensure_future(self._closing.wait()), | ||
| ] | ||
| try: | ||
| await asyncio.wait( | ||
| waiters, timeout=max(timeout, 0.0), return_when=asyncio.FIRST_COMPLETED |
There was a problem hiding this comment.
🔴 Earlier final releases latest segment wait
If an earlier flush returns an empty final after the latest flush, _final_transcript_received releases the latest segment's wait. EOT can commit the previous transcript before the latest segment's final arrives.
Learn more
A flush sentinel is queued into the STT pipeline, but the final-transcript event carries no identifier linking it to that flush. The new wait listens to a shared event set by every final, including an empty final from an earlier flush. If the earlier response arrives after the next VAD end, it releases the next segment's wait without updating its transcript. The EOT task can then score and commit the earlier words while the next segment is still being transcribed. Final processing sets the event before rejecting empty text.
Example: The first segment finalizes as "hello" before VAD ends; its flush produces a delayed empty final. A second segment begins and ends, sending another flush. The first flush's empty final now arrives, releasing the second wait. The pending "world" final has not arrived when EOT commits "hello".
Recommended fix: Associate flush completion with the segment or flush request before releasing the wait. If the provider does not expose IDs, account for ordered pending flushes and treat an earlier empty response as completion of the earlier flush, not the current one.
Was this helpful? React with 👍 or 👎 to provide feedback.
| async def _on_vad_event(self, ev: vad.VADEvent) -> None: | ||
| if ev.type == vad.VADEventType.START_OF_SPEECH: | ||
| self._stt_flush_requested = False | ||
| self._segment_transcribed = False |
There was a problem hiding this comment.
🟡 Early final incurs full endpointing delay
When STT finalizes before VAD starts, _segment_transcribed resets despite the completed transcript. If the flush produces no further final, EOT waits max_delay before committing that turn.
Learn more
STT and VAD events arrive through separate asynchronous consumers, so their event order does not establish which audio a final covers. A provider can send a final before local VAD emits START_OF_SPEECH for the same audio. Final processing marks that segment transcribed, but VAD start resets the mark. At VAD end the subsequent flush may produce no final for a provider that already finalized the words, leaving the new wait active for the entire maximum delay.
Example: An STT emits "hello" at 0.3 seconds, while local VAD emits START_OF_SPEECH at 0.4 seconds and END_OF_SPEECH at 1 second. The STT has no words left to finalize on the flush. The response waits until max_delay rather than the usual minimum delay.
Recommended fix: Track segment coverage using audio timestamps or a reliable segment boundary instead of clearing the final marker solely on VAD event arrival; ensure an already-finalized segment does not wait for a second final.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
The reason why it flushes at EOT boundary and not VAD boundary is that 11labs often misses short speeches with pauses with VAD EOS flushes, e.g. when user is speaking a long phone/card number. ElevenLabs explicitly warns: “Committing manually several times in a short sequence can degrade model performance.” in their docs. |
…the turn With STT manual flush, VAD end of speech could run end-of-turn detection on a transcript that did not include the segment that just ended. The text turn detector then scored the earlier segments only, and a negative score held the flush until half of max_delay. The audio turn detector delayed the flush until its prediction resolved, and the turn could commit before the latest segment's final transcript arrived. VAD end of speech now flushes immediately for every detector type. If the latest segment has no final transcript yet, detection waits until the flushed final re-enters it, bounded by max_delay from the end of speech. An empty final, or a segment the STT already finalized, does not wait.
3db091b to
6127e85
Compare
Follows #7285. The live numbers below include #7287, so ElevenLabs Scribe v2 realtime declares
manual_flushand finalizes a segment only when it is flushed.What was wrong
With STT manual flush, VAD end of speech starts end-of-turn (EOT) detection. When the caller pauses mid-turn and resumes, the transcript at the next VAD end of speech holds only the earlier segments, because the newest segment's final transcript arrives only after the flush.
MultilingualModel): the model scored that stale transcript (selection). A negative score held the flush until half ofmax_delay(delayed flush), so the turn took 1.7 to 2 s at default endpointing. A positive score committed the turn without the newest segment, and that segment's final then started a second user turn.inference.TurnDetector): the flush waited for the audio prediction, so the STT started finalizing only after the prediction resolved. The commit could also race ahead of the newest segment's final.What changed
max_delayfrom the end of speech bounds it for segments that never produce words.STTCapabilities.manual_flushdocstring describes the new behavior.Measurements
Live calls through
lk agent debugger start --audiowith scripted caller lines (lk agent debugger say): ElevenLabsscribe_v2_realtimeSTT with manual commits,openai/gpt-4.1-mini,cartesia/sonic-3. "Before" is #7287's head. "After" is this branch with #7287 merged. Each build ran three takes per line set; a few lines whose caller TTS failed in the harness are excluded. EOT is the caller's last audio in the recording to the committed user turn (about ±100 ms of alignment noise).A. Default stack:
MultilingualModel, Sileromin_silence_duration=0.55Multi-segment lines pause mid-sentence long enough for VAD to end the segment, for example "Yes. [300 ms] But only if it's before noon." or "I want the blue one. Actually... no, wait, the red one." The control lines have short pauses that VAD does not split.
Before, three of the four stale commits split the line into two user turns. In the fourth, the newest segment ("John Smith.") never reached the conversation. The one slow turn after the change is a single-segment line whose complete transcript scored 0.0108 against the 0.011 threshold, so it waited
max_delayas intended. Lines where the detector ended the turn after a complete first phrase, with the full transcript so far, went from 8 to 6.B.
inference.TurnDetector(version="v1"), Sileromin_silence_duration=0.25Five short questions plus the multi-segment lines.
Turns with a negative audio prediction wait
max_delay(2.5 s) in both builds, so they are left out of the EOT columns. The audio detector does not read the transcript, and how many lines it scored negative varied from take to take.With this config, every first phrase that the detector scored as complete ("Perfect, thanks.", "Yes.", "I want the blue one.") became its own user turn after the change: 10 of 10, against 5 of 7 before. The slower commit before the change let a caller who resumed within about 120 ms cancel 2 of them. Committing about 150 ms sooner with a 0.25 s VAD silence moves that boundary. The committed transcript was complete in every case.
Known limits
max_delayif the STT sends no final for it. ElevenLabs sends no final for an empty commit.