[u9g]

What was wrong

AgentActivity.on_preemptive_generation cancels the in-flight preemptive reply and starts a new LLM request on every call. AudioRecognition calls it on every PREFLIGHT_TRANSCRIPT, and on a FINAL_TRANSCRIPT that is not an exact string match of the last preflight.

STTs revise punctuation within a turn. Deepgram Flux through LiveKit Inference (deepgram/flux-general-en, default settings) often sends a second EagerEndOfTurn that only adds a question mark:

2.170s  preflight  "Could you tell me the opening hours"
2.289s  preflight  "Could you tell me the opening hours?"
2.333s  final      "Could you tell me the opening hours?"

Across 42 synthesized Flux turns (default settings and eager_eot_threshold=0.3), 8 had a second preflight that differed only in punctuation; for that question it happened in 7 of 8 takes. Each one threw away the running LLM request and started over, so the head start shrank by the gap between the two preflights (about 120 to 280 ms here) and the LLM was called twice. The restart never changes what the agent says: at end of turn the reply is already reused when the transcripts are _transcripts_equivalent (casefolded, punctuation-insensitive).

What changed

on_preemptive_generation returns early, before cancelling or counting a retry, when the in-flight attempt's transcript is _transcripts_equivalent to the new one. That covers a formatting-only second preflight and a final that only reformats the preflight. A transcript with new words still cancels and restarts, and the end-of-turn check still compares transcript, chat context, tools and tool choice before reusing the reply.

Same fix in agents-js: livekit/agents-js#2794

Verification

• New test_preemptive_generation_restarts_only_on_new_words: a second attempt that only changes punctuation or case keeps the first one (fails on main); one with new words restarts.
• make type-check, ruff check and ruff format --check pass. pytest --unit --audio_eot passes apart from tests that fail the same way on main locally (silero VAD buffer resize, Google credentials) and two loop-monitor timing tests that pass on rerun.
• Demo in the comment below: the captured Flux turn replayed into AgentSession against openai/gpt-4.1-mini on LiveKit Inference. On main it sends 2 LLM requests and reuses the second with preemptive_lead_time=0.382s. On this branch it sends 1 and reuses it with preemptive_lead_time=0.502s.

[u9g]

Replay of a captured Deepgram Flux turn (second preflight only adds a question mark) into AgentSession against openai/gpt-4.1-mini on LiveKit Inference. On main the second preflight cancels the reply and a second LLM request is sent (preemptive_lead_time=0.382s); on this branch the first request is kept and reused (preemptive_lead_time=0.502s, 1 LLM request).

preemptive-formatting-only.mp4

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

🟡 Adjacent punctuation restarts preemptive replies

When a transcript changes hello world to hello,world, _formatting_free_words treats hello,world as one word. The equivalent reply is discarded and regenerated despite unchanged spoken words.

Learn more

The word splitter divides on whitespace and selected character-based scripts, not on punctuation. Keeping internal punctuation protects meaningful numbers such as 1.5 from matching 15, but it also combines ordinary words when a transcript inserts a comma without a following space. The resulting token lists differ and both preflight deduplication and end-of-turn reuse discard the running reply.

Example: A preflight of hello world yields ["hello", "world"]; a final transcript of hello,world yields ["hello,world"]. The agent sends another LLM request although the spoken words have not changed.

Recommended fix: Split punctuation between alphabetic words into word boundaries when comparing transcripts, while retaining punctuation inside numeric values and other meaning-bearing tokens. Cover adjoining-punctuation and decimal cases in the equivalence tests.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[u9g]

Not changing this. The only cost is one extra LLM request, which is the safe direction: a mismatch regenerates and never reuses a wrong reply. Before this PR, the preflight path regenerated on any change to the string at all. STTs put a space after punctuation between words, so hello,world against hello world does not happen in practice. Splitting on punctuation between letters while keeping it inside numbers adds rules to the comparison and fixes no case we have seen.