PocketTTS: the 512-position KV cache forces sentences over 50 tokens to be cut; could the conversion use a longer cache?

[arvindvenkataramani]

Summary

The PocketTTS Core ML conversion has a fixed 512-position KV cache, and because of it FluidAudio cuts every sentence longer than 50 tokens into chunks that are synthesized separately. Prosody restarts at each cut, and where a cut falls between words it is heard as a pause in the middle of a phrase. With ‖ marking the cuts, the 69-token sentence in the steps below is spoken as:

The humidity feels comfortable despite the approaching storm, ‖ and the air carries that distinctive electric charge that precedes rain — a mix of ozone and wet soil that makes everything feel fresh and alive before the first drops hit ‖ the pavement.

This is the normal case, not an edge case: in two passages of plain English prose (below), all seven sentences measure 52 to 70 tokens, so every one of them is cut.

mlx-audio runs the same model with the same voice and speaks these sentences whole; side by side, the difference is obvious. Kyutai's original Python implementation of Pocket-TTS, which FluidAudio and mlx-audio are both ports of, sits in between: it cuts an oversized sentence only at a comma, semicolon or colon, never between words, and its cache grows with the utterance instead of being fixed.

Steps to reproduce

fluidaudiocli tts --backend pocket --voice alba --seed 42 -o pavement.wav --text "The humidity feels comfortable despite the approaching storm, and the air carries that distinctive electric charge that precedes rain — a mix of ozone and wet soil that makes everything feel fresh and alive before the first drops hit the pavement."

Listen for the pause before "the pavement".

Environment

FluidAudio main @ b68f4847, FluidInference/pocket-tts-coreml v2.1 English. macOS 26.6.2, M4 Max. Comparison: mlx-audio serving mlx-community/pocket-tts.

Request

Is 512 a deliberate limit, for latency, memory or ANE placement? If it is not needed, a longer cache (1024, offered as an opt-in export in the way #925 did for Chatterbox Nano) would let sentences be synthesized whole, as mlx-audio does.

Related

#931 (fix in #932) corrects the token counts this behaviour depends on. #934 (change in #935) makes the cuts less audible while the cache stays at 512. #584 (closed) addressed how the pieces of a cut sentence are normalized.

---

Notes — by Claude Code

Written by Claude Code, an AI assistant working with the reporter. Measured on b68f4847.

• The cache holds the voice prompt, the text tokens, and one position per generated 80 ms audio frame. With a 125-frame voice prompt (the shipped alba, or a 10-second cloned voice), whole sentences of 58 to 69 tokens use 286 to 356 of the 512 positions. Ordinary sentences therefore already fit; it is the 50-token rule that cuts them.
• Only flowlm_step and cond_prefill contain the cache tensor ([2, 1, 512, 16, 64] in their model.mil). flow_decoder_fused, mimi_decoder and the shipped voice files (constants_bin/*.safetensors) do not depend on it.
• In a local experiment that kept a sentence whole whenever it fit the cache, FluidAudio split the two passages below into exactly the chunks mlx-audio produces, one per sentence, and a listener judged the two runtimes equal.
• Kyutai's implementation logs that a chunk over 50 tokens "may skip words". None were heard across these seven sentences, through mlx-audio or the experiment above; that is a small sample.

The two test passages, with token counts per sentence

Passage 1:

1. (64) The sky stretches out in a brilliant shade of blue today, dotted with scattered clouds that drift lazily across the horizon as the afternoon sun warms the air to a comfortable seventy-two degrees.
2. (58) A gentle breeze from the west carries the scent of rain and damp earth, hinting at the possibility of an evening shower that might cool things down before nightfall settles in over the valley.
3. (60) Clouds are gathering on the western ridge by late afternoon, their gray undersides promising scattered showers that could pass through quickly or linger longer depending on how the wind shifts overnight.
4. (69) The humidity feels comfortable despite the approaching storm, and the air carries that distinctive electric charge that precedes rain — a mix of ozone and wet soil that makes everything feel fresh and alive before the first drops hit the pavement.

Passage 2:

1. (69) The control plane on the Pi (which also serves the dashboard) polls the shard every fifteen minutes and merges whatever services it finds into a single response that the rest of the platform treats as the source of truth.
2. (70) If a health check fails once the counter simply goes up and nothing else happens… but a second failure in a row flips the service to down - and that change is written to the event log so the state can always be rebuilt later.
3. (52) Nobody should ever need to restart the whole machine just because one small demand loaded model ran out of memory while another bigger one was still busy answering a long question from the voice loop.

Counts are from Google's sentencepiece Python package with the English tokenizer.model.