Chatterbox Nano: real usable budget is ~135 input tokens / ~10s output, not 512 — consider larger/multiple buckets

[Abusayid693]

What problem does this solve?

ChatterboxNanoManager's documented caps (prefillLength = 512, flowTokenBucket = 500) read as generous, but the real usable budget for a caller is much smaller than either number suggests, and it's easy to hit in normal use.

Input side — ChatterboxNanoSynthesizer.swift#L45-L50:

let condLen = models.voice.condEmb.rows        // 376 for the bundled default voice
let contextLen = condLen + textIds.count + 1   // + text + 1 BOS
guard contextLen <= ChatterboxNanoConstants.prefillLength else {  // 512
    throw ChatterboxError.textTooLong(...)
}

condEmb for tables/voice-default.safetensors is [1, 376, 768] (checked directly against the file's safetensors header) — so 376 of the 512 prefill slots (73%) are consumed by the voice's own conditioning before any input text is counted. Real text budget: ≤135 BPE tokens (~500-550 chars at ~4 chars/token), not 512.

Output side — ChatterboxNanoSynthesizer.swift#L134-L139:

let promptLen = models.voice.promptTokens.count   // 250 for the default voice
let totalTokens = promptLen + speechTokens.count  // + 3 appended silence tokens
guard totalTokens <= ChatterboxNanoConstants.flowTokenBucket else {  // 500
    throw ChatterboxError.generationTooLong(...)
}

prompt_token for the default voice is [1, 250] — so 250 of the 500 flow-bucket slots (50%) are the voice's own prompt, leaving ≤247 generated speech tokens. At the documented 25 Hz S3 rate, that's a hard ceiling of ~9.9 seconds of generated audio — regardless of the input cap above.

In practice the output cap binds first: input well under the 135-token cap (a single short, expressive sentence with a paralinguistic tag) can already produce audio close to or over ~10s, since generation length depends on pacing/punctuation/tags, not a fixed ratio to input length. Confirmed on-device (iOS): a ~148-character clip generated successfully; the same text plus two more words (~156 chars) failed with generationTooLong.

The two caps themselves are defined at ChatterboxNanoConstants.swift#L22 (prefillLength) and #L45 (flowTokenBucket), and are baked into the compiled model shapes themselves (T3Nano-Prefill-T512-M1536-fp16.mlmodelc, FlowMean-N500-fp16.mlmodelc — the T512/N500 in the filenames), not app-tunable constants — see the required-file list at ModelNames.swift#L1576. Only one shape of each is currently published to FluidInference/chatterbox-nano-coreml.

Proposed solution

Publish a larger-bucket Chatterbox Nano variant (or multiple bucket sizes), analogous to how Supertonic-3 already publishes VectorEstimatorVariants/{L128,L256,L512} for a similar shape-selection problem — e.g. a T3Nano-Prefill-T768-... / FlowMean-N1000-... pairing, or whatever headroom the underlying T3/S3Gen architecture supports, so callers whose voice conditioning already consumes half the budget aren't left with only ~10 seconds of usable output.

Short of a larger bucket, documenting the real usable budget (accounting for condEmb.rows/promptTokens.count) rather than the raw prefillLength/flowTokenBucket numbers in ChatterboxNanoConstants's doc comments would at least set correct caller expectations.

Alternatives considered

• Chunking long input client-side — doesn't help with the output cap, which is the one that actually binds; chunking would also mean multiple independently-seeded generations with no shared prosody, audible discontinuities at each boundary.
• Reducing the voice conditioning footprint (condEmb/promptTokens) — not something a caller can do; these come from the published voice file.

Additional context

• Model files inspected directly: tables/voice-default.safetensors header (t3_cond_emb: [1, 376, 768], prompt_token: [1, 250]) from FluidInference/chatterbox-nano-coreml.
• Found while integrating Chatterbox Nano as a fourth TTS engine in a downstream app; happy to attach real on-device (iOS) runtime numbers (prefill/decode/flow/vocoder timings, the SDK's own ChatterboxNanoManager logs them per-call) as a follow-up comment once collected.