openai TTS hardcodes response_format: 'pcm' and never checks the response, so a compatible endpoint returning anything else is played as raw samples

[abidullahdev]

Describe the bug

plugins/openai/src/tts.ts asks for response_format: 'pcm' and then treats whatever comes back as raw samples, without decoding it or looking at the response Content-Type:

response_format: 'pcm',
...
const buffer = await this.stream.then((r) => r.arrayBuffer());
const audioByteStream = new AudioByteStream(OPENAI_TTS_SAMPLE_RATE, OPENAI_TTS_CHANNELS);
const frames = audioByteStream.write(buffer);

response_format is also hardcoded, so it can't be changed through TTSOptions or updateOptions().

Against api.openai.com this is fine you ask for pcm, you get pcm. But baseURL is a public option in TTSOptions, and get provider() reads the host off this.#client.baseURL, so pointing the plugin at an OpenAI-compatible server looks like an intended use. If that server doesn't support pcm, or ignores response_format and answers with its own default, the bytes are written into the audio buffer as 16-bit samples.

Serving a 9644-byte WAV (44-byte RIFF header + 9600 bytes of 24 kHz mono PCM) emits 9600 bytes whose first four bytes are the ASCII characters RIFF. The container header is played as audio; only a trailing partial frame is dropped.

For a WAV that happens to match the expected sample rate, the result is a short click before otherwise-correct audio, which is easy to miss. For a compressed format mp3, opus, aac, flac none of the body is PCM, so the whole utterance is noise. Either way there is no error, no warning and no retry.

livekit/agents#6930 fixed the equivalent problem on the Python side by choosing the parser from the response Content-Type.

Relevant log output

(none the failure is silent; no error, warning or retry is emitted)

Describe your environment

  System:
    OS: Windows 11 10.0.26200
    CPU: (14) x64 Intel(R) Core(TM) Ultra 5 235U
  Binaries:
    Node: 24.19.0
    pnpm: 11.13.1
  npmPackages:
    @livekit/agents-plugin-openai: 1.7.0 (verified against main @ 31d66e4)

Minimal reproducible example

Run as a vitest test inside the repo:

import { OpenAI } from 'openai';
import { describe, expect, it } from 'vitest';
import { TTS } from './tts.js';

const pcm = Buffer.alloc(9600);
const wav = (() => {
  const h = Buffer.alloc(44);
  h.write('RIFF', 0); h.writeUInt32LE(36 + pcm.length, 4); h.write('WAVE', 8);
  h.write('fmt ', 12); h.writeUInt32LE(16, 16); h.writeUInt16LE(1, 20); h.writeUInt16LE(1, 22);
  h.writeUInt32LE(24000, 24); h.writeUInt32LE(48000, 28); h.writeUInt16LE(2, 32); h.writeUInt16LE(16, 34);
  h.write('data', 36); h.writeUInt32LE(pcm.length, 40);
  return Buffer.concat([h, pcm]);
})();

it('does not play a wav container as if it were pcm', async () => {
  const client = new OpenAI({
    apiKey: 'test',
    baseURL: 'https://compatible.example.com/v1',
    maxRetries: 0,
    fetch: async () =>
      new Response(new Uint8Array(wav), { status: 200, headers: { 'content-type': 'audio/wav' } }),
  });
  const tts = new TTS({ client, model: 'kokoro' });

  const chunks: Buffer[] = [];
  for await (const ev of tts.synthesize('hello')) {
    chunks.push(Buffer.from(ev.frame.data.buffer, ev.frame.data.byteOffset, ev.frame.data.byteLength));
  }
  const out = Buffer.concat(chunks);

  expect(out.subarray(0, 4).toString('latin1')).not.toBe('RIFF'); // currently fails
});

Additional information

A fix could mirror what landed in Python: read Content-Type off the response and decode accordingly, falling back to the requested format when the server doesn't declare something usable. Exposing response_format through TTSOptions would also help, since some compatible backends don't offer pcm at all.

Worth noting that plugins/openai/src/tts.test.ts currently requires a live OPENAI_API_KEY and skips without one, so nothing exercises this path in CI. A hermetic test like the one above would cover it without network access.

I'm happy to open a PR if you'd like I wrote the equivalent change for the Python plugin, so the shape would be familiar.