Skip to content

Commit 9c0665f

Browse files
authored
feat(livekit): Trace LiveKit voice sessions with audio schema v3 (#2726)
Based on #2725 ## Why LiveKit voice traces now follow audio schema v3. Some of the major changes from this: - Speech and user turns carry `audio.selections` in reference to the session's timeline, so a span points at its exact place in the recording. This is recorded even when audio capture is off. - Attribution for sources, participants (e.g. a caller), and speakers (individual voices) including support for diarization (identifying which text belonged to what speaker.) - Links between when operations, when possible. e.g. A model response links to the transcripts in its context, a tool to the response that requested it, and played speech to the synthesis and model response that generated it. ## Example A two-caller order call with a diarizing STT (from the e2e fixture): ``` session audio.schema.version: 3.0.0 · sources · speakers · segments · timeline ├── user_turn input: "Hi. I'm calling with my partner, … Where is order demo one zero four two? …" │ ├── user_speech ×4 audio.selections │ ├── stt "Hi. I'm calling with my partner," speaker-1 │ ├── stt "who has a question about our order." speaker-1 │ ├── stt "Where is order demo one zero four two?" speaker-2 │ └── … └── agent_turn input: (the transcript above) · turn.reply_to → user_turn ├── llm_response uses_context → each stt ├── lookup_order requested_by → llm_response ├── llm_response uses_context → each stt · consumes_tool_result → lookup_order ├── tts generated_by → llm_response └── agent_speech "Your order … is scheduled for delivery on Friday." audio.selections · generated_by → tts ``` Session metadata, illustrated for a call with a room participant: ```json { "audio.schema.version": "3.0.0", "audio.sources": { "session-input": { "label": "Agent input audio" }, "agent-output": { "label": "Agent audio" } }, "audio.speakers": { "speaker-1": { "type": "user", "attributes": { "contrib.livekit.speaker_id": "0" } } }, "audio.participants": { "participant-1": { "type": "user", "attributes": { "contrib.livekit.participant_id": "PA_x7Kq", "contrib.livekit.participant_kind": "SIP" } } }, "audio.segments": { "part-0": { "state": "ready", "start_offset_ms": 0, "duration_ms": 30000, "attachment": { "ref": "/input/audio/part-0" } } } } ``` ## Design notes - Prefer keys and IDs native to the framework rather than heuristics. Replies match turns by LiveKit's identities, not timestamps. An STT span runs from the first result to the final, under the turn it began in. Heard text comes from the sink's synchronized transcript. - Lean on more reliable mechanics for timing. Agent playout is timed by the sink, even after an interruption. User speech is selected only with a local VAD, since provider speech events arrive late. - Identities can only be added when LiveKit knows them. Without rooms, diarization, or PII permitted, some forms of identification are not always possible. - New fields follow #2724's PII gate.
1 parent eb59da8 commit 9c0665f

44 files changed

Lines changed: 17338 additions & 2166 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.changeset/livekit-audio-v3.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"braintrust": minor
3+
---
4+
5+
feat(livekit): Trace LiveKit voice sessions with audio schema v3

0 commit comments

Comments
 (0)