Repository navigation
Commit 9c0665f
authored
feat(livekit): Trace LiveKit voice sessions with audio schema v3 (#2726)
Based on #2725
## Why
LiveKit voice traces now follow audio schema v3. Some of the major
changes from this:
- Speech and user turns carry `audio.selections` in reference to the
session's timeline, so a span points at its exact place in the
recording. This is recorded even when audio capture is off.
- Attribution for sources, participants (e.g. a caller), and speakers
(individual voices) including support for diarization (identifying which
text belonged to what speaker.)
- Links between when operations, when possible. e.g. A model response
links to the transcripts in its context, a tool to the response that
requested it, and played speech to the synthesis and model response that
generated it.
## Example
A two-caller order call with a diarizing STT (from the e2e fixture):
```
session audio.schema.version: 3.0.0 · sources · speakers · segments · timeline
├── user_turn input: "Hi. I'm calling with my partner, … Where is order demo one zero four two? …"
│ ├── user_speech ×4 audio.selections
│ ├── stt "Hi. I'm calling with my partner," speaker-1
│ ├── stt "who has a question about our order." speaker-1
│ ├── stt "Where is order demo one zero four two?" speaker-2
│ └── …
└── agent_turn input: (the transcript above) · turn.reply_to → user_turn
├── llm_response uses_context → each stt
├── lookup_order requested_by → llm_response
├── llm_response uses_context → each stt · consumes_tool_result → lookup_order
├── tts generated_by → llm_response
└── agent_speech "Your order … is scheduled for delivery on Friday."
audio.selections · generated_by → tts
```
Session metadata, illustrated for a call with a room participant:
```json
{
"audio.schema.version": "3.0.0",
"audio.sources": { "session-input": { "label": "Agent input audio" }, "agent-output": { "label": "Agent audio" } },
"audio.speakers": { "speaker-1": { "type": "user", "attributes": { "contrib.livekit.speaker_id": "0" } } },
"audio.participants": { "participant-1": { "type": "user", "attributes": { "contrib.livekit.participant_id": "PA_x7Kq", "contrib.livekit.participant_kind": "SIP" } } },
"audio.segments": { "part-0": { "state": "ready", "start_offset_ms": 0, "duration_ms": 30000, "attachment": { "ref": "/input/audio/part-0" } } }
}
```
## Design notes
- Prefer keys and IDs native to the framework rather than heuristics.
Replies match turns by LiveKit's identities, not timestamps. An STT span
runs from the first result to the final, under the turn it began in.
Heard text comes from the sink's synchronized transcript.
- Lean on more reliable mechanics for timing. Agent playout is timed by
the sink, even after an interruption. User speech is selected only with
a local VAD, since provider speech events arrive late.
- Identities can only be added when LiveKit knows them. Without rooms,
diarization, or PII permitted, some forms of identification are not
always possible.
- New fields follow #2724's PII gate.1 parent eb59da8 commit 9c0665f
44 files changed
Lines changed: 17338 additions & 2166 deletions
File tree
- .changeset
- e2e/scenarios/livekit-instrumentation
- __cassettes__
- livekit-v1-diarized.cassette.blobs
- livekit-v1-latest-diarized.cassette.blobs
- __snapshots__
- fixtures
- js/src
- instrumentation/plugins
- vendor-sdk-types
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
0 commit comments