Repository navigation
telemetry: send a session summary in the recording header - #2824
jmcclanahan wants to merge 9 commits into
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 881dcea The changes in this PR will be included in the next version bump. This PR includes changesets to release 41 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…ssion summary The report's events are not uploaded, so the header is the only way agent evals can see the close reason and error counts. Drop the unused "version" key: every key is optional and new keys are additive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
yeah as you stated in slack, once we merge livekit/protocol#1884, then the failing CI checks will pass. |
JustinScitech
left a comment
JustinScitech
left a comment
There was a problem hiding this comment.
Seems like there's some failing builds
| if (report.duration !== undefined) { | ||
| summary.duration_ms = Math.round(report.duration); | ||
| } |
There was a problem hiding this comment.
Wondering the same thing in PR https://github.com/livekit/agents/pull/8069/changes about the report duration non-existing for audio transcripts
There was a problem hiding this comment.
The recording header's session summary is now the session report without chat history, events and the local recording path, plus agent_name, duration_ms, user_turns, error counts and close_reason as a livekit.agent.CloseReason value name. sessionReportToJSON takes options to omit chat_history and events; its default output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ToJSON sessionReportToJSON now always appends agent_name, duration_ms, user_turns, errors and close_reason after sdk_version, with null for unknown values. The recording header summary is only the trimmed report JSON without audio_recording_path, so the logic lives in one place. SessionReport gains agentName, filled by JobContext.makeSessionReport. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol#1887) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…helper
The local audio_recording_path is harmless, so the summary is exactly
sessionReportToJSON(report, { includeChatHistory: false, includeEvents: false }).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing-session-summary # Conflicts: # agents/src/voice/report.test.ts
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| sdk_version: version, | ||
| agent_name: report.agentName ?? null, | ||
| // The report has no duration when audio was not recorded. | ||
| duration_ms: report.duration !== undefined ? Math.round(report.duration) : null, |
There was a problem hiding this comment.
🟡 Recording setup inflates session duration
With audio recording enabled, duration_ms includes startup time before the session starts. RecorderIO.start() precedes the session's _startedAt, so duration-based eval selection uses an inflated length.
Learn more
The session report exposes a duration calculated from the audio recording start when audio is present createSessionReport. Audio recording starts during session startup RecorderIO.start, while the session start timestamp is assigned after startup work finishes AgentSession. The new summary field therefore includes setup time in sessions with audio, unlike sessions without audio. The header uses the same report duration uploadSessionReport.
Example: Recording begins at 12:00:00, session startup finishes at 12:00:05, and the session ends at 12:01:05. The summary reports 65 seconds instead of the session's 60 seconds.
Recommended fix: Derive the session summary duration from report.timestamp - report.startedAt for started sessions. Decide separately whether the recording header needs audio duration or session duration, and add a startup-delay test for both.
Was this helpful? React with 👍 or 👎 to provide feedback.
CR_<REASON>, checked against livekit.agent.CloseReason; task_completed is the one exception (CR_COMPLETED). Replaces the hand-written mapping, matching the Python SDK. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
… fill agent_name from the upload Addresses review: recording starts before the session, so duration_ms is timestamp - startedAt; a custom 'completed' shutdown string is CR_UNKNOWN, not CR_COMPLETED; a report without agentName takes the upload's agentName. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Description
Agent evals choose which sessions to score from a JSON "session summary" the SDK sends in the recording upload header (
MetricsRecordingHeader.session_summary, livekit/protocol#1884). This PR makes agents-js send it. All summary fields are computed insessionReportToJSON(), so the logic exists in one place; the upload only trims the report JSON.Draft until
@livekit/protocol1.53.1+ is released withsession_summaryandCloseReason. Depends on livekit/protocol#1884 (field) and livekit/protocol#1887 (livekit.agent.CloseReasonenum). This commit does not typecheck against the published@livekit/protocol@1.50.4.Changes Made
sessionReportToJSON(report, options?)takes{ includeChatHistory?, includeEvents? }, both defaulttrue;falseomits the key. New exported typeSessionReportJSONOptions.sessionReportToJSONnow always appends, aftersdk_version:agent_name(string ornull),duration_ms(rounded ms ornull),user_turns(usermessageitems in the chat history),errors(counts forllm,stt,tts,realtime,interruption, always all five),close_reason(thelivekit.agent.CloseReasonvalue name of the last close event, e.g.CR_PARTICIPANT_DISCONNECTED, ornullwithout a close event; custom shutdown strings map toCR_UNKNOWN). The existing keys keep their key, value and order.errorsandclose_reasonare computed from the events even wheneventsis omitted.SessionReportandSessionReportOptionsgainagentName?;JobContext.makeSessionReportsets it from the job (API report updated).sessionReportToJSON(report, { includeChatHistory: false, includeEvents: false }).usageis the report's per-model list (each entry has itstype, e.g.llm_usage), ornull.report.tsdoes not need the enum at runtime. Notags(agents-js has no Tagger), nomodels(derivable fromusage).durationis now set from the report (ms), which removes the TODO.@livekit/agentspatch.Pre-Review Checklist
@livekit/protocolpatched withsession_summaryandCloseReason. Against the published 1.50.4,tscfails only on the missingCloseReasonandsessionSummary.Testing
agents/src/voice/report.test.ts: default output keeps the old keys in order with the new keys appended;includeChatHistory/includeEventsomit their keys; summary fields (agent name, rounded duration, user turns, error counts, last close reason) andnullfor unknown values; every SDKCloseReasonmaps to a proto name other thanCR_UNKNOWNthat exists in the generated enum, and custom reasons map toCR_UNKNOWN.agents/src/telemetry/traces.test.ts: the decoded headersession_summaryequals the trimmed report JSON, anddurationis set.agents/src/telemetry,agents/src/voice,agents/src/jobpass 1864/1864 against the patched protocol package.@livekit/protocolto 1.53.1+ once released. That bump will also failchat_item_proto_coverageon the newFunctionCall.extrafield, which is unrelated to this PR.Additional Notes
Related: livekit/protocol#1884 (field), livekit/protocol#1887 (close reason enum), livekit/cloud-observability#656 (stores it), livekit/agents-private#745 (reads it). The schema there must allow
nullforduration_ms,close_reason,usageandagent_name.🤖 Generated with Claude Code