Repository navigation
Merge main into v4 - #2773
Merged
Merged
Merge main into v4#2773
Conversation
…2724) ## Problem LiveKit can be configured to withhold PII from its telemetry, but Braintrust's LiveKit traces ignored that setting. The plugin reads LiveKit's span attributes and events before LiveKit's own redaction runs, so traces still carried transcripts, prompts, model responses, tool arguments and results, and exception details. LiveKit withholds PII when any of these apply: - `LIVEKIT_TELEMETRY_ALLOW_PII` is falsy (`0`, `false`, `no`, `off`) - the job's LiveKit Cloud project enables redaction - a span carries `lk.redaction.enabled` ## Solution When LiveKit's configuration withholds PII, the plugin applies LiveKit's own redaction rule before anything is logged. **When PII is disallowed it omits:** - `lk.*pii*` attributes, gen_ai content attributes, prompt variables and message events - transcripts, assistant and heard text, and tool definitions - exception messages and stack traces (errors show `exception details redacted`) **When PII is disallowed it retains:** - span structure, timing, models, usage, tool choice, participant IDs - opt-in audio (governed by `BRAINTRUST` flags). NOTE: `setTracerProvider(..., { allowPii: false })`, is **not** covered because LiveKit doesn't expose it to the plugin. Separately, this also fixes the LiveKit e2e websocket cassette being saved after the scenario exited.
…2725) Based on #2724 ## Why Every voice integration has to describe its session the same way under audio schema v3: sources, a shared timeline, participants, speakers, recorded segments, and each span's place on that timeline. `AudioSessionRecorder` owns that shape once, so integrations only translate their framework's events into calls. - **Provides common implementation for session shape** to reduce duplication per integration. - **Provides session-local IDs** (`participant-N`, `speaker-N`) derived from native values. - **With audio capture disabled, it can still describes the session**, so timing and identities can still be relayed on the trace. In sum, this enables LiveKit's move to schema v3 (next PR), and future voice integrations. ## Example How an integration uses it: ```ts const audio = new AudioSessionRecorder({ origin: Date.now(), sources: { "session-input": { label: "Agent input audio" }, "agent-output": { label: "Agent audio" }, }, audio: { sampleRate: 48_000, segmentBytes }, // omit to describe without recording async onFile({ id, data, mimeType, metadata }) { // Upload `data` as input.audio[id], then log it with its audio.segments entry. session.log({ input: { audio: { [id]: await upload(data, mimeType) } }, metadata }); }, }); session.log({ metadata: audio.session() }); // schema version, sources, timeline const { id, added } = audio.participant("PA_x7Kq", { type: "user", attributes: { "contrib.livekit.participant_id": "PA_x7Kq" }, }); if (added) session.log({ metadata: added }); // audio.participants, first time only span.log({ metadata: { "audio.participant_ids": [id] } }); audio.record("session-input", pcm, { sampleRate: 16_000, channelCount: 1, offsetMs }); span.log({ metadata: { "audio.selections": audioSelections([{ start, end }]) } }); await audio.finish(); ``` ## Design notes ``` integration ──calls──► AudioSessionRecorder ──returns──► metadata to log │ └─ AudioRecorder (when recording) ─► onFile(segment + entry) ``` - The `AudioSessionRecorder` is responsible for managing the session, resolving references, etc but it is not responsible for instrumentation or trace data itself... it merely returns metadata that instrumentation can log. - It tracks commonly referenced objects like `participant` and `speaker`, and helps spans map to a common, PII-less ID. (e.g. `participant-1`)
Based on #2725 ## Why LiveKit voice traces now follow audio schema v3. Some of the major changes from this: - Speech and user turns carry `audio.selections` in reference to the session's timeline, so a span points at its exact place in the recording. This is recorded even when audio capture is off. - Attribution for sources, participants (e.g. a caller), and speakers (individual voices) including support for diarization (identifying which text belonged to what speaker.) - Links between when operations, when possible. e.g. A model response links to the transcripts in its context, a tool to the response that requested it, and played speech to the synthesis and model response that generated it. ## Example A two-caller order call with a diarizing STT (from the e2e fixture): ``` session audio.schema.version: 3.0.0 · sources · speakers · segments · timeline ├── user_turn input: "Hi. I'm calling with my partner, … Where is order demo one zero four two? …" │ ├── user_speech ×4 audio.selections │ ├── stt "Hi. I'm calling with my partner," speaker-1 │ ├── stt "who has a question about our order." speaker-1 │ ├── stt "Where is order demo one zero four two?" speaker-2 │ └── … └── agent_turn input: (the transcript above) · turn.reply_to → user_turn ├── llm_response uses_context → each stt ├── lookup_order requested_by → llm_response ├── llm_response uses_context → each stt · consumes_tool_result → lookup_order ├── tts generated_by → llm_response └── agent_speech "Your order … is scheduled for delivery on Friday." audio.selections · generated_by → tts ``` Session metadata, illustrated for a call with a room participant: ```json { "audio.schema.version": "3.0.0", "audio.sources": { "session-input": { "label": "Agent input audio" }, "agent-output": { "label": "Agent audio" } }, "audio.speakers": { "speaker-1": { "type": "user", "attributes": { "contrib.livekit.speaker_id": "0" } } }, "audio.participants": { "participant-1": { "type": "user", "attributes": { "contrib.livekit.participant_id": "PA_x7Kq", "contrib.livekit.participant_kind": "SIP" } } }, "audio.segments": { "part-0": { "state": "ready", "start_offset_ms": 0, "duration_ms": 30000, "attachment": { "ref": "/input/audio/part-0" } } } } ``` ## Design notes - Prefer keys and IDs native to the framework rather than heuristics. Replies match turns by LiveKit's identities, not timestamps. An STT span runs from the first result to the final, under the turn it began in. Heard text comes from the sink's synchronized transcript. - Lean on more reliable mechanics for timing. Agent playout is timed by the sink, even after an interruption. User speech is selected only with a local VAD, since provider speech events arrive late. - Identities can only be added when LiveKit knows them. Without rooms, diarization, or PII permitted, some forms of identification are not always possible. - New fields follow #2724's PII gate.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paul Tancre <paultancre@p-tancre.home> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#2731) > 🤖 **This text was AI-generated.** The `invoke()` docstring said top-level `metadata` is passed to the function as its `metadata` argument. That is true for prompts (`{{metadata}}`), but not for code functions. The runtime sends top-level `metadata` as hook data, and a Python handler's `metadata` argument is only filled from `input`: ```typescript await invoke({ ..., input: { input: "x" }, metadata: { marker: "top" } }); ``` ```python def handler(input=None, metadata=None, hooks=None): metadata # None hooks.metadata # {"marker": "top"} ``` A customer's scorer therefore saw an empty `metadata` argument on every row. The docstring now explains that code functions receive top-level metadata through the hooks (the second handler argument in TypeScript, `hooks.metadata` in Python), and shows how to fill the `metadata` argument by nesting it in `input`: ```typescript await invoke({ ..., input: { input: "x", metadata: { marker: "nested" } } }); ``` The runtime behaviour is unchanged. The same wrong sentence also appears in the generated `js/src/generated_plain_types.ts`. It comes from the backend typespec (`typespecs/src/functions.ts`) and needs fixing there, then regenerating. Mirrors braintrustdata/braintrust-sdk-python#1073. Refs: SDK-791 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
build off of #2751 fixes #2750 > 🤖 **This text was AI-generated.** ## Problem With AI SDK v7, file parts carry their bytes in a tagged wrapper (`SharedV4FileData`): ```ts { type: "file", mediaType: "image/png", data: { type: "data", data: Uint8Array } } ``` The AI SDK plugin only turned bare bytes into attachments, so it copied this wrapper into the span as is. `deepCopyEvent` then turned the `Uint8Array` into an object with one key per byte. 1 MB of image became about 120 MB of heap on every model call (#2750). The plugin fix in #2751 had only a unit test. No e2e scenario sent a v7 tagged file part, so the real `streamText` path was never tested. ## Solution The plugin now converts the inner bytes to an attachment and keeps the v7 wrapper around it. The logged span has the same shape as the v7 input and matches how decision images are already logged: ```text before: data: { type: "data", data: { "0": 137, "1": 80, ... } } after: data: { type: "data", data: <braintrust_attachment image/png> } ``` The `ai-sdk-instrumentation` scenario gets a new `ai-sdk-stream-image-file-operation`. It streams a real 32x32 red PNG to OpenAI and checks that the top-level call span and the model call span each log one PNG attachment and no raw bytes. v7 sends the tagged form; v5 and v6 send plain bytes. The test fails without the fix. Other changes in this commit: - Remove the plugin unit test and two live-API image tests in `ai-sdk.test.ts`, which the new e2e test covers. Delete their unused `test-image.png` fixture. - Replace `gpt-4.1-nano` (deprecated) and `gpt-4o-mini` with `gpt-5.4-nano-2026-03-17`. AI SDK v3 stays on `gpt-4o-mini` because its OpenAI provider sends `max_tokens`, which GPT-5 models reject. - Send `providerOptions: { openai: { store: false } }` on multi-step tool calls for AI SDK v5+. This sends earlier tool calls in full instead of by ID, so recording also works for OpenAI orgs with Zero Data Retention. - Re-record all scenario cassettes and snapshots. --------- Co-authored-by: Paulo Martins <paulo.martins@nectir.io>
- Record `requested_service_tier` and `served_service_tier` in trace metadata so users can compare the tier requested with the tier the provider actually used. - Cover OpenAI, Anthropic, Google, Bedrock, Groq, OpenRouter, AI SDK, LangChain, and OpenAI Agents, including supported batch calls. - Normalize tier values consistently and leave missing values unset, following the [service-level spec](braintrustdata/braintrust-spec#86). - Preserve the last reported tier when a stream fails or is cancelled.
## Summary Upgrades the `braintrust` package's `minimatch` dependency from `^10.2.5` to `^11.0.1`. All vulnerable `minimatch` versions are below 11.0.1. Because `braintrust` (up to and including 3.43.0) pins `minimatch` to 10.x, downstream users can't get the fix without adding an override. - `js/package.json`: `minimatch` `^10.2.5` → `^11.0.1`. The lockfile resolves to 11.0.5, which adds `@isaacs/nonrecursive-glob-match` as a transitive dependency. - Only the CLI uses `minimatch`, for file pattern matching in `js/src/cli/index.ts`. The named `minimatch` export it uses hasn't changed, so no code changes were needed. - Node support: 11.0.4+ declares `node: 20 || >=22`, which matches our CI matrix (20/22/24/26). The `^11.0.1` floor still lets Node 18 installs resolve to a patched 11.0.1–11.0.3. ## Testing - `pnpm run check:typings` in `js/` - `pnpm exec vitest run src/cli` (7 files, 56 tests passing)
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Prepares a release by updating package versions, changelogs, and consumed changeset files. ## Packages - `@braintrust/langchain-js`: `0.3.0` - `@braintrust/temporal`: `1.1.0` - `braintrust`: `3.44.0` Source-main-sha: 8f1bac6 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Abhijeet Prasad (AbhiPrasad)
approved these changes
Nov 12, 2026
- Add tracing for `client.decisions.create()` through `wrapGoogleGenAI` and automatic instrumentation. These calls previously produced no decision spans. - Record inputs, questions, answers, model details, and token usage. Convert Google’s `noul` answers to boolean probabilities in traces. - Capture media inputs as attachments, subject to the existing attachment and audio capture settings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges
mainat7801fc33intov4.Before the release metadata update, validated the workspace build, package type checks, lint, 2,611 core tests, Vitest wrapper tests, and Temporal tests.
The full e2e replay passed: 73 files, 2,886 tests passed, and 10 skipped.
The changed e2e scenarios passed three consecutive replay runs, with 2,046 tests passed in each of the two additional runs.
Verified production forwarding and every generated log and experiment link against recorded traces in Braintrust.
Also includes the 3.44.0 release metadata from main while keeping the old LangChain package removed.
After that update, the workspace build, package type checks, 23 Temporal tests, and 746 replay tests passed.
Verified all nine generated replay log links against traces in Braintrust.
Use a merge commit for this PR to retain the
mainancestry.