Skip to content

[AI Agents] Add GPT Live WebSocket support, samples, and tests #50877

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mike Guo (guotuofeng) wants to merge 13 commits into from
Open

[AI Agents] Add GPT Live WebSocket support, samples, and tests #50877

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
774b5c8
[AI Agents] Set up stacked GPT Live WebSocket work
guotuofeng Oct 20, 2026
eda4ad3
[AI Agents] Add native GPT Live WebSocket clients and voice samples
guotuofeng Oct 20, 2026
b62a157
Add GPT Live persisted conversation and audio samples
guotuofeng Oct 21, 2026
2452236
Refactor GPT Live samples to share conversation utilities
guotuofeng Oct 25, 2026
2b26a8c
Add typed GPT Live events and extract shared WebSocket transports
guotuofeng Oct 25, 2026
7222caa
Move voice agent definition helper into realtime sample utilities
guotuofeng Oct 26, 2026
4a27c4a
Use typed GPT Live events by default and update samples
guotuofeng Oct 26, 2026
68c12da
Use typed GPT Live client events and preserve explicit JSON nulls
guotuofeng Oct 26, 2026
1fe7462
Refactor and organize GPT Live sample conversation helpers
guotuofeng Oct 26, 2026
0f408f9
Unify GPT Live sync and async lifecycle handling
guotuofeng Oct 26, 2026
bc0eb75
Suppress external event type checks for live WebSocket clients
guotuofeng Oct 27, 2026
9d0c84d
Regenerate agents Checkstyle suppressions
guotuofeng Oct 27, 2026
50815d3
Move WebSocket transport release note to unreleased section
guotuofeng Oct 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 34 additions & 1 deletion sdk/ai/azure-ai-agents/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,50 @@
### Features Added

- Added Live voice-agent definitions and preview Live conversation REST clients for persisted conversations, items,
and audio. Live WebSocket transport is not included.
and audio.
- Added preview synchronous and asynchronous GPT Live WebSocket clients through
`AgentsClientBuilder.beta().buildBetaVoiceAgentLiveWebSocketClient()` and
`buildBetaVoiceAgentLiveWebSocketAsyncClient()`. Native Live events are sent and received as lossless JSON
using `BinaryData`, separately from the Realtime protocol clients.
- Added typed GPT Live sends through `sendEvent(com.openai.models.live.ClientEvent)` and typed receives through
`receiveTypedEvents()`, retaining the raw JSON APIs. The synchronous client also supports per-event receive timeouts.
Live transports use native OpenAI `ServerEvent` models while retaining original JSON for raw receivers.
Live and Realtime clients select their own routes and headers without a shared protocol-mode flag.
- Added synchronous/asynchronous persisted GPT Live conversation history and audio-download examples with offline tests.
- Added opt-in persistence to the GPT Live text and microphone/speaker samples using `FOUNDRY_GPT_LIVE_STORE` (default `false`),
with explicit startup output when the service does not provide a persisted conversation ID.
- Added runnable native GPT Live samples for synchronous/asynchronous multi-turn text, asynchronous microphone
and speaker audio, and Responses backend function tools, including sample-agent creation and cleanup.
Model deployment names default to `gpt-live-1` and `gpt-5.6-terra` when their environment variables are unset.
The audio sample requests an opening greeting, labels spoken transcripts without duplicating backend text,
reports moderation failures explicitly, and shuts down shared Reactor schedulers when its standalone main exits.
Both text samples support interactive terminal input with blank-line, `exit`, and `quit` termination,
preserve scripted command-line prompts, and shut down shared Reactor schedulers when their standalone mains exit.
Both interactive text modes request a spoken greeting, play frontend audio and print spoken transcripts
instead of backend working notes. It preserves server closure diagnostics and handles remote hangup normally.
Interactive voice mode streams paced silent PCM input while typing (consuming voice-session usage), uses
blocking console line reads instead of readiness polling, and uses the native console reader when attached.
Assistant transcript fragments are assembled into utterances independently of backend and typed turns.
Completed text is buffered while typing on the `You:` line; audio and input remain available, and incomplete
text is not split or relabeled at Enter. Speech-inactivity boundaries remain heuristic.
The first input prompt waits for greeting audio and text, a one-second quiet period, and queued speaker playback.
Continuous silent output does not extend that quiet period, and paced silent input tolerates send backpressure
without overflowing a periodic timer.
Removed redundant blank lines between console input, assistant transcripts, and session closure.

### Breaking Changes

### Bugs Fixed

- Updated the synchronous live text sample to replace an existing sample agent on a creation conflict and
delete the replacement on exit, including when the conversation fails.
- Fixed voice-agent definition deserialization through `AgentDefinition` and `VoiceAgentBaseDefinition` to select
Realtime for an omitted or `realtime` sub-protocol and Live for `live`.

### Other Changes

- Extracted shared protocol-neutral WebSocket transports for Live and Realtime, keeping independent blocking
and asynchronous implementations and protocol-specific event decoding in the respective clients.
- Deprecated `BrowserAutomationPreviewTool`, `BrowserAutomationPreviewToolboxTool`, and the corresponding
`BROWSER_AUTOMATION_PREVIEW` tool-type values. Use `BrowserAutomationTool`, `BrowserAutomationToolboxTool`,
and the `BROWSER_AUTOMATION` values instead. See [Browser Automation documentation](https://aka.ms/bat/docs).
Expand Down
211 changes: 211 additions & 0 deletions sdk/ai/azure-ai-agents/README.md
View file
Open in desktop

Large diffs are not rendered by default.

52 changes: 24 additions & 28 deletions sdk/ai/azure-ai-agents/checkstyle-suppressions.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,56 +3,52 @@
<!-- This file is generated by the /eng/scripts/linting_suppression_generator.py script. -->

<suppressions>
<suppress files="com.azure.ai.agents.ResponsesAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.ResponsesClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.util.StreamingResponseUtils.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.AgentsClientBuilder.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.AgentsAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.AgentsClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.FileSearchTool.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.PromptAgentDefinition.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.AgentsClientBuilder.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.BetaMemoryStoresAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.BetaMemoryStoresClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.BetaVoiceAgentLiveWebSocketSessionAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.BetaVoiceAgentLiveWebSocketSessionClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.ResponsesAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.ResponsesClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.AgentOptimizationBaselineAgentConfiguration.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.AgentOptimizationToolsMutation.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.ApiError.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.AudioTranscription.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimePcmAudioFormat.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.FileSearchTool.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.PromptAgentDefinition.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationCreatedEvent.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemAssistantMessage.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemFunctionCall.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemFunctionCallOutput.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemAssistantMessage.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemSystemMessage.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemUserMessage.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationCreatedEvent.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemInputAudioTranscriptionCompletedEvent.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemInputAudioTranscriptionDeltaEvent.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemInputAudioTranscriptionFailedEvent.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemSystemMessage.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeConversationItemUserMessage.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeErrorEvent.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimePcmAudioFormat.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeRateLimitsUpdatedEvent.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeResponseContentPartAddedEvent.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeResponseContentPart.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeSessionConfiguration.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeResponseContentPartAddedEvent.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeSemanticVadTurnDetection.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeSessionAudioInput.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeSessionConfiguration.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeTranscriptionSessionAudioInput.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.RealtimeSemanticVadTurnDetection.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentDefinition.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentInputTranscription.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentSemanticVadTurnDetection.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentLlmGeneratedGreetingConfiguration.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentRealtimeResponse.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentRealtimeResponseBase.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentResponseCreateOptions.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentSemanticVadTurnDetection.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentSessionResponseConfiguration.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentSessionUpdateConfiguration.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceConversation.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceResponse.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceResponseBase.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentDefinition.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.models.VoiceAgentLlmGeneratedGreetingConfiguration.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.ResponsesClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ServiceClientCheck" />
<suppress files="com.azure.ai.agents.ResponsesAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ServiceClientCheck" />
<suppress files="com.azure.ai.agents.implementation.MultipartFormDataHelper.java" checks="io.clientcore.linting.extensions.checkstyle.checks.EnforceFinalFieldsCheck" />
<suppress files="com.azure.ai.agents.implementation.http.FoundryPolicyHelper.java" checks="io.clientcore.linting.extensions.checkstyle.checks.HttpPipelinePolicyCheck" />
<suppress files="com.azure.ai.agents.AgentsClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ServiceClientCheck" />
<suppress files="com.azure.ai.agents.util.StreamingResponseUtils.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.AgentsAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ServiceClientCheck" />
<suppress files="com.azure.ai.agents.BetaMemoryStoresAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.BetaMemoryStoresClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ExternalDependencyExposedCheck" />
<suppress files="com.azure.ai.agents.BetaAgentsClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ServiceClientCheck" />
<suppress files="com.azure.ai.agents.BetaAgentsAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ServiceClientCheck" />
<suppress files="com.azure.ai.agents.AgentsClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ServiceClientCheck" />
<suppress files="com.azure.ai.agents.ResponsesAsyncClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ServiceClientCheck" />
<suppress files="com.azure.ai.agents.ResponsesClient.java" checks="io.clientcore.linting.extensions.checkstyle.checks.ServiceClientCheck" />
</suppressions>
30 changes: 30 additions & 0 deletions sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/AgentsClientBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -629,11 +629,13 @@ public BetaAgentsClientBuilder beta() {
BetaAgentsClient.class,
BetaMemoryStoresClient.class,
BetaVoiceAgentWebSocketClient.class,
BetaVoiceAgentLiveWebSocketClient.class,
BetaVoiceAgentsTelephonyClient.class,
BetaVoiceAgentsConversationsClient.class,
BetaAgentsAsyncClient.class,
BetaMemoryStoresAsyncClient.class,
BetaVoiceAgentWebSocketAsyncClient.class,
BetaVoiceAgentLiveWebSocketAsyncClient.class,
BetaVoiceAgentsTelephonyAsyncClient.class,
BetaVoiceAgentsConversationsAsyncClient.class })
public final class BetaAgentsClientBuilder {
Expand Down Expand Up @@ -769,6 +771,34 @@ public BetaVoiceAgentWebSocketClient buildBetaVoiceAgentWebSocketClient() {
return new BetaVoiceAgentWebSocketClient(createVoiceAgentWebSocketConfiguration());
}

/**
* Builds an asynchronous WebSocket client for persisted GPT Live voice agents.
* <p>
* Applies the same endpoint, credential, proxy and client options as the realtime WebSocket client.
* Custom HTTP clients, pipelines, policies and retry settings are not supported.
*
* @return an asynchronous Live WebSocket client.
* @throws IllegalStateException if unsupported HTTP pipeline configuration is present.
*/
@Beta
public BetaVoiceAgentLiveWebSocketAsyncClient buildBetaVoiceAgentLiveWebSocketAsyncClient() {
return new BetaVoiceAgentLiveWebSocketAsyncClient(createVoiceAgentWebSocketConfiguration());
}

/**
* Builds a synchronous WebSocket client for persisted GPT Live voice agents.
* <p>
* Applies the same endpoint, credential, proxy and client options as the realtime WebSocket client.
* Custom HTTP clients, pipelines, policies and retry settings are not supported.
*
* @return a synchronous Live WebSocket client.
* @throws IllegalStateException if unsupported HTTP pipeline configuration is present.
*/
@Beta
public BetaVoiceAgentLiveWebSocketClient buildBetaVoiceAgentLiveWebSocketClient() {
return new BetaVoiceAgentLiveWebSocketClient(createVoiceAgentWebSocketConfiguration());
}

/**
* Builds a synchronous beta client for preview voice-agent telephony operations.
* <p>
Expand Down
Loading
Loading