diff --git a/sdk/ai/azure-ai-agents/CHANGELOG.md b/sdk/ai/azure-ai-agents/CHANGELOG.md index f91b39f88c345..7075ee9c7bf27 100644 --- a/sdk/ai/azure-ai-agents/CHANGELOG.md +++ b/sdk/ai/azure-ai-agents/CHANGELOG.md @@ -14,6 +14,38 @@ voice-agent response input and user-message content, with a shared `RealtimeResponseInputItemStatus` enum. - Added preview voice-agent system tools, transfer routing, and system-tool intent and input-collection events. - Added preview voice-agent telephony campaigns with recipient validation and scheduling. +- Added Live voice-agent definitions and preview Live conversation REST clients for persisted conversations, items, + and audio. +- Added preview synchronous and asynchronous GPT Live WebSocket clients through + `AgentsClientBuilder.beta().buildBetaVoiceAgentsLiveWebSocketClient()` and + `buildBetaVoiceAgentsLiveWebSocketAsyncClient()`. Native Live events are sent and received as lossless JSON + using `BinaryData`, separately from the Realtime protocol clients. +- Added typed GPT Live sends through `sendEvent(com.openai.models.live.ClientEvent)` and typed receives through + `receiveEvents()`, retaining the raw JSON APIs. The synchronous client also supports per-event receive timeouts. + Live session clients decode native OpenAI `ServerEvent` models; shared transports buffer only raw messages. + Raw receivers do not depend on JSON parsing or typed decoding succeeding. + Live and Realtime clients select their own routes and headers without a shared protocol-mode flag. +- Added synchronous/asynchronous persisted GPT Live conversation history and audio-download examples with offline tests. +- Added opt-in persistence to the GPT Live text and microphone/speaker samples using `FOUNDRY_GPT_LIVE_STORE` (default `false`), + with explicit startup output when the service does not provide a persisted conversation ID. +- Added runnable native GPT Live samples for synchronous/asynchronous multi-turn text, asynchronous microphone + and speaker audio, and Responses backend function tools, including sample-agent creation and cleanup. + Model deployment names default to `gpt-live-1` and `gpt-5.6-terra` when their environment variables are unset. + The audio sample requests an opening greeting, labels spoken transcripts without duplicating backend text, + reports moderation failures explicitly, and shuts down shared Reactor schedulers when its standalone main exits. + Both text samples support interactive terminal input with blank-line, `exit`, and `quit` termination, + preserve scripted command-line prompts, and shut down shared Reactor schedulers when their standalone mains exit. + Both interactive text modes request a spoken greeting, play frontend audio and print spoken transcripts + instead of backend working notes. It preserves server closure diagnostics and handles remote hangup normally. + Interactive voice mode streams paced silent PCM input while typing (consuming voice-session usage), uses + blocking console line reads instead of readiness polling, and uses the native console reader when attached. + Assistant transcript fragments are assembled into utterances independently of backend and typed turns. + Completed text is buffered while typing on the `You:` line; audio and input remain available, and incomplete + text is not split or relabeled at Enter. Speech-inactivity boundaries remain heuristic. + The first input prompt waits for greeting audio and text, a one-second quiet period, and queued speaker playback. + Continuous silent output does not extend that quiet period, and paced silent input tolerates send backpressure + without overflowing a periodic timer. + Removed redundant blank lines between console input, assistant transcripts, and session closure. ### Breaking Changes @@ -51,9 +83,16 @@ - Preserved handoff state when serializing and deserializing voice-agent session response configurations. - Preserved exact numeric values when round-tripping agent structured input defaults and schemas, telephony structured inputs, and response structured inputs through `BinaryData`. +- Fixed OpenAI model JSON conversion to use an explicit Jackson serializer, preserving explicit nulls without + depending on the application's Azure JSON serializer provider. +- Removed query values from WebSocket proxy-failure diagnostics to avoid logging structured inputs. +- Updated the synchronous live text sample to replace an existing sample agent on a creation conflict and + delete the replacement on exit, including when the conversation fails. ### Other Changes +- Extracted shared protocol-neutral WebSocket transports for Live and Realtime, keeping independent blocking + and asynchronous implementations and protocol-specific event decoding in the respective clients. - Regenerated the client with `@azure-tools/openai-typespec` version `1.34.1`. ## 2.9.0 (2026-10-30) @@ -235,8 +274,8 @@ subagents, persistence, avatars, and animation. - Added `createAgentFromPrompt` and `createAgentFromPromptWithResponse` to `BetaAgentsClient` and `BetaAgentsAsyncClient` for generating and creating agents from kind-specific high-level inputs. -- Added preview `BetaVoiceAgentWebSocketClient`, `BetaVoiceAgentWebSocketAsyncClient`, - `BetaVoiceAgentWebSocketSessionClient`, and `BetaVoiceAgentWebSocketSessionAsyncClient` for authenticated realtime +- Added preview `BetaVoiceAgentsWebSocketClient`, `BetaVoiceAgentsWebSocketAsyncClient`, + `BetaVoiceAgentsWebSocketSessionClient`, and `BetaVoiceAgentsWebSocketSessionAsyncClient` for authenticated realtime `wss://` sessions. The clients support typed and raw events, text and audio input, response cancellation, client-executed function results, persisted conversations, agent-version overrides, and the related realtime session, conversation-item, response, transcription, MCP, audio, avatar, and RTC event models. diff --git a/sdk/ai/azure-ai-agents/README.md b/sdk/ai/azure-ai-agents/README.md index 003d4a3770b34..e3fff2b159dc4 100644 --- a/sdk/ai/azure-ai-agents/README.md +++ b/sdk/ai/azure-ai-agents/README.md @@ -70,7 +70,8 @@ The Agents client library has the following sub-clients which group the differen - `ResponsesClient` / `ResponsesAsyncClient`: Create responses that require Azure-specific request fields, such as an explicit `AgentReference` or structured inputs. For standard OpenAI Responses API calls through an agent endpoint, use an agent-scoped OpenAI client. See the [OpenAI Responses API documentation][openai_responses_api_docs] for more information. - `BetaMemoryStoresClient` / `BetaMemoryStoresAsyncClient` **(preview)**: Manage memory stores and individual memory items for agents. - `ToolboxesClient` / `ToolboxesAsyncClient`: Manage toolboxes and toolbox versions. -- `BetaVoiceAgentWebSocketClient` / `BetaVoiceAgentWebSocketAsyncClient` **(preview)**: Open typed realtime WebSocket sessions with voice agents. +- `BetaVoiceAgentsWebSocketClient` / `BetaVoiceAgentsWebSocketAsyncClient` **(preview)**: Open typed realtime WebSocket sessions with voice agents. +- `BetaVoiceAgentsLiveWebSocketClient` / `BetaVoiceAgentsLiveWebSocketAsyncClient` **(preview)**: Open native GPT Live WebSocket sessions with persisted Live voice agents. - `BetaVoiceAgentsTelephonyClient` / `BetaVoiceAgentsTelephonyAsyncClient` **(preview)**: Manage telephony bindings, calls, transfer targets, and outbound call jobs. - `BetaVoiceAgentsConversationsClient` / `BetaVoiceAgentsConversationsAsyncClient` **(preview)**: Read and delete persisted voice-agent conversations and retrieve their responses, items, and audio. @@ -197,7 +198,8 @@ Build clients whose names start with `Beta` from `AgentsClientBuilder.beta()`. T |---|---| | `BetaAgentsClient` | `WorkflowAgents=V1Preview,ExternalAgents=V1Preview,DraftAgents=V1Preview,VoiceAgents=V1Preview,DigitalWorker=V1Preview,GitHubCopilot=V1Preview,Skills=V1Preview,MicrosoftCopilot=V1Preview` | | `BetaMemoryStoresClient` | `MemoryStores=V1Preview` | -| `BetaVoiceAgentWebSocketClient` | `VoiceAgents=V1Preview` | +| `BetaVoiceAgentsWebSocketClient` | `VoiceAgents=V1Preview` | +| `BetaVoiceAgentsLiveWebSocketClient` / `BetaVoiceAgentsLiveWebSocketAsyncClient` | `VoiceAgents=V1Preview` | | `BetaVoiceAgentsTelephonyClient` | `VoiceAgents=V1Preview` | | `BetaVoiceAgentsConversationsClient` | `VoiceAgents=V1Preview` | @@ -206,7 +208,7 @@ The async `Beta*AsyncClient` counterparts follow the same behavior. ### Realtime voice-agent sessions Create and manage preview voice agents with `VoiceAgentDefinition` and an `AgentsClient` or `AgentsAsyncClient` built -using `allowPreview(true)`. Use `BetaVoiceAgentWebSocketClient` or `BetaVoiceAgentWebSocketAsyncClient` to open a typed, +using `allowPreview(true)`. Use `BetaVoiceAgentsWebSocketClient` or `BetaVoiceAgentsWebSocketAsyncClient` to open a typed, bidirectional session with an existing voice agent. The client acquires a token for `https://ai.azure.com/.default`, negotiates the `realtime` WebSocket subprotocol, and sends the required `VoiceAgents=V1Preview` feature header automatically. @@ -989,10 +991,10 @@ AgentsClientBuilder builder = new AgentsClientBuilder() .credential(new DefaultAzureCredentialBuilder().build()) .endpoint(endpoint); -BetaVoiceAgentWebSocketClient realtimeClient - = builder.beta().buildBetaVoiceAgentWebSocketClient(); -BetaVoiceAgentWebSocketAsyncClient realtimeAsyncClient - = builder.beta().buildBetaVoiceAgentWebSocketAsyncClient(); +BetaVoiceAgentsWebSocketClient realtimeClient + = builder.beta().buildBetaVoiceAgentsWebSocketClient(); +BetaVoiceAgentsWebSocketAsyncClient realtimeAsyncClient + = builder.beta().buildBetaVoiceAgentsWebSocketAsyncClient(); ``` #### Send a synchronous text turn @@ -1021,7 +1023,7 @@ VoiceAgentWebSocketConnectionOptions options .setReceiveBufferCapacity(512) .setMaxMessageSize(8 * 1024 * 1024) .setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.ERROR); -try (BetaVoiceAgentWebSocketSessionClient session = realtimeClient.openWebSocketSession(agentName, options)) { +try (BetaVoiceAgentsWebSocketSessionClient session = realtimeClient.openWebSocketSession(agentName, options)) { session.sendEvent(BinaryData.fromString( "{\"type\":\"response.create\",\"event_id\":\"response-1\"}")); for (RealtimeServerEvent event : session.receiveEvents()) { @@ -1042,7 +1044,7 @@ Use `close(code, reason)` or asynchronous `closeAsync(code, reason)` to send a c fit in 123 UTF-8 bytes and close codes must be valid WebSocket codes. The first asynchronous close request wins. ```java -try (BetaVoiceAgentWebSocketSessionClient session = realtimeClient.openWebSocketSession(agentName)) { +try (BetaVoiceAgentsWebSocketSessionClient session = realtimeClient.openWebSocketSession(agentName)) { session.sendText("Hello! Tell me about the services you provide."); session.createResponse(); @@ -1078,9 +1080,9 @@ Mono.usingWhen( }) .takeUntil(event -> event instanceof RealtimeResponseDoneEvent) .then(), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync, + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync, (session, error) -> session.closeAsync(), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync) + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync) .block(); ``` @@ -1101,8 +1103,225 @@ When a `RealtimeResponseFunctionCallArgumentsDoneEvent` requests a client-side t All realtime examples require `FOUNDRY_PROJECT_ENDPOINT` and optionally use `FOUNDRY_VOICE_AGENT_NAME`. The function-tool example also optionally uses `FOUNDRY_VOICE_MODEL` and `FOUNDRY_VOICE_MODEL_TYPE`. The asynchronous text and audio examples delete their generated agents by default; set `FOUNDRY_KEEP_VOICE_AGENT=true` to retain them. +The synchronous live text sample deletes and replaces the selected agent if creation returns a conflict, +then deletes the replacement on exit, including on failure. Use a dedicated sample-agent name: this also +applies when `FOUNDRY_VOICE_AGENT_NAME` is set. This synchronous sample does not support retaining its agent. + The live audio example requires a Java Sound-compatible microphone and speaker. It streams signed, little-endian, mono PCM16 audio at 24 kHz. These examples use WebSocket transport. Although the generated protocol models include WebRTC signaling events, the Java client does not provide a WebRTC peer connection or media implementation. +### GPT Live WebSocket examples (preview) + +GPT Live is a different wire protocol from Realtime. Create a persisted agent with `VoiceAgentLiveDefinition`, +then use `BetaVoiceAgentsLiveWebSocketClient` or `BetaVoiceAgentsLiveWebSocketAsyncClient` to connect. +These clients use +`wss://{project-endpoint}/agents/{agent_name}/endpoint/protocols/voice/live/sessions?api-version=v1`, +send the `Foundry-Features: VoiceAgents=V1Preview` header, and **do not** negotiate the `realtime` WebSocket +subprotocol. `allowPreview(true)` is not required. + +Foundry sends the initial native `session.start`. Wait for `session.started` before sending application events; +do not send `session.start` yourself. Use `sendEvent(ClientEvent)` and `receiveEvents()` with the +`com.openai.models.live` event models from the SDK's existing `openai-java` dependency. These are distinct +from Realtime event models. The synchronous client also offers `receiveEvents(Duration)` for bounded waits. +The `sendEvent(BinaryData)` and `receiveRawEvents()` APIs expose complete JSON objects when the original +wire representation is needed, including unknown properties, nested `response.event` backend events, and the +Foundry `conversation_id` on `session.started`. OpenAI typed models retain extra properties and unknown event +variants, but typed serialization need not preserve the original JSON formatting. +The existing samples named `VoiceAgentLive*ConversationSample` above demonstrate live connections using the +**Realtime** protocol; use the separate GPT Live clients for `VoiceAgentLiveDefinition` agents. + +For example, configure and open a synchronous Live session for an agent using Responses delegation: + +```java readme-sample-com.azure.ai.agents.live_connection_options +BetaVoiceAgentsLiveWebSocketClient liveClient = builder.beta().buildBetaVoiceAgentsLiveWebSocketClient(); +VoiceAgentLiveWebSocketConnectionOptions options = new VoiceAgentLiveWebSocketConnectionOptions() + .setReceiveBufferCapacity(512) + .setMaxMessageSize(8 * 1024 * 1024) + .setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.ERROR) + .setHandshakeTimeout(Duration.ofSeconds(30)) + .setCloseTimeout(Duration.ofSeconds(10)) + .setStoreEnabled(false); + +try (BetaVoiceAgentsLiveWebSocketSessionClient session = liveClient.openWebSocketSession(agentName, options)) { + for (ServerEvent event : session.receiveEvents()) { + if (event.isSessionStarted()) { + session.sendEvent(ClientEvent.ofResponseCreate(ResponseCreateEvent.builder().build())); + } else if (event.isSessionOutputTranscriptDelta()) { + System.out.print(event.asSessionOutputTranscriptDelta().delta()); + } else if (event.isSessionClosed() || event.isErrorEvent()) { + break; + } + } +} +``` + +Send only native Live commands valid for the configured delegation mode, whether typed or raw. +In particular, `response.item.create` and `response.create` are Responses-delegation-only commands. +Service `error` events arrive as events for the application to inspect. Typed receivers decode JSON and +fail on malformed events unless a malformed-event callback was explicitly configured. Raw receivers do not +parse JSON or construct typed models; transport size limits and UTF-8 validation still apply. + +Configure timeouts, bounded receive capacity, overflow handling, message size, persistence, agent version, +and structured inputs with `VoiceAgentLiveWebSocketConnectionOptions`. Structured inputs are passed as +unencoded JSON to `setStructuredInputs` and URL-encoded by the SDK into the `structured_inputs` query parameter. +Do not log endpoint URLs containing sensitive template inputs. Realtime-only transport and agent-session-ID +options are not exposed. Options are copied when opening a session. + +Both transports retain the realtime clients' credential, proxy, custom-header and builder validation behavior. +Custom HTTP clients, pipelines, policies, and retry settings are unsupported and fail at build time rather +than being ignored. Each session permits one receiver, shared by typed and raw receive APIs. +The default receive buffer holds 256 events; overflow +terminates the session unless a drop strategy explicitly permits event loss. + +#### Runnable GPT Live scenarios + +These examples create a uniquely named `VoiceAgentLiveDefinition` agent with **Responses delegation**, connect +using native GPT Live WebSockets, and delete the agent on exit, including on failure. They do not modify or delete +an existing agent. Set these environment variables before running a sample's `main` method: + +| Variable | Value | +|---|---| +| `FOUNDRY_PROJECT_ENDPOINT` | Your Foundry project endpoint. Authentication uses `DefaultAzureCredential`. | +| `FOUNDRY_GPT_LIVE_MODEL` | Optional. Your GPT Live **deployment name**, not a model catalog ID. Defaults to `gpt-live-1` when unset. | +| `FOUNDRY_GPT_LIVE_BACKEND_MODEL` | Optional. Your Responses backend **deployment name in the Live resource**. Defaults to `gpt-5.6-terra` when unset. | +| `FOUNDRY_GPT_LIVE_STORE` | Optional, text and microphone/speaker examples. Set to `true` to persist conversation content and audio. Defaults to `false`. | +| `FOUNDRY_KEEP_GPT_LIVE_AGENT` | Optional. Set to `true` to retain the sample-created agent; defaults to `false`. | + +Each sample reads, defaults, and validates these settings and configures its client builder directly in its own `main` method. +Shared helpers receive explicit values rather than reading environment variables. +Override the model variables if your deployment names differ from these defaults. Explicitly blank values are invalid. +Generated agent names use `gpt-live--` and are at most 57 characters for these scenarios, +leaving headroom below the service's 63-character limit. + +| Scenario | Runnable sample | +|---|---| +| Synchronous multi-turn text | [GptLiveTextConversationSample.java](src/samples/java/com/azure/ai/agents/voice/GptLiveTextConversationSample.java) | +| Asynchronous multi-turn text | [GptLiveTextConversationAsyncSample.java](src/samples/java/com/azure/ai/agents/voice/GptLiveTextConversationAsyncSample.java) | +| Asynchronous microphone and speaker | [GptLiveAudioConversationAsyncSample.java](src/samples/java/com/azure/ai/agents/voice/GptLiveAudioConversationAsyncSample.java) | +| Responses backend function tool | [GptLiveFunctionToolSample.java](src/samples/java/com/azure/ai/agents/voice/GptLiveFunctionToolSample.java) | + +#### Read persisted GPT Live conversations and audio + +These REST examples read existing persisted data; they do not start a WebSocket session, create an agent, +or delete resources. + +| Scenario | Runnable sample | +|---|---| +| Synchronous history and transcripts | [GptLiveReadConversationSample.java](src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationSample.java) | +| Asynchronous history and transcripts | [GptLiveReadConversationAsyncSample.java](src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAsyncSample.java) | +| Synchronous stored audio download | [GptLiveReadConversationAudioSample.java](src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAudioSample.java) | +| Asynchronous stored audio download | [GptLiveReadConversationAudioAsyncSample.java](src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAudioAsyncSample.java) | + +Use an agent/session configured with `store=true` and retain the agent. With either text WebSocket example or the +microphone/speaker example, set +both `FOUNDRY_GPT_LIVE_STORE=true` and `FOUNDRY_KEEP_GPT_LIVE_AGENT=true`. The first sets the session persistence +override; the second prevents sample cleanup from deleting the agent. Keeping the agent alone does **not** enable +persistence. Both interactive and scripted modes print the Foundry `conversation_id` from `session.started` +when provided. If absent, startup output explicitly reports that no persisted conversation ID was provided; +it never substitutes the native session ID. +Use `session.started.conversation_id`, not the native session ID. Set: + +| Variable | Value | +|---|---| +| `FOUNDRY_PROJECT_ENDPOINT` | Required. Your Foundry project endpoint; authentication uses `DefaultAzureCredential`. No default. | +| `FOUNDRY_GPT_LIVE_AGENT_NAME` | Required. The existing Live agent's name. No default. | +| `FOUNDRY_GPT_LIVE_CONVERSATION_ID` | Required. The persisted Foundry conversation ID. No default. | +| `FOUNDRY_GPT_LIVE_ITEM_ID` | Optional, audio examples only: a speech item's ID, not a delegation item. Defaults to unset (merged audio only); explicitly blank values are invalid. | + +Model deployment settings such as `FOUNDRY_GPT_LIVE_MODEL` and `FOUNDRY_GPT_LIVE_BACKEND_MODEL` are not needed: +these samples read existing conversations rather than creating agents. +Run a history sample's `main` method with no arguments. For audio, pass one existing local output directory; +there is no default output directory. +Audio examples create uniquely named WAV files without overwriting existing files. History examples explicitly +advance the `after` cursor until an empty page because the generated list clients do not expose automatic +continuation; missing or repeated cursors fail explicitly. They show conversation status and usage seconds, +input/output transcripts, and delegation targets/response IDs. Stored speech segments are not native turns or +proof that audio was played. Treat the printed transcripts and downloaded audio as sensitive data. + +Wait for session completion and recording finalization before downloading. HTTP errors propagate: `404` can mean +missing audio; `409` can mean `recording_not_ready` or `recording_unavailable`. Inspect `Retry-After` before retrying. +BYOS metadata is reported without calling Foundry content endpoints; use the metadata's blob URI and your own +Azure Storage credentials to download BYOS audio. The current generated Live REST builder methods require explicit +preview opt-in, so these samples add `Foundry-Features: VoiceAgents=V1Preview` using an HTTP pipeline header policy. + +#### Using the interactive WebSocket samples + +Both text examples read terminal input interactively when no arguments are supplied. After each reply, +they prompt for another message; submit a blank line or enter `exit` or `quit` to close the session. +With arguments, they instead process each argument as a prompt and exit after the last reply. +They wait for `session.started`, send `response.item.create` followed by `response.create`, and wait for the +nested `response.completed` before advancing to the next turn. Scripted mode prints backend text without +speaker playback. Both interactive examples instead request a spoken opening greeting with +`session.commentary.append`, play the voice frontend's 24-kHz PCM16 audio, and print its spoken transcript +rather than backend working notes. The synchronous example uses synchronous session send/receive calls on +workers and shares the asynchronous example's voice-loop coordination. +Before displaying the first `You:` prompt, it waits for both greeting audio and transcript, one second without +speech activity, and completion of queued speaker playback. Continuous silent audio does not restart the timer. +For this heuristic, PCM16 frames with peak sample magnitude at most 256 (about -42 dBFS) are treated as silence. +The quiet period is a heuristic because native GPT Live +has no speech-completed event: a long pause within a greeting can be mistaken for its end. Missing greeting output +fails with a timeout instead of silently starting input before the introduction. +It sends 20 milliseconds of silent PCM (960 bytes) every 20 milliseconds after startup, providing continuous +audio input without opening a microphone. **This consumes voice-session usage while you type or wait.** +The silent stream is stopped and pending audio sends complete before the sample requests `session.close`. +If sending falls behind, silent-frame production waits for demand instead of accumulating timer ticks or +failing with an interval overflow; it resumes without a burst of stale frames. +Speaker playback runs off the WebSocket receive thread. If a speaker is unavailable, the sample reports that +and still prints transcripts, like the Realtime text example. +Agent creation and deletion occur on the main thread. + +Console input is line-oriented: press Enter to submit a message. The sample flushes the `You:` prompt immediately; +character echo while typing is controlled by Windows CMD/PowerShell and its terminal host, not by the service. +Interactive input uses a normal blocking line read on a worker rather than polling `Reader.ready()`, +matching the Realtime sample's console interaction. The WebSocket receiver continues independently. +When a native Java console is attached, input comes from `System.console().reader()` rather than `System.in`. +Without a console, the sample reads `System.in`, supporting redirected input. End-of-input (EOF), a blank line, +or an explicit exit command closes the session. +The terminal echoes characters on the `You:` line as you type. Assistant transcript fragments are assembled into +utterances independently of typed turns and backend completion. A one-second speech-inactivity period marks an +utterance boundary using the same heuristic as the greeting; a long pause within speech can still divide the text. +Completed utterances are printed with one `Assistant:` label each. While input is active, completed text waits +until Enter so it cannot overwrite typed characters; incomplete text stays buffered across input submissions. +Audio playback and input remain available. This dependency-free approach can delay captions until the next Enter. +Session closure flushes any remaining transcript. Completed and incomplete text share a 65,536-character buffer +limit; exceeding it fails the sample explicitly instead of dropping transcript text or growing memory indefinitely. +The native console's echoed Enter already starts a new line; the sample avoids adding another blank line. +After a remote disconnect, a console read that the operating system cannot interrupt may require Enter to release +the worker. The sample does not close shared standard input to force a pending read to return. +Backend `response.completed` permits the next typed prompt but does not mean that the frontend finished speaking: +native GPT Live has no output-audio-done event. The sample keeps receiving audio and transcripts while you type. +See [GPT-Live audio and transcript events](https://learn.microsoft.com/azure/foundry/openai/gpt-live-reference#sessionoutput_audiodelta). +Server `remote_hangup` closes the conversation normally. Other unexpected closures remain failures and include +the complete `session.closed` event, including its reason, rather than hiding the service diagnostic. + +The audio example requires a Java Sound-compatible microphone and speaker supporting raw, mono PCM16LE at +24 kHz. After `session.started`, it sends a greeting request followed by `response.create` so the agent can +open the conversation. It labels spoken transcript fragments as `You:` or `Assistant:` and omits backend +text deltas, which are not necessarily the words spoken by the voice frontend. +Press Enter to end the call; asking the model to end it does not itself close the session. +Service moderation (`content_filter`) ends the call with an explicit error and the original service event; +it is not treated as a successful completion. Audio devices and the WebSocket are released on failure. +The standalone audio and text samples' `main` methods also shut down shared Reactor schedulers on exit +to avoid lingering timer threads under `exec:java`. Do not invoke those `main` methods inside an application that +shares Reactor schedulers. + +The audio sample sends `session.input_audio.append`, plays base64-decoded `session.output_audio.delta` chunks, and +prints native input/output transcript deltas. It does not send Realtime audio-buffer commit or response-cancel +commands. Capture and playback use dedicated threads; the playback backlog is bounded to 60 seconds. +Device errors and overflow end the call explicitly. Press Enter to stop capture and finalize the session. +This example must be the only reader of standard input. + +The function-tool example configures `get_weather` on the **Responses backend**, consumes completed +`function_call` items from nested `response.output_item.done` events, then sends `function_call_output` items +with the original `call_id` before requesting continuation. Its weather results are **simulated demonstration +data**, not live weather. It does not assume lifecycle snapshots contain output items, and does not add +Foundry-managed-agent tool-result routing. + +All four scenarios request `session.close` and wait for `session.closed` before normal transport cleanup. +The asynchronous text example waits for terminal input without a timeout, while applying a 60-second timeout to +startup, each response, and finalization. The synchronous text and tool examples use a 60-second event-wait timeout; +the audio example uses a 60-second startup/finalization timeout. +An unexpected disconnect or native service error fails the sample rather than being reported as success. + ### Service API versions The client library targets the latest service API version by default. diff --git a/sdk/ai/azure-ai-agents/checkstyle-suppressions.xml b/sdk/ai/azure-ai-agents/checkstyle-suppressions.xml index d4ad356712d22..9b0654ce4e80a 100644 --- a/sdk/ai/azure-ai-agents/checkstyle-suppressions.xml +++ b/sdk/ai/azure-ai-agents/checkstyle-suppressions.xml @@ -64,6 +64,8 @@ + + diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/AgentsClientBuilder.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/AgentsClientBuilder.java index a9753e6bae0e5..233267f8c9c0a 100644 --- a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/AgentsClientBuilder.java +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/AgentsClientBuilder.java @@ -629,13 +629,15 @@ public BetaAgentsClientBuilder beta() { serviceClients = { BetaAgentsClient.class, BetaMemoryStoresClient.class, - BetaVoiceAgentWebSocketClient.class, + BetaVoiceAgentsWebSocketClient.class, + BetaVoiceAgentsLiveWebSocketClient.class, BetaVoiceAgentsTelephonyClient.class, BetaVoiceAgentsConversationsClient.class, BetaVoiceAgentsLiveConversationsClient.class, BetaAgentsAsyncClient.class, BetaMemoryStoresAsyncClient.class, - BetaVoiceAgentWebSocketAsyncClient.class, + BetaVoiceAgentsWebSocketAsyncClient.class, + BetaVoiceAgentsLiveWebSocketAsyncClient.class, BetaVoiceAgentsTelephonyAsyncClient.class, BetaVoiceAgentsConversationsAsyncClient.class, BetaVoiceAgentsLiveConversationsAsyncClient.class }) @@ -766,8 +768,8 @@ public BetaMemoryStoresClient buildBetaMemoryStoresClient() { * @throws IllegalStateException if unsupported HTTP pipeline configuration is present. */ @Beta - public BetaVoiceAgentWebSocketAsyncClient buildBetaVoiceAgentWebSocketAsyncClient() { - return new BetaVoiceAgentWebSocketAsyncClient(createVoiceAgentWebSocketConfiguration()); + public BetaVoiceAgentsWebSocketAsyncClient buildBetaVoiceAgentsWebSocketAsyncClient() { + return new BetaVoiceAgentsWebSocketAsyncClient(createVoiceAgentWebSocketConfiguration()); } /** @@ -782,8 +784,36 @@ public BetaVoiceAgentWebSocketAsyncClient buildBetaVoiceAgentWebSocketAsyncClien * @throws IllegalStateException if unsupported HTTP pipeline configuration is present. */ @Beta - public BetaVoiceAgentWebSocketClient buildBetaVoiceAgentWebSocketClient() { - return new BetaVoiceAgentWebSocketClient(createVoiceAgentWebSocketConfiguration()); + public BetaVoiceAgentsWebSocketClient buildBetaVoiceAgentsWebSocketClient() { + return new BetaVoiceAgentsWebSocketClient(createVoiceAgentWebSocketConfiguration()); + } + + /** + * Builds an asynchronous WebSocket client for persisted GPT Live voice agents. + *

+ * Applies the same endpoint, credential, proxy and client options as the realtime WebSocket client. + * Custom HTTP clients, pipelines, policies and retry settings are not supported. + * + * @return an asynchronous Live WebSocket client. + * @throws IllegalStateException if unsupported HTTP pipeline configuration is present. + */ + @Beta + public BetaVoiceAgentsLiveWebSocketAsyncClient buildBetaVoiceAgentsLiveWebSocketAsyncClient() { + return new BetaVoiceAgentsLiveWebSocketAsyncClient(createVoiceAgentWebSocketConfiguration()); + } + + /** + * Builds a synchronous WebSocket client for persisted GPT Live voice agents. + *

+ * Applies the same endpoint, credential, proxy and client options as the realtime WebSocket client. + * Custom HTTP clients, pipelines, policies and retry settings are not supported. + * + * @return a synchronous Live WebSocket client. + * @throws IllegalStateException if unsupported HTTP pipeline configuration is present. + */ + @Beta + public BetaVoiceAgentsLiveWebSocketClient buildBetaVoiceAgentsLiveWebSocketClient() { + return new BetaVoiceAgentsLiveWebSocketClient(createVoiceAgentWebSocketConfiguration()); } /** @@ -982,8 +1012,8 @@ private VoiceAgentWebSocketClientConfiguration createVoiceAgentWebSocketConfigur * * @return an asynchronous voice-agent WebSocket client. */ - private BetaVoiceAgentWebSocketAsyncClient buildBetaVoiceAgentWebSocketAsyncClient() { - return new BetaVoiceAgentWebSocketAsyncClient(createVoiceAgentWebSocketConfiguration()); + private BetaVoiceAgentsWebSocketAsyncClient buildBetaVoiceAgentsWebSocketAsyncClient() { + return new BetaVoiceAgentsWebSocketAsyncClient(createVoiceAgentWebSocketConfiguration()); } /** @@ -991,8 +1021,8 @@ private BetaVoiceAgentWebSocketAsyncClient buildBetaVoiceAgentWebSocketAsyncClie * * @return a synchronous voice-agent WebSocket client. */ - private BetaVoiceAgentWebSocketClient buildBetaVoiceAgentWebSocketClient() { - return new BetaVoiceAgentWebSocketClient(createVoiceAgentWebSocketConfiguration()); + private BetaVoiceAgentsWebSocketClient buildBetaVoiceAgentsWebSocketClient() { + return new BetaVoiceAgentsWebSocketClient(createVoiceAgentWebSocketConfiguration()); } /** diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketAsyncClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketAsyncClient.java new file mode 100644 index 0000000000000..90fca1b92c29e --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketAsyncClient.java @@ -0,0 +1,63 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents; + +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketClientConfiguration; +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; +import com.azure.ai.agents.implementation.realtime.WebSocketSessionAsyncTransport; +import com.azure.ai.agents.implementation.realtime.WebSocketTransportOptions; +import com.azure.ai.agents.implementation.utils.Beta; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; +import com.azure.core.annotation.ServiceClient; +import reactor.core.publisher.Mono; +import reactor.netty.http.client.HttpClient; + +import java.util.Objects; + +/** + * An asynchronous client for opening native GPT Live sessions with persisted Foundry Live voice agents. + * Create instances with {@link AgentsClientBuilder#beta()}. + */ +@ServiceClient(builder = AgentsClientBuilder.class, isAsync = true) +@Beta(warningText = "This class is in preview and may change in future releases.") +public final class BetaVoiceAgentsLiveWebSocketAsyncClient { + private final VoiceAgentWebSocketClientConfiguration configuration; + + BetaVoiceAgentsLiveWebSocketAsyncClient(VoiceAgentWebSocketClientConfiguration configuration) { + this.configuration = Objects.requireNonNull(configuration, "'configuration' cannot be null."); + } + + /** + * Opens a WebSocket session using the persisted Live agent configuration. + * + * @param agentName the Live voice agent name. + * @return the connected session. Each subscription opens a new connection. + */ + public Mono openWebSocketSession(String agentName) { + return openWebSocketSession(agentName, new VoiceAgentLiveWebSocketConnectionOptions()); + } + + /** + * Opens a WebSocket session. Foundry sends the initial {@code session.start}; consume {@code session.started} + * from the event stream before sending application events. No Realtime subprotocol is negotiated. + * + * @param agentName the Live voice agent name. + * @param options the connection options, copied when this method is called. + * @return the connected session. Each subscription opens a new connection. + */ + public Mono openWebSocketSession(String agentName, + VoiceAgentLiveWebSocketConnectionOptions options) { + Objects.requireNonNull(agentName, "'agentName' cannot be null."); + VoiceAgentLiveWebSocketConnectionOptions snapshot = new VoiceAgentLiveWebSocketConnectionOptions(options); + return Mono.defer(() -> { + WebSocketSessionAsyncTransport transport = new WebSocketSessionAsyncTransport(configuration, + VoiceAgentWebSocketUtils.buildLiveWebSocketUri(configuration, agentName, snapshot), + new WebSocketTransportOptions(snapshot), HttpClient.create(), null, + token -> VoiceAgentWebSocketUtils.buildHeaders(configuration, token)); + return transport.connect() + .thenReturn( + new BetaVoiceAgentsLiveWebSocketSessionAsyncClient(transport, snapshot.getMalformedEventHandler())); + }); + } +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketClient.java new file mode 100644 index 0000000000000..9044c117f70a1 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketClient.java @@ -0,0 +1,64 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents; + +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketClientConfiguration; +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; +import com.azure.ai.agents.implementation.realtime.WebSocketSessionTransport; +import com.azure.ai.agents.implementation.realtime.WebSocketTransportOptions; +import com.azure.ai.agents.implementation.utils.Beta; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; +import com.azure.core.annotation.ServiceClient; +import com.azure.core.util.logging.ClientLogger; + +import java.util.Objects; + +/** + * A synchronous client for opening native GPT Live sessions with persisted Foundry Live voice agents. + * Create instances with {@link AgentsClientBuilder#beta()}. + */ +@ServiceClient(builder = AgentsClientBuilder.class) +@Beta(warningText = "This class is in preview and may change in future releases.") +public final class BetaVoiceAgentsLiveWebSocketClient { + private static final ClientLogger LOGGER = new ClientLogger(BetaVoiceAgentsLiveWebSocketClient.class); + private final VoiceAgentWebSocketClientConfiguration configuration; + + BetaVoiceAgentsLiveWebSocketClient(VoiceAgentWebSocketClientConfiguration configuration) { + this.configuration = Objects.requireNonNull(configuration, "'configuration' cannot be null."); + } + + /** + * Opens a WebSocket session using the persisted Live agent configuration. + * + * @param agentName the Live voice agent name. + * @return the connected session. + */ + public BetaVoiceAgentsLiveWebSocketSessionClient openWebSocketSession(String agentName) { + return openWebSocketSession(agentName, new VoiceAgentLiveWebSocketConnectionOptions()); + } + + /** + * Opens a WebSocket session. Foundry sends the initial {@code session.start}; consume {@code session.started} + * from the event stream before sending application events. No Realtime subprotocol is negotiated. + * + * @param agentName the Live voice agent name. + * @param options the connection options, copied before connecting. + * @return the connected session. + * @throws IllegalArgumentException if the agent name is empty. + */ + public BetaVoiceAgentsLiveWebSocketSessionClient openWebSocketSession(String agentName, + VoiceAgentLiveWebSocketConnectionOptions options) { + Objects.requireNonNull(agentName, "'agentName' cannot be null."); + if (agentName.isEmpty()) { + throw LOGGER.logExceptionAsError(new IllegalArgumentException("'agentName' cannot be empty.")); + } + VoiceAgentLiveWebSocketConnectionOptions snapshot = new VoiceAgentLiveWebSocketConnectionOptions(options); + return new BetaVoiceAgentsLiveWebSocketSessionClient( + WebSocketSessionTransport.connect(configuration, + VoiceAgentWebSocketUtils.buildLiveWebSocketUri(configuration, agentName, snapshot), + new WebSocketTransportOptions(snapshot), null, + token -> VoiceAgentWebSocketUtils.buildHeaders(configuration, token)), + snapshot.getMalformedEventHandler()); + } +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketSessionAsyncClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketSessionAsyncClient.java new file mode 100644 index 0000000000000..352e9034d3723 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketSessionAsyncClient.java @@ -0,0 +1,148 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents; + +import com.azure.ai.agents.implementation.OpenAIJsonHelper; +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; +import com.azure.ai.agents.implementation.realtime.WebSocketSessionAsyncTransport; +import com.azure.ai.agents.implementation.utils.Beta; +import com.azure.core.util.AsyncCloseable; +import com.azure.core.util.BinaryData; +import com.openai.models.live.ClientEvent; +import com.openai.models.live.ServerEvent; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import java.util.Objects; +import java.util.function.Consumer; + +/** + * An asynchronous bidirectional native GPT Live session. + *

+ * Live events can be sent and received using {@link ClientEvent} and {@link ServerEvent}, or as complete JSON objects. + * Raw JSON preserves unknown fields, nested {@code response.event} payloads and Foundry {@code conversation_id} values. + * Typed and raw receivers share one event subscriber. Close sessions when they are no longer needed. + */ +@Beta(warningText = "This class is in preview and may change in future releases.") +public final class BetaVoiceAgentsLiveWebSocketSessionAsyncClient implements AsyncCloseable, AutoCloseable { + private final WebSocketSessionAsyncTransport transport; + private final Consumer malformedEventHandler; + + BetaVoiceAgentsLiveWebSocketSessionAsyncClient(WebSocketSessionAsyncTransport transport, + Consumer malformedEventHandler) { + this.transport = transport; + this.malformedEventHandler = malformedEventHandler; + } + + /** + * Gets the WebSocket endpoint. It may contain structured inputs; do not log sensitive query values. + * + * @return the endpoint. + */ + public String getEndpoint() { + return transport.getEndpoint(); + } + + /** + * Determines whether the transport is open, not whether {@code session.started} has been received. + * + * @return whether the WebSocket is open. + */ + public boolean isOpen() { + return transport.isOpen(); + } + + /** + * Gets the peer close code. + * + * @return the code, or null if no close frame has been received. + */ + public Integer getCloseCode() { + return transport.getCloseCode(); + } + + /** + * Gets the peer close reason. + * + * @return the reason, or null if no close frame has been received. + */ + public String getCloseReason() { + return transport.getCloseReason(); + } + + /** + * Receives native Live server events in wire order, including unknown events. + * Messages are not parsed or validated as JSON. Transport errors terminate the stream. + * + * @return the single-subscriber stream of raw messages. + */ + public Flux receiveRawEvents() { + return transport.receiveEvents(); + } + + /** + * Receives typed native Live server events. Typed and raw receivers share the single-subscriber limit. + * Use {@link #receiveRawEvents()} when the original JSON representation is needed. + * + * @return the typed server event stream. + */ + public Flux receiveEvents() { + return VoiceAgentWebSocketUtils.decodeEvents(transport.receiveEvents(), + json -> OpenAIJsonHelper.fromBinaryData(VoiceAgentWebSocketUtils.deserializeRawEvent(json), + ServerEvent.class), + malformedEventHandler, () -> transport.closeAsync(1007, "Invalid JSON event").subscribe()); + } + + /** + * Sends a typed native Live command. Foundry supplies {@code session.start}; clients must not send it. + * + * @param event the command to send. + * @return completion after the frame is written. + */ + public Mono sendEvent(ClientEvent event) { + Objects.requireNonNull(event, "'event' cannot be null."); + return Mono.defer(() -> sendEvent(OpenAIJsonHelper.toBinaryData(event))); + } + + /** + * Sends a native Live JSON event. Foundry supplies {@code session.start}; clients must not send it. + * The service validates native command semantics, including delegation restrictions. + * + * @param event the complete JSON object. + * @return completion after the frame is written, or an error for invalid JSON or a closed session. + */ + public Mono sendEvent(BinaryData event) { + return transport.sendEvent(event); + } + + /** + * Closes the WebSocket normally. + * + * @return completion when the session is closed. + */ + @Override + public Mono closeAsync() { + return transport.closeAsync(); + } + + /** + * Closes the WebSocket with the specified close frame. The first asynchronous close request wins. + * + * @param code a valid WebSocket close code. + * @param reason a non-null reason of at most 123 UTF-8 bytes. + * @return completion when the session is closed. + * @throws IllegalArgumentException if the code or reason is invalid. + */ + public Mono closeAsync(int code, String reason) { + return transport.closeAsync(code, reason); + } + + /** + * Closes the session, blocking until closure or the configured close timeout. + */ + @Override + public void close() { + transport.close(); + } +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketSessionClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketSessionClient.java new file mode 100644 index 0000000000000..0607e5e888035 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsLiveWebSocketSessionClient.java @@ -0,0 +1,167 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents; + +import com.azure.ai.agents.implementation.OpenAIJsonHelper; +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; +import com.azure.ai.agents.implementation.realtime.WebSocketSessionTransport; +import com.azure.ai.agents.implementation.utils.Beta; +import com.azure.core.util.BinaryData; +import com.azure.core.util.IterableStream; +import com.openai.models.live.ClientEvent; +import com.openai.models.live.ServerEvent; + +import java.time.Duration; +import java.util.Objects; +import java.util.function.Consumer; + +/** + * A synchronous bidirectional native GPT Live session. + *

+ * Events can be sent and received using {@link ClientEvent} and {@link ServerEvent}, or as complete JSON objects. + * Raw JSON preserves unknown fields, nested {@code response.event} payloads and Foundry {@code conversation_id} values. + * Typed and raw receivers share one event iterator. Close sessions when they are no longer needed. + */ +@Beta(warningText = "This class is in preview and may change in future releases.") +public final class BetaVoiceAgentsLiveWebSocketSessionClient implements AutoCloseable { + private final WebSocketSessionTransport transport; + private final Consumer malformedEventHandler; + + BetaVoiceAgentsLiveWebSocketSessionClient(WebSocketSessionTransport transport, + Consumer malformedEventHandler) { + this.transport = transport; + this.malformedEventHandler = malformedEventHandler; + } + + /** + * Gets the WebSocket endpoint. It may contain structured inputs; do not log sensitive query values. + * + * @return the endpoint. + */ + public String getEndpoint() { + return transport.getEndpoint(); + } + + /** + * Determines whether the transport is open, not whether {@code session.started} has been received. + * + * @return whether the WebSocket is open. + */ + public boolean isOpen() { + return transport.isOpen(); + } + + /** + * Gets the peer close code. + * + * @return the code, or null if no close frame has been received. + */ + public Integer getCloseCode() { + return transport.getCloseCode(); + } + + /** + * Gets the peer close reason. + * + * @return the reason, or null if no close frame has been received. + */ + public String getCloseReason() { + return transport.getCloseReason(); + } + + /** + * Receives native Live server events in wire order, including unknown events. + * Messages are not parsed or validated as JSON. Transport errors terminate iteration. + * + * @return the single-consumer stream of raw messages. + * @throws IllegalStateException if the receive stream was already claimed. + */ + public IterableStream receiveRawEvents() { + return receiveRawEvents(null); + } + + /** + * Receives typed native Live server events. Typed and raw receivers share the single-consumer limit. + * + * @return the typed server event stream. + */ + public IterableStream receiveEvents() { + return receiveEvents(null); + } + + /** + * Receives typed native Live events with a timeout per wait. Timeouts leave the session open. + * Use {@link #receiveRawEvents(Duration)} when the original JSON representation is needed. + * + * @param timeout a positive timeout, or null to wait indefinitely. + * @return the single-consumer typed event stream. + * @throws IllegalArgumentException if the timeout is zero or negative. + * @throws IllegalStateException if the receive stream was already claimed, or a wait times out. + */ + public IterableStream receiveEvents(Duration timeout) { + return VoiceAgentWebSocketUtils.decodeEvents( + transport.receiveEvents(timeout), json -> OpenAIJsonHelper + .fromBinaryData(VoiceAgentWebSocketUtils.deserializeRawEvent(json), ServerEvent.class), + malformedEventHandler, () -> transport.close(1007, "Invalid JSON event")); + } + + /** + * Sends a typed native Live command. Foundry supplies {@code session.start}; clients must not send it. + * This method enqueues the event; returning does not mean the frame has been written to the network. + * Later network failures surface through the receive stream and session lifecycle. + * + * @param event the command to send. + * @throws IllegalStateException if the session cannot accept the event. + */ + public void sendEvent(ClientEvent event) { + Objects.requireNonNull(event, "'event' cannot be null."); + sendEvent(OpenAIJsonHelper.toBinaryData(event)); + } + + /** + * Receives raw Live messages without JSON decoding, with a timeout per wait. + * Timeouts leave the session open so iteration can resume. + * + * @param timeout a positive timeout, or null to wait indefinitely. + * @return the single-consumer event stream. + * @throws IllegalArgumentException if the timeout is zero or negative. + * @throws IllegalStateException if the receive stream was already claimed, or a wait times out. + */ + public IterableStream receiveRawEvents(Duration timeout) { + return transport.receiveEvents(timeout); + } + + /** + * Sends a native Live JSON event. Foundry supplies {@code session.start}; clients must not send it. + * The service validates native command semantics, including delegation restrictions. + * This method enqueues the event; returning does not mean the frame has been written to the network. + * Later network failures surface through the receive stream and session lifecycle. + * + * @param event the complete JSON object. + * @throws IllegalArgumentException if the event is not a JSON object. + * @throws IllegalStateException if the session cannot accept the event. + */ + public void sendEvent(BinaryData event) { + transport.sendEvent(event); + } + + /** + * Closes the WebSocket normally. + */ + @Override + public void close() { + transport.close(); + } + + /** + * Closes the WebSocket with the specified close frame. + * + * @param code a valid WebSocket close code. + * @param reason a non-null reason of at most 123 UTF-8 bytes. + * @throws IllegalArgumentException if the code or reason is invalid. + */ + public void close(int code, String reason) { + transport.close(code, reason); + } +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketAsyncClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketAsyncClient.java similarity index 57% rename from sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketAsyncClient.java rename to sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketAsyncClient.java index 4b5ed49dece92..f3a4fa41d0fc6 100644 --- a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketAsyncClient.java +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketAsyncClient.java @@ -4,10 +4,14 @@ package com.azure.ai.agents; import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketClientConfiguration; +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; +import com.azure.ai.agents.implementation.realtime.WebSocketSessionAsyncTransport; +import com.azure.ai.agents.implementation.realtime.WebSocketTransportOptions; import com.azure.ai.agents.implementation.utils.Beta; import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; import com.azure.core.annotation.ServiceClient; import reactor.core.publisher.Mono; +import reactor.netty.http.client.HttpClient; import java.util.Objects; @@ -16,10 +20,10 @@ */ @ServiceClient(builder = AgentsClientBuilder.class, isAsync = true) @Beta(warningText = "This class is in preview and may change in future releases.") -public final class BetaVoiceAgentWebSocketAsyncClient { +public final class BetaVoiceAgentsWebSocketAsyncClient { private final VoiceAgentWebSocketClientConfiguration configuration; - BetaVoiceAgentWebSocketAsyncClient(VoiceAgentWebSocketClientConfiguration configuration) { + BetaVoiceAgentsWebSocketAsyncClient(VoiceAgentWebSocketClientConfiguration configuration) { this.configuration = Objects.requireNonNull(configuration, "'configuration' cannot be null."); } @@ -29,7 +33,7 @@ public final class BetaVoiceAgentWebSocketAsyncClient { * @param agentName the voice agent name. * @return a connected session. */ - public Mono openWebSocketSession(String agentName) { + public Mono openWebSocketSession(String agentName) { return openWebSocketSession(agentName, new VoiceAgentWebSocketConnectionOptions()); } @@ -40,15 +44,20 @@ public Mono openWebSocketSession(Stri * @param options connection options. * @return a connected session. */ - public Mono openWebSocketSession(String agentName, + public Mono openWebSocketSession(String agentName, VoiceAgentWebSocketConnectionOptions options) { Objects.requireNonNull(agentName, "'agentName' cannot be null."); Objects.requireNonNull(options, "'options' cannot be null."); VoiceAgentWebSocketConnectionOptions optionsSnapshot = new VoiceAgentWebSocketConnectionOptions(options); return Mono.defer(() -> { - BetaVoiceAgentWebSocketSessionAsyncClient session - = new BetaVoiceAgentWebSocketSessionAsyncClient(configuration, agentName, optionsSnapshot); - return session.connect().thenReturn(session); + WebSocketSessionAsyncTransport transport = new WebSocketSessionAsyncTransport(configuration, + VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, agentName, optionsSnapshot), + new WebSocketTransportOptions(optionsSnapshot), HttpClient.create(), + VoiceAgentWebSocketUtils.SUBPROTOCOL, + token -> VoiceAgentWebSocketUtils.buildHeaders(configuration, optionsSnapshot, token)); + return transport.connect() + .thenReturn(new BetaVoiceAgentsWebSocketSessionAsyncClient(transport, + optionsSnapshot.getMalformedEventHandler())); }); } } diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketClient.java similarity index 59% rename from sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketClient.java rename to sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketClient.java index 232c956f84313..b895f9de9f48e 100644 --- a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketClient.java +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketClient.java @@ -4,6 +4,9 @@ package com.azure.ai.agents; import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketClientConfiguration; +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; +import com.azure.ai.agents.implementation.realtime.WebSocketSessionTransport; +import com.azure.ai.agents.implementation.realtime.WebSocketTransportOptions; import com.azure.ai.agents.implementation.utils.Beta; import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; import com.azure.core.annotation.ServiceClient; @@ -16,11 +19,11 @@ */ @ServiceClient(builder = AgentsClientBuilder.class) @Beta(warningText = "This class is in preview and may change in future releases.") -public final class BetaVoiceAgentWebSocketClient { - private static final ClientLogger LOGGER = new ClientLogger(BetaVoiceAgentWebSocketClient.class); +public final class BetaVoiceAgentsWebSocketClient { + private static final ClientLogger LOGGER = new ClientLogger(BetaVoiceAgentsWebSocketClient.class); private final VoiceAgentWebSocketClientConfiguration configuration; - BetaVoiceAgentWebSocketClient(VoiceAgentWebSocketClientConfiguration configuration) { + BetaVoiceAgentsWebSocketClient(VoiceAgentWebSocketClientConfiguration configuration) { this.configuration = Objects.requireNonNull(configuration, "'configuration' cannot be null."); } @@ -30,7 +33,7 @@ public final class BetaVoiceAgentWebSocketClient { * @param agentName the voice agent name. * @return a connected session. */ - public BetaVoiceAgentWebSocketSessionClient openWebSocketSession(String agentName) { + public BetaVoiceAgentsWebSocketSessionClient openWebSocketSession(String agentName) { return openWebSocketSession(agentName, new VoiceAgentWebSocketConnectionOptions()); } @@ -39,17 +42,22 @@ public BetaVoiceAgentWebSocketSessionClient openWebSocketSession(String agentNam * * @param agentName the voice agent name. * @param options connection options. - * @throws IllegalArgumentException if {@code agentName} is empty. * @return a connected session. + * @throws IllegalArgumentException if {@code agentName} is empty. */ - public BetaVoiceAgentWebSocketSessionClient openWebSocketSession(String agentName, + public BetaVoiceAgentsWebSocketSessionClient openWebSocketSession(String agentName, VoiceAgentWebSocketConnectionOptions options) { Objects.requireNonNull(agentName, "'agentName' cannot be null."); if (agentName.isEmpty()) { throw LOGGER.logExceptionAsError(new IllegalArgumentException("'agentName' cannot be empty.")); } Objects.requireNonNull(options, "'options' cannot be null."); - return BetaVoiceAgentWebSocketSessionClient.connect(configuration, agentName, - new VoiceAgentWebSocketConnectionOptions(options)); + VoiceAgentWebSocketConnectionOptions snapshot = new VoiceAgentWebSocketConnectionOptions(options); + return new BetaVoiceAgentsWebSocketSessionClient( + WebSocketSessionTransport.connect(configuration, + VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, agentName, snapshot), + new WebSocketTransportOptions(snapshot), VoiceAgentWebSocketUtils.SUBPROTOCOL, + token -> VoiceAgentWebSocketUtils.buildHeaders(configuration, snapshot, token)), + snapshot.getMalformedEventHandler()); } } diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketSessionAsyncClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketSessionAsyncClient.java new file mode 100644 index 0000000000000..b5c58bacf8bbd --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketSessionAsyncClient.java @@ -0,0 +1,271 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents; + +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; +import com.azure.ai.agents.implementation.realtime.WebSocketSessionAsyncTransport; +import com.azure.ai.agents.implementation.utils.Beta; +import com.azure.ai.agents.models.RealtimeClientEvent; +import com.azure.ai.agents.models.RealtimeConversationItem; +import com.azure.ai.agents.models.RealtimeConversationItemCreateEvent; +import com.azure.ai.agents.models.RealtimeConversationItemFunctionCallOutput; +import com.azure.ai.agents.models.RealtimeConversationItemUserMessage; +import com.azure.ai.agents.models.RealtimeInputAudioBufferAppendEvent; +import com.azure.ai.agents.models.RealtimeInputAudioBufferClearEvent; +import com.azure.ai.agents.models.RealtimeInputAudioBufferCommitEvent; +import com.azure.ai.agents.models.RealtimeResponseCancelEvent; +import com.azure.ai.agents.models.RealtimeResponseCreateEvent; +import com.azure.ai.agents.models.RealtimeServerEvent; +import com.azure.ai.agents.models.VoiceAgentResponseCreateOptions; +import com.azure.core.util.AsyncCloseable; +import com.azure.core.util.BinaryData; +import com.openai.models.realtime.RealtimeConversationItemUserMessage.Content; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import java.io.IOException; +import java.util.Base64; +import java.util.Collections; +import java.util.Objects; +import java.util.function.Consumer; + +/** + * An asynchronous bidirectional realtime session connected to a Foundry voice agent. + * + *

Instances are created by {@link BetaVoiceAgentsWebSocketAsyncClient#openWebSocketSession(String)}. A session supports one + * subscriber to {@link #receiveEvents()}. Close the session when it is no longer needed.

+ */ +@Beta(warningText = "This class is in preview and may change in future releases.") +public final class BetaVoiceAgentsWebSocketSessionAsyncClient implements AsyncCloseable, AutoCloseable { + private final WebSocketSessionAsyncTransport transport; + private final Consumer malformedEventHandler; + + BetaVoiceAgentsWebSocketSessionAsyncClient(WebSocketSessionAsyncTransport transport, + Consumer malformedEventHandler) { + this.transport = transport; + this.malformedEventHandler = malformedEventHandler; + } + + /** + * Gets the WebSocket endpoint used by this session. + * + * @return the WebSocket endpoint. + */ + public String getEndpoint() { + return transport.getEndpoint(); + } + + /** + * Determines whether the WebSocket session is open. + * + * @return {@code true} when the session is open. + */ + public boolean isOpen() { + return transport.isOpen(); + } + + /** + * Gets the peer close code after the session closes. + * + * @return the close code, or {@code null} when no close frame has been received. + */ + public Integer getCloseCode() { + return transport.getCloseCode(); + } + + /** + * Gets the peer close reason after the session closes. + * + * @return the close reason, or {@code null} when no close frame has been received. + */ + public String getCloseReason() { + return transport.getCloseReason(); + } + + /** + * Receives typed server events in wire order. Only one subscriber is supported per session. + * + * @return the server event stream. + */ + public Flux receiveEvents() { + return VoiceAgentWebSocketUtils.decodeEvents(transport.receiveEvents(), + VoiceAgentWebSocketUtils::deserializeEvent, malformedEventHandler, + () -> transport.closeAsync(1007, "Invalid JSON event").subscribe()); + } + + /** + * Sends a typed realtime client event. + * + * @param event the event to send. + * @return a completion signal emitted after the frame is written. + */ + public Mono sendEvent(RealtimeClientEvent event) { + Objects.requireNonNull(event, "'event' cannot be null."); + return Mono.defer(() -> { + try { + return transport.sendEvent(BinaryData.fromString(event.toJsonString())); + } catch (IOException error) { + return Mono + .error(new IllegalArgumentException("Failed to serialize the realtime client event.", error)); + } + }); + } + + /** + * Sends a JSON object, including event types and fields unknown to this SDK. + * + * @param event the complete JSON event. + * @return completion after the frame is written, or an error for invalid JSON or a closed session. + */ + public Mono sendEvent(BinaryData event) { + return transport.sendEvent(event); + } + + /** + * Adds an item to the session conversation. + * + * @param item the item to add. + * @return a completion signal emitted after the event is written. + */ + public Mono createConversationItem(RealtimeConversationItem item) { + return createConversationItem(item, null); + } + + /** + * Adds an item after a specific conversation item. + * + * @param item the item to add. + * @param previousItemId the preceding item identifier, or {@code null} to append. + * @return a completion signal emitted after the event is written. + */ + public Mono createConversationItem(RealtimeConversationItem item, String previousItemId) { + Objects.requireNonNull(item, "'item' cannot be null."); + return sendEvent(new RealtimeConversationItemCreateEvent(item).setPreviousItemId(previousItemId)); + } + + /** + * Adds a user text message to the session conversation. + * + * @param text the user message. + * @return a completion signal emitted after the event is written. + */ + public Mono sendText(String text) { + Objects.requireNonNull(text, "'text' cannot be null."); + Content content = Content.builder().type(Content.Type.INPUT_TEXT).text(text).build(); + return createConversationItem(new RealtimeConversationItemUserMessage(Collections.singletonList(content))); + } + + /** + * Appends PCM or encoded audio bytes to the input audio buffer. + * + * @param audio the bytes in the input format configured by the voice agent. + * @return a completion signal emitted after the event is written. + */ + public Mono appendInputAudio(BinaryData audio) { + Objects.requireNonNull(audio, "'audio' cannot be null."); + String encoded = Base64.getEncoder().encodeToString(audio.toBytes()); + return sendEvent(new RealtimeInputAudioBufferAppendEvent(encoded)); + } + + /** + * Clears the input audio buffer. + * + * @return a completion signal emitted after the event is written. + */ + public Mono clearInputAudio() { + return sendEvent(new RealtimeInputAudioBufferClearEvent()); + } + + /** + * Commits the input audio buffer. + * + * @return a completion signal emitted after the event is written. + */ + public Mono commitInputAudio() { + return sendEvent(new RealtimeInputAudioBufferCommitEvent()); + } + + /** + * Requests a response using the persisted voice-agent configuration. + * + * @return a completion signal emitted after the event is written. + */ + public Mono createResponse() { + return sendEvent(new RealtimeResponseCreateEvent()); + } + + /** + * Requests a response with per-response options. + * + * @param responseOptions the response options. + * @return a completion signal emitted after the event is written. + */ + public Mono createResponse(VoiceAgentResponseCreateOptions responseOptions) { + Objects.requireNonNull(responseOptions, "'responseOptions' cannot be null."); + return sendEvent(new RealtimeResponseCreateEvent().setResponse(responseOptions)); + } + + /** + * Cancels the response currently writing to the default conversation. + * + * @return a completion signal emitted after the event is written. + */ + public Mono cancelResponse() { + return sendEvent(new RealtimeResponseCancelEvent()); + } + + /** + * Cancels a specific response. + * + * @param responseId the response identifier. + * @return a completion signal emitted after the event is written. + */ + public Mono cancelResponse(String responseId) { + Objects.requireNonNull(responseId, "'responseId' cannot be null."); + return sendEvent(new RealtimeResponseCancelEvent().setResponseId(responseId)); + } + + /** + * Sends a function-call result and requests the next response. + * + * @param callId the function call identifier. + * @param output the serialized function result. + * @return a completion signal emitted after both events are written. + */ + public Mono sendFunctionCallOutput(String callId, String output) { + RealtimeConversationItemFunctionCallOutput item + = new RealtimeConversationItemFunctionCallOutput(callId, output); + return createConversationItem(item).then(createResponse()); + } + + /** + * Closes the WebSocket session. + * + * @return a completion signal for closing the session. + */ + @Override + public Mono closeAsync() { + return transport.closeAsync(); + } + + /** + * Closes the session with an application-selected WebSocket close frame. + * The first requested close frame wins when close is called more than once. + * + * @param code a valid WebSocket close code. + * @param reason non-null reason of at most 123 UTF-8 bytes. + * @return a completion signal, or an error if the code or reason is invalid. + */ + public Mono closeAsync(int code, String reason) { + return transport.closeAsync(code, reason); + } + + /** + * Closes the WebSocket session synchronously. + */ + @Override + public void close() { + transport.close(); + } +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketSessionClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketSessionClient.java new file mode 100644 index 0000000000000..422090b0aae86 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketSessionClient.java @@ -0,0 +1,254 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents; + +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; +import com.azure.ai.agents.implementation.realtime.WebSocketSessionTransport; +import com.azure.ai.agents.implementation.utils.Beta; +import com.azure.ai.agents.models.RealtimeClientEvent; +import com.azure.ai.agents.models.RealtimeConversationItem; +import com.azure.ai.agents.models.RealtimeConversationItemCreateEvent; +import com.azure.ai.agents.models.RealtimeConversationItemFunctionCallOutput; +import com.azure.ai.agents.models.RealtimeConversationItemUserMessage; +import com.azure.ai.agents.models.RealtimeInputAudioBufferAppendEvent; +import com.azure.ai.agents.models.RealtimeInputAudioBufferClearEvent; +import com.azure.ai.agents.models.RealtimeInputAudioBufferCommitEvent; +import com.azure.ai.agents.models.RealtimeResponseCancelEvent; +import com.azure.ai.agents.models.RealtimeResponseCreateEvent; +import com.azure.ai.agents.models.RealtimeServerEvent; +import com.azure.ai.agents.models.VoiceAgentResponseCreateOptions; +import com.azure.core.util.BinaryData; +import com.azure.core.util.IterableStream; +import com.azure.core.util.logging.ClientLogger; +import com.openai.models.realtime.RealtimeConversationItemUserMessage.Content; + +import java.io.IOException; +import java.time.Duration; +import java.util.Base64; +import java.util.Collections; +import java.util.Objects; +import java.util.function.Consumer; + +/** + * A synchronous bidirectional realtime session connected to a Foundry voice agent. + */ +@Beta(warningText = "This class is in preview and may change in future releases.") +public final class BetaVoiceAgentsWebSocketSessionClient implements AutoCloseable { + private static final ClientLogger LOGGER = new ClientLogger(BetaVoiceAgentsWebSocketSessionClient.class); + private final WebSocketSessionTransport transport; + private final Consumer malformedEventHandler; + + BetaVoiceAgentsWebSocketSessionClient(WebSocketSessionTransport transport, + Consumer malformedEventHandler) { + this.transport = transport; + this.malformedEventHandler = malformedEventHandler; + } + + /** + * Gets the WebSocket endpoint used by this session. + * + * @return the WebSocket endpoint. + */ + public String getEndpoint() { + return transport.getEndpoint(); + } + + /** + * Determines whether the session is open. + * + * @return {@code true} when the session is open. + */ + public boolean isOpen() { + return transport.isOpen(); + } + + /** + * Gets the peer close code. + * + * @return the close code, or {@code null}. + */ + public Integer getCloseCode() { + return transport.getCloseCode(); + } + + /** + * Gets the peer close reason. + * + * @return the close reason, or {@code null}. + */ + public String getCloseReason() { + return transport.getCloseReason(); + } + + /** + * Receives typed server events in wire order. The returned stream may be iterated once. + * + * @return the server event stream. + * @throws IllegalStateException if the event stream has already been claimed. + */ + public IterableStream receiveEvents() { + return receiveEvents(null); + } + + /** + * Receives events with a timeout for each wait. A timeout leaves the session open and the iterator can be retried. + * + * @param timeout positive per-event timeout, or null to wait indefinitely. + * @return the single-consumer event stream. + * @throws IllegalArgumentException if timeout is zero or negative. + * @throws IllegalStateException if another receiver exists or a wait times out (with a TimeoutException cause). + */ + public IterableStream receiveEvents(Duration timeout) { + return VoiceAgentWebSocketUtils.decodeEvents(transport.receiveEvents(timeout), + VoiceAgentWebSocketUtils::deserializeEvent, malformedEventHandler, + () -> transport.close(1007, "Invalid JSON event")); + } + + /** + * Sends a typed realtime client event. + * + * @param event the event to send. + * @throws IllegalArgumentException if the event cannot be serialized. + * @throws IllegalStateException if the session is closed or cannot accept another event. + */ + public void sendEvent(RealtimeClientEvent event) { + Objects.requireNonNull(event, "'event' cannot be null."); + try { + transport.sendEvent(BinaryData.fromString(event.toJsonString())); + } catch (IOException error) { + throw LOGGER.logExceptionAsError( + new IllegalArgumentException("Failed to serialize the realtime client event.", error)); + } + } + + /** + * Sends a JSON object, including event types and fields unknown to this SDK. + * + * @param event the complete JSON event. + * @throws IllegalArgumentException if the event is not a JSON object. + * @throws IllegalStateException if the session cannot accept the event. + */ + public void sendEvent(BinaryData event) { + transport.sendEvent(event); + } + + /** + * Adds a conversation item. + * + * @param item the item to add. + */ + public void createConversationItem(RealtimeConversationItem item) { + createConversationItem(item, null); + } + + /** + * Adds a conversation item after another item. + * + * @param item the item to add. + * @param previousItemId the preceding item identifier. + */ + public void createConversationItem(RealtimeConversationItem item, String previousItemId) { + Objects.requireNonNull(item, "'item' cannot be null."); + sendEvent(new RealtimeConversationItemCreateEvent(item).setPreviousItemId(previousItemId)); + } + + /** + * Adds a user text message. + * + * @param text the user message. + */ + public void sendText(String text) { + Objects.requireNonNull(text, "'text' cannot be null."); + Content content = Content.builder().type(Content.Type.INPUT_TEXT).text(text).build(); + createConversationItem(new RealtimeConversationItemUserMessage(Collections.singletonList(content))); + } + + /** + * Appends audio to the input buffer. + * + * @param audio the audio bytes. + */ + public void appendInputAudio(BinaryData audio) { + Objects.requireNonNull(audio, "'audio' cannot be null."); + sendEvent(new RealtimeInputAudioBufferAppendEvent(Base64.getEncoder().encodeToString(audio.toBytes()))); + } + + /** + * Clears the input audio buffer. + */ + public void clearInputAudio() { + sendEvent(new RealtimeInputAudioBufferClearEvent()); + } + + /** + * Commits the input audio buffer. + */ + public void commitInputAudio() { + sendEvent(new RealtimeInputAudioBufferCommitEvent()); + } + + /** + * Requests a response using the voice agent's configuration. + */ + public void createResponse() { + sendEvent(new RealtimeResponseCreateEvent()); + } + + /** + * Requests a response with per-response options. + * + * @param responseOptions response options. + */ + public void createResponse(VoiceAgentResponseCreateOptions responseOptions) { + Objects.requireNonNull(responseOptions, "'responseOptions' cannot be null."); + sendEvent(new RealtimeResponseCreateEvent().setResponse(responseOptions)); + } + + /** + * Cancels the active response. + */ + public void cancelResponse() { + sendEvent(new RealtimeResponseCancelEvent()); + } + + /** + * Cancels a specific response. + * + * @param responseId the response identifier. + */ + public void cancelResponse(String responseId) { + Objects.requireNonNull(responseId, "'responseId' cannot be null."); + sendEvent(new RealtimeResponseCancelEvent().setResponseId(responseId)); + } + + /** + * Sends a function-call result and requests the next response. + * + * @param callId the function call identifier. + * @param output the serialized function output. + */ + public void sendFunctionCallOutput(String callId, String output) { + createConversationItem(new RealtimeConversationItemFunctionCallOutput(callId, output)); + createResponse(); + } + + /** + * Closes the session. + */ + @Override + public void close() { + transport.close(); + } + + /** + * Closes the session with an application-selected WebSocket close frame. + * + * @param code a valid WebSocket close code. + * @param reason non-null reason of at most 123 UTF-8 bytes. + * @throws IllegalArgumentException if the code or reason is invalid. + */ + public void close(int code, String reason) { + transport.close(code, reason); + } +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/OpenAIJsonHelper.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/OpenAIJsonHelper.java index 6d7c41624bd0f..ed80f517b9b6a 100644 --- a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/OpenAIJsonHelper.java +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/OpenAIJsonHelper.java @@ -4,6 +4,8 @@ package com.azure.ai.agents.implementation; import com.azure.core.util.BinaryData; +import com.azure.core.util.logging.ClientLogger; +import com.azure.core.util.serializer.JsonSerializer; import com.azure.json.JsonProviders; import com.azure.json.JsonReader; import com.azure.json.JsonSerializable; @@ -16,16 +18,53 @@ import com.openai.models.responses.ResponseCreateParams; import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; import java.io.StringReader; +import java.io.UncheckedIOException; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.stream.Collectors; +import reactor.core.publisher.Mono; public final class OpenAIJsonHelper { + private static final ClientLogger LOGGER = new ClientLogger(OpenAIJsonHelper.class); private static final ObjectMapper OPENAI_MODEL_MAPPER = ObjectMappers.jsonMapper(); + private static final JsonSerializer OPENAI_MODEL_SERIALIZER = new JsonSerializer() { + @Override + public T deserialize(InputStream stream, com.azure.core.util.serializer.TypeReference typeReference) { + try { + return OPENAI_MODEL_MAPPER.readValue(stream, + OPENAI_MODEL_MAPPER.constructType(typeReference.getJavaType())); + } catch (IOException error) { + throw LOGGER.logExceptionAsError(new UncheckedIOException(error)); + } + } + + @Override + public Mono deserializeAsync(InputStream stream, + com.azure.core.util.serializer.TypeReference typeReference) { + return Mono.fromCallable(() -> deserialize(stream, typeReference)); + } + + @Override + public void serialize(OutputStream stream, Object value) { + try { + stream.write(OPENAI_MODEL_MAPPER.writeValueAsBytes(value)); + } catch (IOException error) { + throw LOGGER.logExceptionAsError(new UncheckedIOException(error)); + } + } + + @Override + public Mono serializeAsync(OutputStream stream, Object value) { + return Mono.fromRunnable(() -> serialize(stream, value)); + } + }; + private static final ObjectMapper MAPPER = ObjectMappers.jsonMapper() .rebuild() .configure(MapperFeature.AUTO_DETECT_FIELDS, true) @@ -114,10 +153,7 @@ public static BinaryData toBinaryData(Object openAIObject) { } try { String json = OPENAI_MODEL_MAPPER.writeValueAsString(openAIObject); - try (JsonReader reader = JsonProviders.createReader(new StringReader(json))) { - reader.nextToken(); - return BinaryData.fromObject(reader.readUntyped()); - } + return BinaryData.fromObject(OPENAI_MODEL_MAPPER.readTree(json), OPENAI_MODEL_SERIALIZER); } catch (IOException e) { throw new RuntimeException("Failed to convert OpenAI type to BinaryData", e); } diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketClientConfiguration.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketClientConfiguration.java index d123c4d966303..3b154026caa09 100644 --- a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketClientConfiguration.java +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketClientConfiguration.java @@ -80,7 +80,7 @@ public String getUserAgent() { /** * Gets safe additional headers. * - * @return safe additional headers, or {@code null}. + * @return safe additional headers, or {@code null}. */ public HttpHeaders getHeaders() { return headers; diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketUtils.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketUtils.java index 1ee135629245e..fe292a2882e1e 100644 --- a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketUtils.java +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketUtils.java @@ -6,6 +6,7 @@ import com.azure.ai.agents.implementation.models.AgentDefinitionOptInKeys; import com.azure.ai.agents.models.RawRealtimeServerEvent; import com.azure.ai.agents.models.RealtimeServerEvent; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; import com.azure.ai.agents.models.VoiceAgentTransport; import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; import com.azure.core.credential.TokenRequestContext; @@ -13,10 +14,14 @@ import com.azure.core.http.HttpHeaderName; import com.azure.core.http.HttpHeaders; import com.azure.core.util.BinaryData; +import com.azure.core.util.IterableStream; import com.azure.core.util.UrlBuilder; +import com.azure.core.util.logging.ClientLogger; import com.azure.json.JsonProviders; import com.azure.json.JsonReader; import com.azure.json.JsonToken; +import reactor.core.publisher.Flux; + import java.io.IOException; import java.io.UnsupportedEncodingException; import java.net.URI; @@ -25,10 +30,77 @@ import java.nio.charset.CharacterCodingException; import java.nio.charset.CodingErrorAction; import java.nio.charset.StandardCharsets; +import java.util.Iterator; import java.util.Locale; +import java.util.NoSuchElementException; import java.util.Objects; +import java.util.function.Consumer; public final class VoiceAgentWebSocketUtils { + private static final ClientLogger LOGGER = new ClientLogger(VoiceAgentWebSocketUtils.class); + + public static IterableStream decodeEvents(IterableStream messages, + WebSocketEventDecoder decoder, Consumer malformedEventHandler, Runnable closeOnError) { + return IterableStream.of(() -> new Iterator() { + private final Iterator source = messages.iterator(); + private T next; + private boolean terminated; + + @Override + public boolean hasNext() { + while (next == null && !terminated && source.hasNext()) { + BinaryData message = source.next(); + try { + next = decodeEvent(message, decoder, malformedEventHandler); + } catch (RuntimeException error) { + terminated = true; + closeOnError.run(); + throw LOGGER.logExceptionAsError(error); + } + } + return next != null; + } + + @Override + public T next() { + if (!hasNext()) { + throw LOGGER.logExceptionAsError(new NoSuchElementException()); + } + T event = next; + next = null; + return event; + } + }); + } + + public static Flux decodeEvents(Flux messages, WebSocketEventDecoder decoder, + Consumer malformedEventHandler, Runnable closeOnError) { + return messages.handle((message, sink) -> { + try { + T event = decodeEvent(message, decoder, malformedEventHandler); + if (event != null) { + sink.next(event); + } + } catch (RuntimeException error) { + closeOnError.run(); + sink.error(error); + } + }); + } + + private static T decodeEvent(BinaryData message, WebSocketEventDecoder decoder, + Consumer malformedEventHandler) { + try { + return decoder.decode(message.toString()); + } catch (IOException | RuntimeException error) { + if (malformedEventHandler != null) { + malformedEventHandler.accept(error); + return null; + } + throw LOGGER.logExceptionAsError(new IllegalArgumentException("Invalid JSON event.", error)); + } + } + public static String decodeEvent(byte[] bytes) throws CharacterCodingException { return StandardCharsets.UTF_8.newDecoder() .onMalformedInput(CodingErrorAction.REPORT) @@ -71,6 +143,12 @@ public static RealtimeServerEvent deserializeEvent(String json) throws IOExcepti private VoiceAgentWebSocketUtils() { } + public static BinaryData deserializeRawEvent(String json) { + BinaryData event = BinaryData.fromString(json); + validateEvent(event); + return event; + } + public static void validateClose(int code, String reason) { if (code < 1000 || code >= 5000 @@ -98,11 +176,37 @@ private static boolean isProtectedHeader(String name) { public static URI buildWebSocketUri(VoiceAgentWebSocketClientConfiguration configuration, String agentName, VoiceAgentWebSocketConnectionOptions options) { + UrlBuilder url = buildSessionUrl(configuration, agentName, "", options.isStoreEnabled(), + options.getAgentVersionOverride()); + VoiceAgentTransport transport = options.getTransportType(); + url.setQueryParameter("transport", transport == null ? "websocket" : encode(transport.toString())); + if (options.getAgentSessionId() != null) { + url.setQueryParameter("agent_session_id", encode(options.getAgentSessionId())); + } + return URI.create(url.toString()); + } + + public static URI buildLiveWebSocketUri(VoiceAgentWebSocketClientConfiguration configuration, String agentName, + VoiceAgentLiveWebSocketConnectionOptions options) { + UrlBuilder url = buildSessionUrl(configuration, agentName, "/live/sessions", options.isStoreEnabled(), + options.getAgentVersionOverride()); + if (options.getStructuredInputs() != null) { + url.setQueryParameter("structured_inputs", encode(options.getStructuredInputs())); + } + return URI.create(url.toString()); + } + + private static UrlBuilder buildSessionUrl(VoiceAgentWebSocketClientConfiguration configuration, String agentName, + String suffix, Boolean store, String agentVersionOverride) { + Objects.requireNonNull(agentName, "'agentName' cannot be null."); + if (agentName.isEmpty()) { + throw new IllegalArgumentException("'agentName' cannot be empty."); + } + if (!suffix.isEmpty() && !suffix.startsWith("/")) { + throw new IllegalArgumentException("A voice-agent route suffix must start with '/'."); + } URI endpoint = configuration.getEndpoint(); - String scheme; - if ("https".equalsIgnoreCase(endpoint.getScheme()) || "wss".equalsIgnoreCase(endpoint.getScheme())) { - scheme = "wss"; - } else { + if (!"https".equalsIgnoreCase(endpoint.getScheme()) && !"wss".equalsIgnoreCase(endpoint.getScheme())) { throw new IllegalArgumentException( "Voice-agent WebSocket endpoints must use https or wss to protect credentials."); } @@ -112,23 +216,20 @@ public static URI buildWebSocketUri(VoiceAgentWebSocketClientConfiguration confi } String basePath = endpoint.getRawPath() == null ? "" : endpoint.getRawPath().replaceAll("/$", ""); - String path = basePath + "/agents/" + encode(agentName) + "/endpoint/protocols/voice"; - URI baseUri = URI.create(scheme + "://" + endpoint.getRawAuthority() + path); - UrlBuilder url = UrlBuilder.parse(baseUri.toString()); + String path = basePath + "/agents/" + encode(agentName) + "/endpoint/protocols/voice" + suffix; + UrlBuilder url = UrlBuilder.parse(endpoint.toString()).setScheme("wss").setPath(path); + // Initialize the lazy query before clearing it; the pinned Azure Core version otherwise retains it. + url.getQuery(); + url.clearQuery(); url.setQueryParameter("api-version", encode(configuration.getApiVersion())); url.setQueryParameter("x-ms-client-sdk", encode(configuration.getUserAgent())); - VoiceAgentTransport transport = options.getTransportType(); - url.setQueryParameter("transport", transport == null ? "websocket" : encode(transport.toString())); - if (options.isStoreEnabled() != null) { - url.setQueryParameter("store", options.isStoreEnabled().toString()); + if (store != null) { + url.setQueryParameter("store", store.toString()); } - if (options.getAgentVersionOverride() != null) { - url.setQueryParameter("x-agent-version-override", encode(options.getAgentVersionOverride())); + if (agentVersionOverride != null) { + url.setQueryParameter("x-agent-version-override", encode(agentVersionOverride)); } - if (options.getAgentSessionId() != null) { - url.setQueryParameter("agent_session_id", encode(options.getAgentSessionId())); - } - return URI.create(url.toString()); + return url; } public static TokenRequestContext createTokenRequestContext() { @@ -137,6 +238,14 @@ public static TokenRequestContext createTokenRequestContext() { public static HttpHeaders buildHeaders(VoiceAgentWebSocketClientConfiguration configuration, VoiceAgentWebSocketConnectionOptions options, String token) { + HttpHeaders headers = buildHeaders(configuration, token); + if (options.getStructuredInputs() != null) { + headers.set(HttpHeaderName.fromString("x-ms-voice-structured-inputs"), options.getStructuredInputs()); + } + return headers; + } + + public static HttpHeaders buildHeaders(VoiceAgentWebSocketClientConfiguration configuration, String token) { HttpHeaders headers = new HttpHeaders().set(HttpHeaderName.USER_AGENT, configuration.getUserAgent()); if (configuration.getHeaders() != null) { for (HttpHeader header : configuration.getHeaders()) { @@ -147,9 +256,6 @@ public static HttpHeaders buildHeaders(VoiceAgentWebSocketClientConfiguration co } headers.set(HttpHeaderName.fromString("Foundry-Features"), AgentDefinitionOptInKeys.VOICE_AGENTS_V1_PREVIEW.toString()); - if (options.getStructuredInputs() != null) { - headers.set(HttpHeaderName.fromString("x-ms-voice-structured-inputs"), options.getStructuredInputs()); - } return headers.set(HttpHeaderName.AUTHORIZATION, "Bearer " + token); } diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketEventDecoder.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketEventDecoder.java new file mode 100644 index 0000000000000..487f459f81e66 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketEventDecoder.java @@ -0,0 +1,15 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.implementation.realtime; + +import java.io.IOException; + +/** + * Decodes a raw message in a protocol session client. + * @param the decoded event type. + */ +@FunctionalInterface +public interface WebSocketEventDecoder { + T decode(String json) throws IOException; +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionAsyncClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketSessionAsyncTransport.java similarity index 51% rename from sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionAsyncClient.java rename to sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketSessionAsyncTransport.java index 06cba1221d988..2e867c929c615 100644 --- a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionAsyncClient.java +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketSessionAsyncTransport.java @@ -1,37 +1,17 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -package com.azure.ai.agents; - -import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketClientConfiguration; -import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketHandshakeHandler; -import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketHttpResponse; -import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; -import com.azure.ai.agents.implementation.utils.Beta; -import com.azure.ai.agents.models.RealtimeClientEvent; -import com.azure.ai.agents.models.RealtimeConversationItemCreateEvent; -import com.azure.ai.agents.models.RealtimeInputAudioBufferAppendEvent; -import com.azure.ai.agents.models.RealtimeInputAudioBufferClearEvent; -import com.azure.ai.agents.models.RealtimeInputAudioBufferCommitEvent; -import com.azure.ai.agents.models.RealtimeResponseCancelEvent; -import com.azure.ai.agents.models.RealtimeResponseCreateEvent; -import com.azure.ai.agents.models.RealtimeConversationItem; -import com.azure.ai.agents.models.RealtimeConversationItemFunctionCallOutput; -import com.azure.ai.agents.models.RealtimeConversationItemUserMessage; -import com.azure.ai.agents.models.RealtimeServerEvent; -import com.azure.ai.agents.models.VoiceAgentResponseCreateOptions; -import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; +package com.azure.ai.agents.implementation.realtime; + import com.azure.core.credential.AccessToken; -import com.azure.core.credential.TokenRequestContext; import com.azure.core.exception.ClientAuthenticationException; import com.azure.core.exception.HttpResponseException; import com.azure.core.exception.ResourceModifiedException; import com.azure.core.exception.ResourceNotFoundException; import com.azure.core.http.HttpHeader; +import com.azure.core.http.HttpHeaders; import com.azure.core.http.ProxyOptions; -import com.azure.core.util.AsyncCloseable; import com.azure.core.util.BinaryData; -import com.openai.models.realtime.RealtimeConversationItemUserMessage.Content; import com.azure.core.util.logging.ClientLogger; import io.netty.channel.Channel; import io.netty.channel.ChannelOption; @@ -39,18 +19,6 @@ import io.netty.handler.codec.http.websocketx.TextWebSocketFrame; import io.netty.handler.codec.http.websocketx.WebSocketClientHandshakeException; import io.netty.handler.codec.http.websocketx.WebSocketFrame; -import java.io.IOException; -import java.net.URI; -import java.time.Duration; -import java.util.Base64; -import java.util.Collections; -import java.util.Objects; -import java.util.Queue; -import java.util.concurrent.ArrayBlockingQueue; -import java.util.concurrent.CancellationException; -import java.util.concurrent.Semaphore; -import java.util.concurrent.atomic.AtomicBoolean; -import java.util.concurrent.atomic.AtomicReference; import reactor.core.Disposable; import reactor.core.publisher.Flux; import reactor.core.publisher.Mono; @@ -62,161 +30,100 @@ import reactor.netty.http.websocket.WebsocketOutbound; import reactor.netty.transport.ProxyProvider; +import java.io.IOException; +import java.net.URI; +import java.time.Duration; +import java.util.Objects; +import java.util.Queue; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.CancellationException; +import java.util.concurrent.Semaphore; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Function; + /** - * An asynchronous bidirectional realtime session connected to a Foundry voice agent. - * - *

Instances are created by {@link BetaVoiceAgentWebSocketAsyncClient#openWebSocketSession(String)}. A session supports one - * subscriber to {@link #receiveEvents()}. Close the session when it is no longer needed.

+ * Shared asynchronous WebSocket connection and raw message I/O. */ -@Beta(warningText = "This class is in preview and may change in future releases.") -public final class BetaVoiceAgentWebSocketSessionAsyncClient implements AsyncCloseable, AutoCloseable { - private static final ClientLogger LOGGER = new ClientLogger(BetaVoiceAgentWebSocketSessionAsyncClient.class); +public final class WebSocketSessionAsyncTransport implements AutoCloseable { + private static final ClientLogger LOGGER = new ClientLogger(WebSocketSessionAsyncTransport.class); private static final int MAX_OUTSTANDING_SENDS = 256; - private final VoiceAgentWebSocketClientConfiguration configuration; - private final VoiceAgentWebSocketConnectionOptions options; + private final WebSocketTransportOptions options; private final HttpClient httpClient; private final URI websocketUri; - private final AtomicReference state = new AtomicReference<>(State.NEW); + private final String subprotocol; + private final Function headersFactory; + private final AtomicReference state = new AtomicReference<>(WebSocketSessionState.NEW); private final AtomicReference channel = new AtomicReference<>(); private final AtomicReference outbound = new AtomicReference<>(); private final AtomicReference connectionOperation = new AtomicReference<>(); private final AtomicReference lifecycle = new AtomicReference<>(); private final AtomicBoolean receiveClaimed = new AtomicBoolean(); private final Semaphore sendPermits = new Semaphore(MAX_OUTSTANDING_SENDS); - private final Queue eventQueue; - private final Sinks.Many events; + private final Queue eventQueue; + private final Sinks.Many events; private final Sinks.One ready = Sinks.one(); private final Sinks.One closeSignal = Sinks.one(); private final AtomicReference> closeOperation = new AtomicReference<>(); - private volatile Integer closeCode; private volatile String closeReason; - BetaVoiceAgentWebSocketSessionAsyncClient(VoiceAgentWebSocketClientConfiguration configuration, String agentName, - VoiceAgentWebSocketConnectionOptions options) { - this(configuration, agentName, options, HttpClient.create()); - } - - BetaVoiceAgentWebSocketSessionAsyncClient(VoiceAgentWebSocketClientConfiguration configuration, String agentName, - VoiceAgentWebSocketConnectionOptions options, HttpClient httpClient) { + public WebSocketSessionAsyncTransport(VoiceAgentWebSocketClientConfiguration configuration, URI websocketUri, + WebSocketTransportOptions options, HttpClient httpClient, String subprotocol, + Function headersFactory) { this.configuration = Objects.requireNonNull(configuration, "'configuration' cannot be null."); - Objects.requireNonNull(agentName, "'agentName' cannot be null."); - if (agentName.isEmpty()) { - throw new IllegalArgumentException("'agentName' cannot be empty."); - } - this.options = options == null ? new VoiceAgentWebSocketConnectionOptions() : options; + this.options = Objects.requireNonNull(options, "'options' cannot be null."); this.eventQueue = new ArrayBlockingQueue<>(this.options.getReceiveBufferCapacity()); this.events = Sinks.many().unicast().onBackpressureBuffer(eventQueue); this.httpClient = Objects.requireNonNull(httpClient, "'httpClient' cannot be null."); - this.websocketUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, agentName, this.options); + this.websocketUri = Objects.requireNonNull(websocketUri, "'websocketUri' cannot be null."); + this.subprotocol = subprotocol; + this.headersFactory = Objects.requireNonNull(headersFactory, "'headersFactory' cannot be null."); } - Mono connect() { + public Mono connect() { return Mono.defer(() -> { - if (!state.compareAndSet(State.NEW, State.CONNECTING)) { + if (!state.compareAndSet(WebSocketSessionState.NEW, WebSocketSessionState.CONNECTING)) { return Mono.error(new IllegalStateException("The voice-agent session has already been started.")); } - - TokenRequestContext tokenContext = VoiceAgentWebSocketUtils.createTokenRequestContext(); - return configuration.getCredential().getToken(tokenContext).map(AccessToken::getToken).flatMap(token -> { - Disposable connection = openWebSocket(token).subscribe(unused -> { - }, this::terminateWithError, this::terminateNormally); - connectionOperation.set(connection); - return ready.asMono().timeout(options.getHandshakeTimeout()); - }) + return configuration.getCredential() + .getToken(VoiceAgentWebSocketUtils.createTokenRequestContext()) + .map(AccessToken::getToken) + .flatMap(token -> { + Disposable connection = openWebSocket(token).subscribe(unused -> { + }, this::terminateWithError, this::terminateNormally); + connectionOperation.set(connection); + return ready.asMono().timeout(options.getHandshakeTimeout()); + }) .doOnError(this::terminateWithError) .doOnCancel(() -> terminateWithError(new CancellationException("WebSocket connection cancelled."))); }); } - /** - * Gets the WebSocket endpoint used by this session. - * - * @return the WebSocket endpoint. - */ public String getEndpoint() { return websocketUri.toString(); } - /** - * Determines whether the WebSocket session is open. - * - * @return {@code true} when the session is open. - */ public boolean isOpen() { Channel current = channel.get(); - return state.get() == State.OPEN && current != null && current.isActive(); + return state.get() == WebSocketSessionState.OPEN && current != null && current.isActive(); } - /** - * Gets the peer close code after the session closes. - * - * @return the close code, or {@code null} when no close frame has been received. - */ public Integer getCloseCode() { return closeCode; } - /** - * Gets the peer close reason after the session closes. - * - * @return the close reason, or {@code null} when no close frame has been received. - */ public String getCloseReason() { return closeReason; } - /** - * Receives typed server events in wire order. Only one subscriber is supported per session. - * - * @return the server event stream. - */ - public Flux receiveEvents() { + public Flux receiveEvents() { return Flux.defer(() -> receiveClaimed.compareAndSet(false, true) ? events.asFlux() : Flux.error(new IllegalStateException("Only one receiveEvents subscriber is supported per session."))); } - /** - * Sends a typed realtime client event. - * - * @param event the event to send. - * @return a completion signal emitted after the frame is written. - */ - public Mono sendEvent(RealtimeClientEvent event) { - Objects.requireNonNull(event, "'event' cannot be null."); - return Mono.defer(() -> { - Channel current = requireOpenChannel(); - if (!sendPermits.tryAcquire()) { - return Mono.error(new IllegalStateException("Too many voice-agent WebSocket sends are outstanding.")); - } - - final String json; - try { - json = serialize(event); - } catch (IOException error) { - sendPermits.release(); - return Mono - .error(new IllegalArgumentException("Failed to serialize the realtime client event.", error)); - } - - return Mono.create(sink -> current.writeAndFlush(new TextWebSocketFrame(json)).addListener(result -> { - sendPermits.release(); - if (result.isSuccess()) { - sink.success(); - } else { - sink.error(result.cause()); - } - })); - }); - } - - /** - * Sends a JSON object, including event types and fields unknown to this SDK. - * @param event the complete JSON event. - * @return completion after the frame is written, or an error for invalid JSON or a closed session. - */ public Mono sendEvent(BinaryData event) { Objects.requireNonNull(event, "'event' cannot be null."); return Mono.defer(() -> { @@ -236,140 +143,10 @@ public Mono sendEvent(BinaryData event) { }); } - /** - * Adds an item to the session conversation. - * - * @param item the item to add. - * @return a completion signal emitted after the event is written. - */ - public Mono createConversationItem(RealtimeConversationItem item) { - return createConversationItem(item, null); - } - - /** - * Adds an item after a specific conversation item. - * - * @param item the item to add. - * @param previousItemId the preceding item identifier, or {@code null} to append. - * @return a completion signal emitted after the event is written. - */ - public Mono createConversationItem(RealtimeConversationItem item, String previousItemId) { - Objects.requireNonNull(item, "'item' cannot be null."); - return sendEvent(new RealtimeConversationItemCreateEvent(item).setPreviousItemId(previousItemId)); - } - - /** - * Adds a user text message to the session conversation. - * - * @param text the user message. - * @return a completion signal emitted after the event is written. - */ - public Mono sendText(String text) { - Objects.requireNonNull(text, "'text' cannot be null."); - Content content = Content.builder().type(Content.Type.INPUT_TEXT).text(text).build(); - return createConversationItem(new RealtimeConversationItemUserMessage(Collections.singletonList(content))); - } - - /** - * Appends PCM or encoded audio bytes to the input audio buffer. - * - * @param audio the bytes in the input format configured by the voice agent. - * @return a completion signal emitted after the event is written. - */ - public Mono appendInputAudio(BinaryData audio) { - Objects.requireNonNull(audio, "'audio' cannot be null."); - String encoded = Base64.getEncoder().encodeToString(audio.toBytes()); - return sendEvent(new RealtimeInputAudioBufferAppendEvent(encoded)); - } - - /** - * Clears the input audio buffer. - * - * @return a completion signal emitted after the event is written. - */ - public Mono clearInputAudio() { - return sendEvent(new RealtimeInputAudioBufferClearEvent()); - } - - /** - * Commits the input audio buffer. - * - * @return a completion signal emitted after the event is written. - */ - public Mono commitInputAudio() { - return sendEvent(new RealtimeInputAudioBufferCommitEvent()); - } - - /** - * Requests a response using the persisted voice-agent configuration. - * - * @return a completion signal emitted after the event is written. - */ - public Mono createResponse() { - return sendEvent(new RealtimeResponseCreateEvent()); - } - - /** - * Requests a response with per-response options. - * - * @param responseOptions the response options. - * @return a completion signal emitted after the event is written. - */ - public Mono createResponse(VoiceAgentResponseCreateOptions responseOptions) { - Objects.requireNonNull(responseOptions, "'responseOptions' cannot be null."); - return sendEvent(new RealtimeResponseCreateEvent().setResponse(responseOptions)); - } - - /** - * Cancels the response currently writing to the default conversation. - * - * @return a completion signal emitted after the event is written. - */ - public Mono cancelResponse() { - return sendEvent(new RealtimeResponseCancelEvent()); - } - - /** - * Cancels a specific response. - * - * @param responseId the response identifier. - * @return a completion signal emitted after the event is written. - */ - public Mono cancelResponse(String responseId) { - Objects.requireNonNull(responseId, "'responseId' cannot be null."); - return sendEvent(new RealtimeResponseCancelEvent().setResponseId(responseId)); - } - - /** - * Sends a function-call result and requests the next response. - * - * @param callId the function call identifier. - * @param output the serialized function result. - * @return a completion signal emitted after both events are written. - */ - public Mono sendFunctionCallOutput(String callId, String output) { - RealtimeConversationItemFunctionCallOutput item - = new RealtimeConversationItemFunctionCallOutput(callId, output); - return createConversationItem(item).then(createResponse()); - } - - /** - * Closes the WebSocket session. - * - * @return a completion signal for closing the session. - */ - @Override public Mono closeAsync() { return closeAsync(1000, ""); } - /** - * Closes the session with an application-selected WebSocket close frame. - * The first requested close frame wins when close is called more than once. - * @param code a valid WebSocket close code. - * @param reason non-null reason of at most 123 UTF-8 bytes. - * @return a completion signal, or an error if the code or reason is invalid. - */ public Mono closeAsync(int code, String reason) { try { VoiceAgentWebSocketUtils.validateClose(code, reason); @@ -380,15 +157,14 @@ public Mono closeAsync(int code, String reason) { if (existing != null) { return existing; } - Mono created = Mono.defer(() -> { - State current = state.get(); - if (current == State.CLOSED || current == State.NEW) { - state.set(State.CLOSED); + WebSocketSessionState current = state.get(); + if (current == WebSocketSessionState.CLOSED || current == WebSocketSessionState.NEW) { + state.set(WebSocketSessionState.CLOSED); events.tryEmitComplete(); return Mono.empty(); } - state.set(State.CLOSING); + state.set(WebSocketSessionState.CLOSING); WebsocketOutbound currentOutbound = outbound.get(); Channel currentChannel = channel.get(); Mono graceful = currentOutbound == null ? Mono.empty() : currentOutbound.sendClose(code, reason); @@ -400,16 +176,9 @@ public Mono closeAsync(int code, String reason) { return Mono.empty(); }).doFinally(signal -> terminateNormally()); }).cache(); - - if (closeOperation.compareAndSet(null, created)) { - return created; - } - return closeOperation.get(); + return closeOperation.compareAndSet(null, created) ? created : closeOperation.get(); } - /** - * Closes the WebSocket session synchronously. - */ @Override public void close() { closeAsync().block(options.getCloseTimeout().plusSeconds(1)); @@ -421,16 +190,16 @@ private Mono openWebSocket(String token) { .doOnConnected(connection -> connection.addHandlerLast("voiceAgentHandshakeResponseObserver", new VoiceAgentWebSocketHandshakeHandler(this::terminateWithError))) .headers(headers -> { - for (HttpHeader header : VoiceAgentWebSocketUtils.buildHeaders(configuration, options, token)) { + for (HttpHeader header : headersFactory.apply(token)) { headers.set(header.getName(), header.getValue()); } }); - WebsocketClientSpec spec = WebsocketClientSpec.builder() - .protocols(VoiceAgentWebSocketUtils.SUBPROTOCOL) - .maxFramePayloadLength(options.getMaxMessageSize()) - .build(); - - return client.websocket(spec).uri(websocketUri.toString()).connect().flatMap(connection -> { + WebsocketClientSpec.Builder specBuilder + = WebsocketClientSpec.builder().maxFramePayloadLength(options.getMaxMessageSize()); + if (subprotocol != null) { + specBuilder.protocols(subprotocol); + } + return client.websocket(specBuilder.build()).uri(websocketUri.toString()).connect().flatMap(connection -> { if (!(connection instanceof WebsocketInbound) || !(connection instanceof WebsocketOutbound)) { return Mono.error(new IllegalStateException("The WebSocket transport returned an invalid connection.")); } @@ -448,14 +217,12 @@ private static int toConnectTimeoutMillis(Duration timeout) { private Mono handleConnection(WebsocketInbound inbound, WebsocketOutbound outbound) { this.outbound.set(outbound); inbound.withConnection(connection -> channel.set(connection.channel())); - state.set(State.OPEN); + state.set(WebSocketSessionState.OPEN); ready.tryEmitEmpty(); - inbound.receiveCloseStatus().subscribe(status -> { closeCode = status.code(); closeReason = status.reasonText(); }, error -> LOGGER.atVerbose().addKeyValue("error", error.getMessage()).log("Close status unavailable.")); - Disposable receive = inbound.aggregateFrames(options.getMaxMessageSize()) .receiveFrames() .subscribe(this::handleFrame, this::terminateWithError, this::terminateNormally); @@ -468,8 +235,8 @@ private void handleFrame(WebSocketFrame frame) { try { byte[] bytes = new byte[frame.content().readableBytes()]; frame.content().getBytes(frame.content().readerIndex(), bytes); - RealtimeServerEvent event - = VoiceAgentWebSocketUtils.deserializeEvent(VoiceAgentWebSocketUtils.decodeEvent(bytes)); + String json = VoiceAgentWebSocketUtils.decodeEvent(bytes); + BinaryData event = BinaryData.fromString(json); Sinks.EmitResult result = events.tryEmitNext(event); if (result == Sinks.EmitResult.FAIL_OVERFLOW || result == Sinks.EmitResult.FAIL_ZERO_SUBSCRIBER) { switch (options.getOverflowStrategy()) { @@ -498,7 +265,7 @@ private void handleFrame(WebSocketFrame frame) { failure = callbackError; } } - closeWithProtocolError(1007, "Invalid JSON event", failure); + closeWithProtocolError(1007, "Invalid UTF-8 message", failure); } } } @@ -515,8 +282,8 @@ private void closeWithProtocolError(int code, String reason, Throwable error) { private void terminateWithError(Throwable error) { Throwable mappedError = mapHandshakeError(error); - State previous = state.getAndSet(State.CLOSED); - if (previous == State.CLOSED) { + WebSocketSessionState previous = state.getAndSet(WebSocketSessionState.CLOSED); + if (previous == WebSocketSessionState.CLOSED) { return; } ready.tryEmitError(mappedError); @@ -556,11 +323,11 @@ private Throwable mapHandshakeError(Throwable error) { } private void terminateNormally() { - State previous = state.getAndSet(State.CLOSED); - if (previous == State.CLOSED) { + WebSocketSessionState previous = state.getAndSet(WebSocketSessionState.CLOSED); + if (previous == WebSocketSessionState.CLOSED) { return; } - if (previous == State.CONNECTING) { + if (previous == WebSocketSessionState.CONNECTING) { ready.tryEmitError(new IllegalStateException("The WebSocket closed before the handshake completed.")); } events.tryEmitComplete(); @@ -581,7 +348,7 @@ private void disposeReceive() { private Channel requireOpenChannel() { Channel current = channel.get(); - if (state.get() != State.OPEN || current == null || !current.isActive()) { + if (state.get() != WebSocketSessionState.OPEN || current == null || !current.isActive()) { throw LOGGER .logExceptionAsError(new IllegalStateException("The voice-agent WebSocket session is not open.")); } @@ -618,11 +385,4 @@ private HttpClient configureProxy(HttpClient client) { }); } - private static String serialize(RealtimeClientEvent event) throws IOException { - return event.toJsonString(); - } - - private enum State { - NEW, CONNECTING, OPEN, CLOSING, CLOSED - } } diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketSessionState.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketSessionState.java new file mode 100644 index 0000000000000..9a88118abc8cf --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketSessionState.java @@ -0,0 +1,11 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.implementation.realtime; + +/** + * Connection lifecycle shared by the blocking and asynchronous WebSocket transports. + */ +public enum WebSocketSessionState { + NEW, CONNECTING, OPEN, CLOSING, CLOSED +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionClient.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketSessionTransport.java similarity index 60% rename from sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionClient.java rename to sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketSessionTransport.java index 7227280cd8326..1992a3fcc74f7 100644 --- a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionClient.java +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketSessionTransport.java @@ -1,35 +1,26 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -package com.azure.ai.agents; - -import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketClientConfiguration; -import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketHttpResponse; -import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; -import com.azure.ai.agents.implementation.utils.Beta; -import com.azure.ai.agents.models.RealtimeClientEvent; -import com.azure.ai.agents.models.RealtimeConversationItemCreateEvent; -import com.azure.ai.agents.models.RealtimeInputAudioBufferAppendEvent; -import com.azure.ai.agents.models.RealtimeInputAudioBufferClearEvent; -import com.azure.ai.agents.models.RealtimeInputAudioBufferCommitEvent; -import com.azure.ai.agents.models.RealtimeResponseCancelEvent; -import com.azure.ai.agents.models.RealtimeResponseCreateEvent; -import com.azure.ai.agents.models.RealtimeConversationItem; -import com.azure.ai.agents.models.RealtimeConversationItemFunctionCallOutput; -import com.azure.ai.agents.models.RealtimeConversationItemUserMessage; -import com.azure.ai.agents.models.RealtimeServerEvent; -import com.azure.ai.agents.models.VoiceAgentResponseCreateOptions; -import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; +package com.azure.ai.agents.implementation.realtime; + import com.azure.core.exception.ClientAuthenticationException; import com.azure.core.exception.HttpResponseException; import com.azure.core.exception.ResourceModifiedException; import com.azure.core.exception.ResourceNotFoundException; import com.azure.core.http.HttpHeader; +import com.azure.core.http.HttpHeaders; import com.azure.core.http.ProxyOptions; import com.azure.core.util.BinaryData; -import com.openai.models.realtime.RealtimeConversationItemUserMessage.Content; import com.azure.core.util.IterableStream; import com.azure.core.util.logging.ClientLogger; +import okhttp3.Credentials; +import okhttp3.OkHttpClient; +import okhttp3.Request; +import okhttp3.Response; +import okhttp3.WebSocket; +import okhttp3.WebSocketListener; +import okio.ByteString; + import java.io.IOException; import java.net.Proxy; import java.net.ProxySelector; @@ -38,7 +29,6 @@ import java.nio.charset.CharacterCodingException; import java.nio.charset.StandardCharsets; import java.time.Duration; -import java.util.Base64; import java.util.Collections; import java.util.Iterator; import java.util.List; @@ -51,125 +41,88 @@ import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Function; import java.util.regex.Pattern; -import okhttp3.Credentials; -import okhttp3.OkHttpClient; -import okhttp3.Request; -import okhttp3.Response; -import okhttp3.WebSocket; -import okhttp3.WebSocketListener; -import okio.ByteString; /** - * A synchronous bidirectional realtime session connected to a Foundry voice agent. + * Shared blocking WebSocket connection and I/O, independent of the asynchronous transport. */ -@Beta(warningText = "This class is in preview and may change in future releases.") -public final class BetaVoiceAgentWebSocketSessionClient implements AutoCloseable { - private static final ClientLogger LOGGER = new ClientLogger(BetaVoiceAgentWebSocketSessionClient.class); +public final class WebSocketSessionTransport implements AutoCloseable { + private static final ClientLogger LOGGER = new ClientLogger(WebSocketSessionTransport.class); private final URI websocketUri; - private final VoiceAgentWebSocketConnectionOptions options; + private final WebSocketTransportOptions options; private final OkHttpClient httpClient; private final BlockingQueue events; private final int receiveBufferCapacity; private final CountDownLatch handshakeCompleted = new CountDownLatch(1); private final CountDownLatch closeCompleted = new CountDownLatch(1); private final AtomicReference connectionError = new AtomicReference<>(); - private final AtomicBoolean handshakeSucceeded = new AtomicBoolean(); private final AtomicBoolean receiveClaimed = new AtomicBoolean(); - private final AtomicBoolean open = new AtomicBoolean(); - private final AtomicBoolean closed = new AtomicBoolean(); + private final AtomicReference state = new AtomicReference<>(WebSocketSessionState.NEW); private final AtomicBoolean clientShutdown = new AtomicBoolean(); - private volatile WebSocket webSocket; private volatile Integer closeCode; private volatile String closeReason; - private BetaVoiceAgentWebSocketSessionClient(VoiceAgentWebSocketClientConfiguration configuration, String agentName, - VoiceAgentWebSocketConnectionOptions options) { + private WebSocketSessionTransport(VoiceAgentWebSocketClientConfiguration configuration, URI websocketUri, + WebSocketTransportOptions options) { this.options = options; this.receiveBufferCapacity = options.getReceiveBufferCapacity(); this.events = new ArrayBlockingQueue<>(receiveBufferCapacity + 1); - this.websocketUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, agentName, options); + this.websocketUri = Objects.requireNonNull(websocketUri, "'websocketUri' cannot be null."); + this.httpClient = createHttpClient(configuration, options); + } + + private void open(VoiceAgentWebSocketClientConfiguration configuration, String subprotocol, + Function headersFactory) { + state.set(WebSocketSessionState.CONNECTING); String token = configuration.getCredential() .getTokenSync(VoiceAgentWebSocketUtils.createTokenRequestContext()) .getToken(); - this.httpClient = createHttpClient(configuration, options); - Request.Builder request = new Request.Builder().url(websocketUri.toString()) - .header("Sec-WebSocket-Protocol", VoiceAgentWebSocketUtils.SUBPROTOCOL); - for (HttpHeader header : VoiceAgentWebSocketUtils.buildHeaders(configuration, options, token)) { + Request.Builder request = new Request.Builder().url(websocketUri.toString()); + if (subprotocol != null) { + request.header("Sec-WebSocket-Protocol", subprotocol); + } + for (HttpHeader header : headersFactory.apply(token)) { request.header(header.getName(), header.getValue()); } this.webSocket = httpClient.newWebSocket(request.build(), new Listener()); } - static BetaVoiceAgentWebSocketSessionClient connect(VoiceAgentWebSocketClientConfiguration configuration, - String agentName, VoiceAgentWebSocketConnectionOptions options) { - BetaVoiceAgentWebSocketSessionClient session - = new BetaVoiceAgentWebSocketSessionClient(configuration, agentName, options); + public static WebSocketSessionTransport connect(VoiceAgentWebSocketClientConfiguration configuration, + URI websocketUri, WebSocketTransportOptions options, String subprotocol, + Function headersFactory) { + WebSocketSessionTransport session = new WebSocketSessionTransport(configuration, websocketUri, options); try { + session.open(configuration, subprotocol, headersFactory); session.awaitHandshake(); return session; } catch (RuntimeException error) { - session.webSocket.cancel(); + if (session.webSocket != null) { + session.webSocket.cancel(); + } session.shutdownHttpClient(); throw error; } } - /** - * Gets the WebSocket endpoint used by this session. - * - * @return the WebSocket endpoint. - */ public String getEndpoint() { return websocketUri.toString(); } - /** - * Determines whether the session is open. - * - * @return {@code true} when the session is open. - */ public boolean isOpen() { - return open.get() && !closed.get(); + return state.get() == WebSocketSessionState.OPEN; } - /** - * Gets the peer close code. - * - * @return the close code, or {@code null}. - */ public Integer getCloseCode() { return closeCode; } - /** - * Gets the peer close reason. - * - * @return the close reason, or {@code null}. - */ public String getCloseReason() { return closeReason; } - /** - * Receives typed server events in wire order. The returned stream may be iterated once. - * - * @throws IllegalStateException if the event stream has already been claimed. - * @return the server event stream. - */ - public IterableStream receiveEvents() { - return receiveEvents(null); - } - - /** - * Receives events with a timeout for each wait. A timeout leaves the session open and the iterator can be retried. - * @param timeout positive per-event timeout, or null to wait indefinitely. - * @return the single-consumer event stream. - * @throws IllegalArgumentException if timeout is zero or negative. - * @throws IllegalStateException if another receiver exists or a wait times out (with a TimeoutException cause). - */ - public IterableStream receiveEvents(Duration timeout) { + public IterableStream receiveEvents(Duration timeout) { if (timeout != null && (timeout.isZero() || timeout.isNegative())) { throw LOGGER.logExceptionAsError(new IllegalArgumentException("Receive timeout must be positive.")); } @@ -187,155 +140,29 @@ public IterableStream receiveEvents(Duration timeout) { }); } - /** - * Sends a typed realtime client event. - * - * @param event the event to send. - * @throws IllegalArgumentException if the event cannot be serialized. - * @throws IllegalStateException if the session is closed or cannot accept another event. - */ - public void sendEvent(RealtimeClientEvent event) { - Objects.requireNonNull(event, "'event' cannot be null."); - ensureOpen(); - final String json; - try { - json = event.toJsonString(); - } catch (IOException error) { - throw LOGGER.logExceptionAsError( - new IllegalArgumentException("Failed to serialize the realtime client event.", error)); - } - if (!webSocket.send(json)) { - throw LOGGER.logExceptionAsError( - new IllegalStateException("The voice-agent WebSocket send queue is full or closed.")); - } - } - - /** - * Sends a JSON object, including event types and fields unknown to this SDK. - * @param event the complete JSON event. - * @throws IllegalArgumentException if the event is not a JSON object. - * @throws IllegalStateException if the session cannot accept the event. - */ public void sendEvent(BinaryData event) { String json = VoiceAgentWebSocketUtils.validateEvent(event); - ensureOpen(); + if (!isOpen()) { + throw LOGGER + .logExceptionAsError(new IllegalStateException("The voice-agent WebSocket session is not open.")); + } if (!webSocket.send(json)) { throw LOGGER.logExceptionAsError( new IllegalStateException("The voice-agent WebSocket send queue is full or closed.")); } } - /** - * Adds a conversation item. - * - * @param item the item to add. - */ - public void createConversationItem(RealtimeConversationItem item) { - createConversationItem(item, null); - } - - /** - * Adds a conversation item after another item. - * - * @param item the item to add. - * @param previousItemId the preceding item identifier. - */ - public void createConversationItem(RealtimeConversationItem item, String previousItemId) { - Objects.requireNonNull(item, "'item' cannot be null."); - sendEvent(new RealtimeConversationItemCreateEvent(item).setPreviousItemId(previousItemId)); - } - - /** - * Adds a user text message. - * - * @param text the user message. - */ - public void sendText(String text) { - Objects.requireNonNull(text, "'text' cannot be null."); - Content content = Content.builder().type(Content.Type.INPUT_TEXT).text(text).build(); - createConversationItem(new RealtimeConversationItemUserMessage(Collections.singletonList(content))); - } - - /** - * Appends audio to the input buffer. - * - * @param audio the audio bytes. - */ - public void appendInputAudio(BinaryData audio) { - Objects.requireNonNull(audio, "'audio' cannot be null."); - sendEvent(new RealtimeInputAudioBufferAppendEvent(Base64.getEncoder().encodeToString(audio.toBytes()))); - } - - /** Clears the input audio buffer. */ - public void clearInputAudio() { - sendEvent(new RealtimeInputAudioBufferClearEvent()); - } - - /** Commits the input audio buffer. */ - public void commitInputAudio() { - sendEvent(new RealtimeInputAudioBufferCommitEvent()); - } - - /** Requests a response using the voice agent's configuration. */ - public void createResponse() { - sendEvent(new RealtimeResponseCreateEvent()); - } - - /** - * Requests a response with per-response options. - * - * @param responseOptions response options. - */ - public void createResponse(VoiceAgentResponseCreateOptions responseOptions) { - Objects.requireNonNull(responseOptions, "'responseOptions' cannot be null."); - sendEvent(new RealtimeResponseCreateEvent().setResponse(responseOptions)); - } - - /** Cancels the active response. */ - public void cancelResponse() { - sendEvent(new RealtimeResponseCancelEvent()); - } - - /** - * Cancels a specific response. - * - * @param responseId the response identifier. - */ - public void cancelResponse(String responseId) { - Objects.requireNonNull(responseId, "'responseId' cannot be null."); - sendEvent(new RealtimeResponseCancelEvent().setResponseId(responseId)); - } - - /** - * Sends a function-call result and requests the next response. - * - * @param callId the function call identifier. - * @param output the serialized function output. - */ - public void sendFunctionCallOutput(String callId, String output) { - createConversationItem(new RealtimeConversationItemFunctionCallOutput(callId, output)); - createResponse(); - } - - /** Closes the session. */ @Override public void close() { close(1000, ""); } - /** - * Closes the session with an application-selected WebSocket close frame. - * @param code a valid WebSocket close code. - * @param reason non-null reason of at most 123 UTF-8 bytes. - * @throws IllegalArgumentException if the code or reason is invalid. - */ public void close(int code, String reason) { VoiceAgentWebSocketUtils.validateClose(code, reason); - if (!closed.compareAndSet(false, true)) { + if (!state.compareAndSet(WebSocketSessionState.OPEN, WebSocketSessionState.CLOSING)) { shutdownHttpClient(); return; } - open.set(false); if (!webSocket.close(code, reason)) { webSocket.cancel(); closeCompleted.countDown(); @@ -348,6 +175,7 @@ public void close(int code, String reason) { Thread.currentThread().interrupt(); webSocket.cancel(); } finally { + state.set(WebSocketSessionState.CLOSED); signal(EventSignal.complete()); shutdownHttpClient(); } @@ -366,9 +194,6 @@ private void awaitHandshake() { throw LOGGER.logExceptionAsError( new IllegalStateException("Interrupted while opening the voice-agent WebSocket session.", error)); } - if (handshakeSucceeded.get()) { - return; - } Throwable error = connectionError.get(); if (error instanceof RuntimeException) { throw LOGGER.logExceptionAsError((RuntimeException) error); @@ -379,13 +204,6 @@ private void awaitHandshake() { } } - private void ensureOpen() { - if (!isOpen()) { - throw LOGGER - .logExceptionAsError(new IllegalStateException("The voice-agent WebSocket session is not open.")); - } - } - private void fail(Throwable error, Response response) { Throwable mapped; try { @@ -395,9 +213,10 @@ private void fail(Throwable error, Response response) { response.close(); } } - connectionError.compareAndSet(null, mapped); - open.set(false); - closed.set(true); + WebSocketSessionState previous = state.getAndSet(WebSocketSessionState.CLOSED); + if (previous == WebSocketSessionState.CONNECTING) { + connectionError.compareAndSet(null, mapped); + } handshakeCompleted.countDown(); signal(EventSignal.error(mapped)); closeCompleted.countDown(); @@ -443,8 +262,7 @@ private synchronized void signal(EventSignal signal) { if (current != null) { current.cancel(); } - open.set(false); - closed.set(true); + state.set(WebSocketSessionState.CLOSED); shutdownHttpClient(); } } @@ -456,8 +274,8 @@ private void shutdownHttpClient() { } } - static OkHttpClient createHttpClient(VoiceAgentWebSocketClientConfiguration configuration, - VoiceAgentWebSocketConnectionOptions options) { + public static OkHttpClient createHttpClient(VoiceAgentWebSocketClientConfiguration configuration, + WebSocketTransportOptions options) { OkHttpClient.Builder builder = new OkHttpClient.Builder(); builder.connectTimeout(options.getHandshakeTimeout().toMillis(), TimeUnit.MILLISECONDS) .readTimeout(0, TimeUnit.MILLISECONDS) @@ -483,7 +301,8 @@ public List select(URI uri) { @Override public void connectFailed(URI uri, SocketAddress address, IOException error) { LOGGER.atVerbose() - .addKeyValue("uri", uri) + .addKeyValue("host", uri.getHost()) + .addKeyValue("path", uri.getPath()) .addKeyValue("proxyAddress", address) .log("Failed to connect through the configured proxy."); } @@ -503,8 +322,7 @@ public void connectFailed(URI uri, SocketAddress address, IOException error) { private final class Listener extends WebSocketListener { @Override public void onOpen(WebSocket webSocket, Response response) { - handshakeSucceeded.set(true); - open.set(true); + state.compareAndSet(WebSocketSessionState.CONNECTING, WebSocketSessionState.OPEN); handshakeCompleted.countDown(); } @@ -515,11 +333,7 @@ public void onMessage(WebSocket webSocket, String text) { webSocket.close(1009, "Message too large"); return; } - try { - signal(EventSignal.event(VoiceAgentWebSocketUtils.deserializeEvent(text))); - } catch (IOException | RuntimeException error) { - malformedEvent(webSocket, error); - } + signal(EventSignal.event(BinaryData.fromString(text))); } @Override @@ -545,8 +359,8 @@ private void malformedEvent(WebSocket webSocket, Throwable error) { error = callbackError; } } - fail(new IllegalArgumentException("Invalid JSON event.", error), null); - webSocket.close(1007, "Invalid JSON event"); + fail(new IllegalArgumentException("Invalid UTF-8 message.", error), null); + webSocket.close(1007, "Invalid UTF-8 message"); } @Override @@ -560,8 +374,11 @@ public void onClosing(WebSocket webSocket, int code, String reason) { public void onClosed(WebSocket webSocket, int code, String reason) { closeCode = code; closeReason = reason; - open.set(false); - closed.set(true); + WebSocketSessionState previous = state.getAndSet(WebSocketSessionState.CLOSED); + if (previous == WebSocketSessionState.CONNECTING) { + connectionError.compareAndSet(null, + new IllegalStateException("The WebSocket closed before the handshake completed.")); + } signal(EventSignal.complete()); handshakeCompleted.countDown(); closeCompleted.countDown(); @@ -575,17 +392,17 @@ public void onFailure(WebSocket webSocket, Throwable error, Response response) { } private static final class EventSignal { - private final RealtimeServerEvent event; + private final BinaryData event; private final Throwable error; private final boolean complete; - private EventSignal(RealtimeServerEvent event, Throwable error, boolean complete) { + private EventSignal(BinaryData event, Throwable error, boolean complete) { this.event = event; this.error = error; this.complete = complete; } - private static EventSignal event(RealtimeServerEvent event) { + private static EventSignal event(BinaryData event) { return new EventSignal(event, null, false); } @@ -598,7 +415,7 @@ private static EventSignal complete() { } } - private static final class EventIterator implements Iterator { + private static final class EventIterator implements Iterator { private final BlockingQueue events; private final Duration timeout; private EventSignal next; @@ -634,11 +451,11 @@ public boolean hasNext() { } @Override - public RealtimeServerEvent next() { + public BinaryData next() { if (!hasNext()) { throw new NoSuchElementException(); } - RealtimeServerEvent event = next.event; + BinaryData event = next.event; next = null; return event; } diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketTransportOptions.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketTransportOptions.java new file mode 100644 index 0000000000000..71599f32f3b4a --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/WebSocketTransportOptions.java @@ -0,0 +1,67 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.implementation.realtime; + +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; +import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; +import com.azure.ai.agents.models.VoiceAgentWebSocketOverflowStrategy; +import java.time.Duration; +import java.util.Objects; +import java.util.function.Consumer; + +/** + * Immutable transport settings captured from a protocol-specific public options bag. + */ +public final class WebSocketTransportOptions { + private final int receiveBufferCapacity; + private final int maxMessageSize; + private final VoiceAgentWebSocketOverflowStrategy overflowStrategy; + private final Consumer malformedEventHandler; + private final Duration handshakeTimeout; + private final Duration closeTimeout; + + public WebSocketTransportOptions(VoiceAgentWebSocketConnectionOptions options) { + Objects.requireNonNull(options, "'options' cannot be null."); + this.receiveBufferCapacity = options.getReceiveBufferCapacity(); + this.maxMessageSize = options.getMaxMessageSize(); + this.overflowStrategy = options.getOverflowStrategy(); + this.malformedEventHandler = options.getMalformedEventHandler(); + this.handshakeTimeout = options.getHandshakeTimeout(); + this.closeTimeout = options.getCloseTimeout(); + } + + public WebSocketTransportOptions(VoiceAgentLiveWebSocketConnectionOptions options) { + Objects.requireNonNull(options, "'options' cannot be null."); + this.receiveBufferCapacity = options.getReceiveBufferCapacity(); + this.maxMessageSize = options.getMaxMessageSize(); + this.overflowStrategy = options.getOverflowStrategy(); + this.malformedEventHandler = options.getMalformedEventHandler(); + this.handshakeTimeout = options.getHandshakeTimeout(); + this.closeTimeout = options.getCloseTimeout(); + } + + public int getReceiveBufferCapacity() { + return receiveBufferCapacity; + } + + public int getMaxMessageSize() { + return maxMessageSize; + } + + public VoiceAgentWebSocketOverflowStrategy getOverflowStrategy() { + return overflowStrategy; + } + + public Consumer getMalformedEventHandler() { + return malformedEventHandler; + } + + public Duration getHandshakeTimeout() { + return handshakeTimeout; + } + + public Duration getCloseTimeout() { + return closeTimeout; + } +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentLiveWebSocketConnectionOptions.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentLiveWebSocketConnectionOptions.java new file mode 100644 index 0000000000000..06bc43c606477 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentLiveWebSocketConnectionOptions.java @@ -0,0 +1,258 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.models; + +import com.azure.ai.agents.implementation.utils.Beta; +import com.azure.core.annotation.Fluent; +import com.azure.core.util.logging.ClientLogger; + +import java.time.Duration; +import java.util.Objects; +import java.util.function.Consumer; + +/** + * Connection and buffering options for a persisted GPT Live voice agent. + * Configure these options before opening a session; clients snapshot them when opening. + */ +@Beta(warningText = "Preview API. VoiceAgents=V1Preview") +@Fluent +public final class VoiceAgentLiveWebSocketConnectionOptions { + private static final ClientLogger LOGGER = new ClientLogger(VoiceAgentLiveWebSocketConnectionOptions.class); + private int receiveBufferCapacity = 256; + private int maxMessageSize = 32 * 1024 * 1024; + private VoiceAgentWebSocketOverflowStrategy overflowStrategy = VoiceAgentWebSocketOverflowStrategy.ERROR; + private Consumer malformedEventHandler; + private Duration handshakeTimeout = Duration.ofSeconds(30); + private Duration closeTimeout = Duration.ofSeconds(10); + private Boolean store; + private String agentVersionOverride; + private String structuredInputs; + + /** + * Creates options with a 30-second handshake timeout and a 256-event receive buffer. + */ + public VoiceAgentLiveWebSocketConnectionOptions() { + } + + /** + * Copies connection options. + * + * @param source the options to copy. + */ + public VoiceAgentLiveWebSocketConnectionOptions(VoiceAgentLiveWebSocketConnectionOptions source) { + Objects.requireNonNull(source, "'source' cannot be null."); + this.receiveBufferCapacity = source.receiveBufferCapacity; + this.maxMessageSize = source.maxMessageSize; + this.overflowStrategy = source.overflowStrategy; + this.malformedEventHandler = source.malformedEventHandler; + this.handshakeTimeout = source.handshakeTimeout; + this.closeTimeout = source.closeTimeout; + this.store = source.store; + this.agentVersionOverride = source.agentVersionOverride; + this.structuredInputs = source.structuredInputs; + } + + /** + * Gets the receive queue capacity. + * + * @return the maximum queued events, default 256. + */ + public int getReceiveBufferCapacity() { + return receiveBufferCapacity; + } + + /** + * Sets the receive queue capacity. + * + * @param capacity the capacity, between 1 and 65536. + * @return this options instance. + * @throws IllegalArgumentException if the capacity is outside the supported range. + */ + public VoiceAgentLiveWebSocketConnectionOptions setReceiveBufferCapacity(int capacity) { + if (capacity < 1 || capacity > 65536) { + throw LOGGER + .logExceptionAsError(new IllegalArgumentException("Receive capacity must be between 1 and 65536.")); + } + this.receiveBufferCapacity = capacity; + return this; + } + + /** + * Gets the maximum accepted message size. + * + * @return the size in bytes, default 32 MiB. + */ + public int getMaxMessageSize() { + return maxMessageSize; + } + + /** + * Sets the maximum accepted message size. + * + * @param bytes the positive maximum message size in bytes. + * @return this options instance. + * @throws IllegalArgumentException if the size is not positive. + */ + public VoiceAgentLiveWebSocketConnectionOptions setMaxMessageSize(int bytes) { + if (bytes <= 0) { + throw LOGGER.logExceptionAsError(new IllegalArgumentException("Message size must be positive.")); + } + this.maxMessageSize = bytes; + return this; + } + + /** + * Gets the receive queue overflow strategy. + * + * @return the strategy, default ERROR. + */ + public VoiceAgentWebSocketOverflowStrategy getOverflowStrategy() { + return overflowStrategy; + } + + /** + * Sets the overflow strategy. Drop strategies explicitly permit event loss. + * + * @param strategy the non-null strategy. + * @return this options instance. + */ + public VoiceAgentLiveWebSocketConnectionOptions setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy strategy) { + this.overflowStrategy = Objects.requireNonNull(strategy, "'strategy' cannot be null."); + return this; + } + + /** + * Gets the callback for malformed events. + * + * @return the callback, or null to terminate the session. + */ + public Consumer getMalformedEventHandler() { + return malformedEventHandler; + } + + /** + * Sets a callback to report and skip malformed events. It runs while receiving events and must not block. + * JSON decoding failures are reported only by typed receivers; raw receivers do not parse JSON. + * Invalid UTF-8 is reported by the transport for either receiver type. + * If the callback throws, the session terminates. Transport errors and oversized messages are not skipped. + * + * @param handler the callback, or null to terminate on malformed events. + * @return this options instance. + */ + public VoiceAgentLiveWebSocketConnectionOptions setMalformedEventHandler(Consumer handler) { + this.malformedEventHandler = handler; + return this; + } + + /** + * Gets the handshake timeout. + * + * @return the timeout, default 30 seconds. + */ + public Duration getHandshakeTimeout() { + return handshakeTimeout; + } + + /** + * Sets the handshake timeout. + * + * @param timeout a non-null positive timeout. + * @return this options instance. + * @throws IllegalArgumentException if the timeout is not positive. + */ + public VoiceAgentLiveWebSocketConnectionOptions setHandshakeTimeout(Duration timeout) { + validatePositive(timeout, "handshakeTimeout"); + this.handshakeTimeout = timeout; + return this; + } + + /** + * Gets the close timeout. + * + * @return the timeout, default 10 seconds. + */ + public Duration getCloseTimeout() { + return closeTimeout; + } + + /** + * Sets the close timeout. + * + * @param timeout a non-null positive timeout. + * @return this options instance. + * @throws IllegalArgumentException if the timeout is not positive. + */ + public VoiceAgentLiveWebSocketConnectionOptions setCloseTimeout(Duration timeout) { + validatePositive(timeout, "closeTimeout"); + this.closeTimeout = timeout; + return this; + } + + /** + * Gets the persistence override. + * + * @return the override, or null to use the persisted agent configuration. + */ + public Boolean isStoreEnabled() { + return store; + } + + /** + * Overrides Foundry conversation and audio persistence for this session. + * + * @param store the override, or null to use the persisted agent configuration. + * @return this options instance. + */ + public VoiceAgentLiveWebSocketConnectionOptions setStoreEnabled(Boolean store) { + this.store = store; + return this; + } + + /** + * Gets the agent version override. + * + * @return the version override, or null. + */ + public String getAgentVersionOverride() { + return agentVersionOverride; + } + + /** + * Sets the agent version to connect to. + * + * @param version the version override, or null to use the default version. + * @return this options instance. + */ + public VoiceAgentLiveWebSocketConnectionOptions setAgentVersionOverride(String version) { + this.agentVersionOverride = version; + return this; + } + + /** + * Gets structured inputs for instruction and greeting templates. + * + * @return the JSON inputs, or null. + */ + public String getStructuredInputs() { + return structuredInputs; + } + + /** + * Sets the JSON inputs for instruction and greeting templates. + * The SDK URL-encodes this value into the {@code structured_inputs} query parameter. + * + * @param inputs the unencoded JSON inputs, or null. + * @return this options instance. + */ + public VoiceAgentLiveWebSocketConnectionOptions setStructuredInputs(String inputs) { + this.structuredInputs = inputs; + return this; + } + + private static void validatePositive(Duration duration, String name) { + if (duration == null || duration.isZero() || duration.isNegative()) { + throw new IllegalArgumentException("'" + name + "' must be positive."); + } + } +} diff --git a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentWebSocketConnectionOptions.java b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentWebSocketConnectionOptions.java index 0184f4f3837b5..bebc3591e569f 100644 --- a/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentWebSocketConnectionOptions.java +++ b/sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentWebSocketConnectionOptions.java @@ -6,6 +6,7 @@ import com.azure.ai.agents.implementation.utils.Beta; import com.azure.core.annotation.Fluent; import com.azure.core.util.logging.ClientLogger; + import java.time.Duration; import java.util.Objects; import java.util.function.Consumer; @@ -24,6 +25,7 @@ public final class VoiceAgentWebSocketConnectionOptions { /** * Gets the maximum number of queued events. + * * @return the capacity, default 256. */ public int getReceiveBufferCapacity() { @@ -32,9 +34,10 @@ public int getReceiveBufferCapacity() { /** * Sets the bounded receive queue capacity. Configure before connecting. + * * @param capacity number of events, between 1 and 65536. * @return this options instance. - * @throws IllegalArgumentException if capacity is outside the supported range. + * @throws IllegalArgumentException if capacity is outside the supported range. */ public VoiceAgentWebSocketConnectionOptions setReceiveBufferCapacity(int capacity) { if (capacity < 1 || capacity > 65536) { @@ -47,6 +50,7 @@ public VoiceAgentWebSocketConnectionOptions setReceiveBufferCapacity(int capacit /** * Gets the maximum accepted JSON message size. + * * @return the size in bytes, default 32 MiB. */ public int getMaxMessageSize() { @@ -55,9 +59,10 @@ public int getMaxMessageSize() { /** * Sets the maximum accepted JSON message size. Oversized messages terminate the connection. + * * @param bytes positive size in bytes. * @return this options instance. - * @throws IllegalArgumentException if bytes is not positive. + * @throws IllegalArgumentException if bytes is not positive. */ public VoiceAgentWebSocketConnectionOptions setMaxMessageSize(int bytes) { if (bytes <= 0) { @@ -69,6 +74,7 @@ public VoiceAgentWebSocketConnectionOptions setMaxMessageSize(int bytes) { /** * Gets the receive queue overflow action. + * * @return the strategy, default ERROR. */ public VoiceAgentWebSocketOverflowStrategy getOverflowStrategy() { @@ -77,6 +83,7 @@ public VoiceAgentWebSocketOverflowStrategy getOverflowStrategy() { /** * Sets the receive queue overflow action. Drop strategies explicitly permit data loss. + * * @param strategy the non-null strategy. * @return this options instance. */ @@ -87,6 +94,7 @@ public VoiceAgentWebSocketConnectionOptions setOverflowStrategy(VoiceAgentWebSoc /** * Gets the callback for skipping malformed events. + * * @return the callback, or null to terminate on malformed events. */ public Consumer getMalformedEventHandler() { @@ -97,6 +105,7 @@ public Consumer getMalformedEventHandler() { * Sets a callback that reports and skips malformed events without terminating reception. * The callback runs on the receive thread and must not block. If it throws, the session terminates. * This does not recover from transport errors or oversized messages. + * * @param handler callback, or null to terminate on malformed events (the default). * @return this options instance. */ @@ -142,6 +151,7 @@ public VoiceAgentWebSocketConnectionOptions(VoiceAgentWebSocketConnectionOptions /** * Gets the session correlation identifier. + * * @return the session identifier, or null. */ public String getAgentSessionId() { @@ -150,6 +160,7 @@ public String getAgentSessionId() { /** * Sets the session correlation identifier. + * * @param agentSessionId the session identifier, or null. * @return this options instance. */ @@ -160,6 +171,7 @@ public VoiceAgentWebSocketConnectionOptions setAgentSessionId(String agentSessio /** * Gets the structured inputs JSON object sent in the handshake header. + * * @return the structured inputs, or null. */ public String getStructuredInputs() { @@ -168,6 +180,7 @@ public String getStructuredInputs() { /** * Sets the structured inputs JSON object sent in the handshake header. + * * @param structuredInputs the JSON object, or null. * @return this options instance. */ diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/ReadmeSamples.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/ReadmeSamples.java index f7884be7b05e1..4cf121f591473 100644 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/ReadmeSamples.java +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/ReadmeSamples.java @@ -10,6 +10,7 @@ import com.azure.ai.agents.models.RawRealtimeServerEvent; import com.azure.ai.agents.models.RealtimeServerEvent; import com.azure.ai.agents.models.SessionLogEvent; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; import com.azure.ai.agents.models.VoiceAgentWebSocketOverflowStrategy; import com.azure.core.util.BinaryData; @@ -25,16 +26,45 @@ import com.openai.models.responses.Response; import com.openai.models.responses.ResponseCreateParams; import com.openai.services.blocking.ConversationService; +import com.openai.models.live.ClientEvent; +import com.openai.models.live.ResponseCreateEvent; +import com.openai.models.live.ServerEvent; +import java.time.Duration; public final class ReadmeSamples { - public void realtimeForwardCompatibility(BetaVoiceAgentWebSocketClient realtimeClient, String agentName) { + public void liveConnectionOptions(AgentsClientBuilder builder, String agentName) { + // BEGIN: readme-sample-com.azure.ai.agents.live_connection_options + BetaVoiceAgentsLiveWebSocketClient liveClient = builder.beta().buildBetaVoiceAgentsLiveWebSocketClient(); + VoiceAgentLiveWebSocketConnectionOptions options = new VoiceAgentLiveWebSocketConnectionOptions() + .setReceiveBufferCapacity(512) + .setMaxMessageSize(8 * 1024 * 1024) + .setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.ERROR) + .setHandshakeTimeout(Duration.ofSeconds(30)) + .setCloseTimeout(Duration.ofSeconds(10)) + .setStoreEnabled(false); + + try (BetaVoiceAgentsLiveWebSocketSessionClient session = liveClient.openWebSocketSession(agentName, options)) { + for (ServerEvent event : session.receiveEvents()) { + if (event.isSessionStarted()) { + session.sendEvent(ClientEvent.ofResponseCreate(ResponseCreateEvent.builder().build())); + } else if (event.isSessionOutputTranscriptDelta()) { + System.out.print(event.asSessionOutputTranscriptDelta().delta()); + } else if (event.isSessionClosed() || event.isErrorEvent()) { + break; + } + } + } + // END: readme-sample-com.azure.ai.agents.live_connection_options + } + + public void realtimeForwardCompatibility(BetaVoiceAgentsWebSocketClient realtimeClient, String agentName) { // BEGIN: com.azure.ai.agents.realtime_forward_compatibility VoiceAgentWebSocketConnectionOptions options = new VoiceAgentWebSocketConnectionOptions() .setReceiveBufferCapacity(512) .setMaxMessageSize(8 * 1024 * 1024) .setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.ERROR); - try (BetaVoiceAgentWebSocketSessionClient session = realtimeClient.openWebSocketSession(agentName, options)) { + try (BetaVoiceAgentsWebSocketSessionClient session = realtimeClient.openWebSocketSession(agentName, options)) { session.sendEvent(BinaryData.fromString( "{\"type\":\"response.create\",\"event_id\":\"response-1\"}")); for (RealtimeServerEvent event : session.receiveEvents()) { diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveAudioConversationAsyncSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveAudioConversationAsyncSample.java new file mode 100644 index 0000000000000..0a59abaffd26a --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveAudioConversationAsyncSample.java @@ -0,0 +1,216 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.BetaVoiceAgentsLiveWebSocketSessionAsyncClient; +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; +import com.azure.core.util.Configuration; +import com.azure.identity.DefaultAzureCredentialBuilder; +import reactor.core.publisher.Mono; +import reactor.core.scheduler.Schedulers; +import com.openai.models.live.ClientEvent; +import com.openai.models.live.ServerEvent; +import com.openai.models.responses.ResponseStreamEvent; +import java.io.InputStream; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Consumer; +import java.util.function.Function; +import java.util.function.Predicate; +import java.util.function.Supplier; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Sinks; + + +/** + * Streams microphone and speaker audio over a native GPT Live WebSocket. + * Requires Java Sound devices supporting 24-kHz mono PCM16LE. Press Enter to finish. + * Audio capture and playback run on dedicated threads, not the WebSocket receive thread. + * Playback is bounded to 60 seconds; overflow and device failures terminate the call explicitly. + * The uniquely named agent is deleted unless FOUNDRY_KEEP_GPT_LIVE_AGENT=true. + * Requests an opening greeting and labels spoken transcripts by speaker. + * The standalone main method shuts down shared Reactor schedulers on exit; do not invoke it inside a shared application. + * + *

Before running the sample, set these environment variables:

+ *
    + *
  • {@code FOUNDRY_PROJECT_ENDPOINT} - The Azure AI Project endpoint.
    • + *
  • {@code FOUNDRY_GPT_LIVE_MODEL} - Optional. The GPT Live deployment name. Defaults to {@code gpt-live-1}.
    • + *
  • {@code FOUNDRY_GPT_LIVE_BACKEND_MODEL} - Optional. The Responses backend deployment name in the Live + * resource. Defaults to {@code gpt-5.6-terra}.
    • + *
  • {@code FOUNDRY_GPT_LIVE_STORE} - Optional. Set to {@code true} to persist conversation content and audio + * and request a Foundry conversation ID. Defaults to {@code false}. Also keep the agent for later REST reads.
    • + *
  • {@code FOUNDRY_KEEP_GPT_LIVE_AGENT} - Optional. Set to {@code true} to keep the agent after the sample. + * Defaults to {@code false}.
    • + *
+ */ +public class GptLiveAudioConversationAsyncSample { + public static void main(String[] args) { + Configuration configuration = Configuration.getGlobalConfiguration(); + String endpoint = configuration.get("FOUNDRY_PROJECT_ENDPOINT"); + String model = configuration.get("FOUNDRY_GPT_LIVE_MODEL"); + String backendModel = configuration.get("FOUNDRY_GPT_LIVE_BACKEND_MODEL"); + boolean keepAgent = Boolean.parseBoolean(configuration.get("FOUNDRY_KEEP_GPT_LIVE_AGENT", "false")); + boolean store = Boolean.parseBoolean(configuration.get("FOUNDRY_GPT_LIVE_STORE", "false")); + + if (endpoint == null || endpoint.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_PROJECT_ENDPOINT before running this sample."); + } + if (model == null) { + model = "gpt-live-1"; + } else if (model.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_GPT_LIVE_MODEL before running this sample."); + } + if (backendModel == null) { + backendModel = "gpt-5.6-terra"; + } else if (backendModel.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_GPT_LIVE_BACKEND_MODEL before running this sample."); + } + + AgentsClientBuilder builder = new AgentsClientBuilder().endpoint(endpoint) + .credential(new DefaultAzureCredentialBuilder().build()) + .allowPreview(true); + try (GptLiveSampleUtils.AgentResource agent + = GptLiveSampleUtils.createAgent(builder, model, backendModel, "audio-async", false, keepAgent)) { + Mono.usingWhen(agent.builder.beta().buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession(agent.name, new VoiceAgentLiveWebSocketConnectionOptions().setStoreEnabled(store)), + GptLiveAudioConversationAsyncSample::runAudioConversationAsync, + BetaVoiceAgentsLiveWebSocketSessionAsyncClient::closeAsync, + (session, error) -> session.closeAsync(), + BetaVoiceAgentsLiveWebSocketSessionAsyncClient::closeAsync).block(); + } finally { + Schedulers.shutdownNow(); + } + } + + static Mono runAudioConversationAsync(BetaVoiceAgentsLiveWebSocketSessionAsyncClient session) { + return Mono.defer(() -> runAudioConversationAsync(session, + new GptLiveSampleUtils.AudioProcessor( + null, null, pcm -> session.sendEvent(GptLiveSampleUtils.audio(pcm.toBytes()))), System.in)); + } + + static Mono runAudioConversationAsync(BetaVoiceAgentsLiveWebSocketSessionAsyncClient session, + GptLiveSampleUtils.AudioProcessor processor, InputStream input) { + return Mono.usingWhen(Mono.just(processor), + audio -> Mono.firstWithSignal(runConversationAsync(session.receiveEvents(), session::sendEvent, + () -> "transport close code=" + session.getCloseCode() + ", reason=" + session.getCloseReason(), + transcripts -> VoiceAgentLiveAudioConversationAsyncSample.waitForEnter(input) + .then(closeAudioAsync(audio)).then(Mono.empty()), + audio::queueAudio, () -> { }, true, audio), audio.failureSignalAsync()), GptLiveAudioConversationAsyncSample::closeAudioAsync, + (audio, error) -> closeAudioAsync(audio), GptLiveAudioConversationAsyncSample::closeAudioAsync); + } + + private static Mono closeAudioAsync(GptLiveSampleUtils.AudioProcessor audio) { + return Mono.fromRunnable(audio::close).subscribeOn(Schedulers.boundedElastic()); + } + + private static Mono runConversationAsync(Flux events, + Function> sendEvent, Supplier closeDetails, + Function> nextPrompt, + Consumer playAudio, Runnable drainAudio, boolean spoken, GptLiveSampleUtils.AudioProcessor microphone) { + Mono opening = microphone == null ? Mono.empty() + : Mono.defer(() -> sendEvent.apply(GptLiveSampleUtils.userMessage("Greet me briefly with: Hi, how can I help you today?")) + .then(sendEvent.apply(GptLiveSampleUtils.responseCreate())) + .then(Mono.fromRunnable(() -> System.out.println( + "Opening greeting requested. Speak into the microphone. Press Enter to end the call.")))); + return runConversationCoreAsync(events, sendEvent, closeDetails, nextPrompt, playAudio, drainAudio, spoken, + spoken && microphone == null, microphone == null && System.console() != null, + () -> Mono.fromRunnable(() -> { + if (microphone != null) { + microphone.start(); + } + }), opening, backend -> Mono.just(false), reason -> microphone != null); + } + + static Mono runConversationCoreAsync(Flux events, + Function> sendEvent, Supplier closeDetails, + Function> nextPrompt, + Consumer playAudio, Runnable drainAudio, boolean spoken, boolean interactive, boolean inputEchoed, + Supplier> onStarted, Mono opening, + Function> handleResponse, Predicate acceptClose) { + return Mono.defer(() -> { + GptLiveSampleUtils.AudioTranscriptPrinter transcripts = new GptLiveSampleUtils.AudioTranscriptPrinter(inputEchoed, interactive); + GptLiveSampleUtils.Greeting greeting = new GptLiveSampleUtils.Greeting(); + Sinks.Empty ready = Sinks.empty(); + Sinks.Empty closed = Sinks.empty(); + Sinks.Empty audioStarted = Sinks.empty(); + Sinks.Empty stopAudio = Sinks.empty(); + Sinks.Empty audioStopped = Sinks.empty(); + AtomicReference> responseCompleted = new AtomicReference<>(); + GptLiveSampleUtils.ConversationEventHandler> handler + = new GptLiveSampleUtils.ConversationEventHandler>(spoken, closeDetails, acceptClose) { + @Override + Mono onStarted() { + return Mono.defer(onStarted).doOnSuccess(ignored -> ready.tryEmitEmpty()); + } + + @Override + Mono onResponse(ResponseStreamEvent event) { + return Mono.defer(() -> handleResponse.apply(event)).defaultIfEmpty(false) + .doOnNext(continuation -> { + if (event.isCompleted() && completeResponse(continuation)) { + responseCompleted.getAndSet(null).tryEmitEmpty(); + } + }).then(); + } + + @Override + Mono onClosed() { + return Mono.fromRunnable(() -> closed.tryEmitEmpty()); + } + + @Override + Mono ignored() { + return Mono.empty(); + } + }; + Mono receive = events + .publishOn(Schedulers.boundedElastic(), 1) + .takeUntil(ServerEvent::isSessionClosed) + .concatMap(event -> Mono.defer(() -> { + if (spoken) { + transcripts.print(event); + if (event.isSessionOutputAudioDelta()) { + byte[] pcm = GptLiveSampleUtils.decodeAudio(event); + playAudio.accept(pcm); + greeting.onAudio(pcm); + } else if (event.isSessionOutputTranscriptDelta() + && !event.asSessionOutputTranscriptDelta().delta().trim().isEmpty()) { + greeting.onTranscript(); + } + } else { + GptLiveSampleUtils.printEvent(event); + } + return handler.process(event); + }), 1) + .then(Mono.defer(() -> handler.awaitConversationAfterReceive() ? Mono.never() : Mono.empty())); + Mono audio = ready.asMono() + .thenMany(GptLiveSampleUtils.silenceFramesAsync().takeUntilOther(stopAudio.asMono()) + .concatMap(frame -> sendEvent.apply(frame).doOnSuccess(ignored -> audioStarted.tryEmitEmpty()), 1)) + .then() + .doOnSuccess(ignored -> audioStopped.tryEmitEmpty()) + .then(Mono.never()); + Mono renderTranscripts = greeting.quietPeriodsAsync() + .doOnNext(ignored -> transcripts.endUtterance()).then(); + Mono conversation = ready.asMono().timeout(GptLiveSampleUtils.TIMEOUT) + .then(Mono.defer(() -> interactive ? audioStarted.asMono().timeout(GptLiveSampleUtils.TIMEOUT) : Mono.empty())) + .then(Mono.defer(() -> interactive + ? sendEvent.apply(GptLiveSampleUtils.commentary("Hi, how can I help you today?")) + .timeout(GptLiveSampleUtils.TIMEOUT) + .then(greeting.awaitPlaybackAsync(drainAudio)) + .then(Mono.fromRunnable(transcripts::endUtterance)) + : opening)) + .then(Mono.defer(() -> GptLiveSampleUtils.promptAsync(sendEvent, nextPrompt, responseCompleted, spoken, transcripts, handler))) + .then(Mono.defer(() -> { + handler.requestClose(); + stopAudio.tryEmitEmpty(); + return (interactive ? audioStopped.asMono() : Mono.empty()) + .then(sendEvent.apply(GptLiveSampleUtils.closeSession())) + .then(closed.asMono()).timeout(GptLiveSampleUtils.TIMEOUT); + })); + return interactive ? Mono.firstWithSignal(receive, audio, renderTranscripts, conversation) + : Mono.firstWithSignal(receive, conversation); + }); + } + +} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveFunctionToolSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveFunctionToolSample.java new file mode 100644 index 0000000000000..beb84b53d35f2 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveFunctionToolSample.java @@ -0,0 +1,201 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.BetaVoiceAgentsLiveWebSocketSessionClient; +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.core.util.BinaryData; +import com.azure.core.util.Configuration; +import com.azure.identity.DefaultAzureCredentialBuilder; +import com.openai.core.JsonValue; +import com.openai.models.live.ClientEvent; +import com.openai.models.live.FunctionTool; +import com.openai.models.live.ResponseItemCreateEvent; +import com.openai.models.live.ServerEvent; +import com.openai.models.responses.ResponseInputItem; +import com.openai.models.responses.ResponseFunctionToolCall; +import com.openai.models.responses.ResponseStreamEvent; +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Iterator; +import java.util.List; +import java.util.function.Function; + +/** + * Executes a sample weather function requested by the GPT Live Responses backend. + * Weather results are explicitly simulated, not a call to a weather service. + * Uses response.event envelopes and native response.item.create function_call_output items. + * This is Responses delegation, not Foundry-managed-agent tool-result routing. + * The uniquely named agent is deleted unless FOUNDRY_KEEP_GPT_LIVE_AGENT=true. + * + *

Before running the sample, set these environment variables:

+ *
    + *
  • {@code FOUNDRY_PROJECT_ENDPOINT} - The Azure AI Project endpoint.
    • + *
  • {@code FOUNDRY_GPT_LIVE_MODEL} - Optional. The GPT Live deployment name. Defaults to {@code gpt-live-1}.
    • + *
  • {@code FOUNDRY_GPT_LIVE_BACKEND_MODEL} - Optional. The Responses backend deployment name in the Live + * resource. Defaults to {@code gpt-5.6-terra}.
    • + *
  • {@code FOUNDRY_KEEP_GPT_LIVE_AGENT} - Optional. Set to {@code true} to keep the agent after the sample. + * Defaults to {@code false}.
    • + *
+ */ +public class GptLiveFunctionToolSample { + public static void main(String[] args) { + Configuration configuration = Configuration.getGlobalConfiguration(); + String endpoint = configuration.get("FOUNDRY_PROJECT_ENDPOINT"); + String model = configuration.get("FOUNDRY_GPT_LIVE_MODEL"); + String backendModel = configuration.get("FOUNDRY_GPT_LIVE_BACKEND_MODEL"); + boolean keepAgent = Boolean.parseBoolean(configuration.get("FOUNDRY_KEEP_GPT_LIVE_AGENT", "false")); + + if (endpoint == null || endpoint.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_PROJECT_ENDPOINT before running this sample."); + } + if (model == null) { + model = "gpt-live-1"; + } else if (model.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_GPT_LIVE_MODEL before running this sample."); + } + if (backendModel == null) { + backendModel = "gpt-5.6-terra"; + } else if (backendModel.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_GPT_LIVE_BACKEND_MODEL before running this sample."); + } + + AgentsClientBuilder builder = new AgentsClientBuilder().endpoint(endpoint) + .credential(new DefaultAzureCredentialBuilder().build()) + .allowPreview(true); + try (GptLiveSampleUtils.AgentResource agent + = GptLiveSampleUtils.createAgent(builder, model, backendModel, "tools", true, keepAgent); + BetaVoiceAgentsLiveWebSocketSessionClient session = agent.builder.beta() + .buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession(agent.name)) { + runConversation(session, + Collections.singletonList(args.length == 0 ? "What is the weather in Seattle?" : args[0]), + GptLiveFunctionToolSample::toolResult); + } + } + + static void runConversation(BetaVoiceAgentsLiveWebSocketSessionClient session, List prompts, + Function toolHandler) { + Iterator pending = prompts.iterator(); + List results = new ArrayList<>(); + boolean started = false; + boolean awaitingResponse = false; + boolean closing = false; + int continuations = 0; + for (ServerEvent event : session.receiveEvents(GptLiveSampleUtils.TIMEOUT)) { + GptLiveSampleUtils.printEvent(event); + boolean nextTurn = false; + if (event.isSessionStarted()) { + if (started) { + throw new IllegalStateException("Duplicate session.started event."); + } + started = true; + nextTurn = true; + } else if (event.isResponseEvent()) { + ResponseStreamEvent response = GptLiveSampleUtils.backendEvent(event); + if (response.isOutputItemDone() && response.asOutputItemDone().item().isFunctionCall()) { + ResponseFunctionToolCall call = response.asOutputItemDone().item().asFunctionCall(); + results.add(toolHandler.apply(call)); + } else if (response.isCompleted()) { + if (!started || !awaitingResponse || closing) { + throw new IllegalStateException("Unexpected GPT Live response lifecycle event."); + } + if (!results.isEmpty()) { + if (++continuations > 8) { + throw new IllegalStateException("Sample tool continuation limit exceeded."); + } + for (ClientEvent result : results) { + session.sendEvent(result); + } + results.clear(); + session.sendEvent(GptLiveSampleUtils.responseCreate()); + } else { + continuations = 0; + awaitingResponse = false; + System.out.println(); + nextTurn = true; + } + } + } else if (event.isSessionClosed()) { + String reason = event.asSessionClosed().reason().toString(); + if (!(started && "remote_hangup".equals(reason)) + && !(closing && "close_requested".equals(reason))) { + throw new IllegalStateException("GPT Live closed before all turns completed. Event: " + event + + "; transport close code=" + session.getCloseCode() + ", reason=" + session.getCloseReason()); + } + System.out.println("Live session closed (" + reason + ")."); + return; + } + if (nextTurn) { + if (pending.hasNext()) { + String prompt = pending.next(); + System.out.println("You: " + prompt); + System.out.print("Assistant: "); + awaitingResponse = true; + session.sendEvent(GptLiveSampleUtils.userMessage(prompt)); + session.sendEvent(GptLiveSampleUtils.responseCreate()); + } else { + closing = true; + session.sendEvent(GptLiveSampleUtils.closeSession()); + } + } + } + throw new IllegalStateException("GPT Live disconnected without session.closed."); + } + + static FunctionTool weatherTool() { + FunctionTool.Parameters schema = FunctionTool.Parameters.builder() + .putAdditionalProperty("type", JsonValue.from("object")) + .putAdditionalProperty("properties", JsonValue.from( + Collections.singletonMap("city", Collections.singletonMap("type", "string")))) + .putAdditionalProperty("required", JsonValue.from(Collections.singletonList("city"))) + .putAdditionalProperty("additionalProperties", JsonValue.from(false)) + .build(); + return FunctionTool.builder().name("get_weather") + .description("Get simulated weather for a city. This sample returns demonstration data only.") + .parameters(schema) + .strict(true) + .build(); + } + + static ClientEvent toolResult(ResponseFunctionToolCall call) { + if (!"get_weather".equals(call.name())) { + throw new IllegalArgumentException("Unsupported sample tool: " + call.name()); + } + Map arguments = object( + BinaryData.fromString(call.arguments())); + Map result = new LinkedHashMap<>(); + result.put("city", string(arguments, "city")); + result.put("condition", "sunny"); + result.put("temperature_f", 72); + result.put("simulated", true); + return ClientEvent.ofResponseItemCreate(ResponseItemCreateEvent.builder() + .item(ResponseInputItem.FunctionCallOutput.builder() + .callId(call.callId()) + .output(BinaryData.fromObject(result).toString()) + .build()) + .build()); + } + + static Map object(BinaryData data) { + return data.toObject(Map.class); + } + + static Map object(Object value) { + if (!(value instanceof Map)) { + throw new IllegalArgumentException("Expected a native Live JSON object: " + value); + } + return (Map) value; + } + + static String string(Map event, String field) { + Object value = event.get(field); + if (!(value instanceof String)) { + throw new IllegalArgumentException("Missing or invalid Live event field: " + field); + } + return (String) value; + } + +} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAsyncSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAsyncSample.java new file mode 100644 index 0000000000000..b5c267619953c --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAsyncSample.java @@ -0,0 +1,84 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.ai.agents.BetaVoiceAgentsLiveConversationsAsyncClient; +import com.azure.ai.agents.models.PageOrder; +import com.azure.ai.agents.models.VoiceAgentLiveConversation; +import com.azure.ai.agents.models.VoiceAgentLiveConversationItem; +import com.azure.core.util.Configuration; +import com.azure.identity.DefaultAzureCredentialBuilder; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +/** + * Asynchronously reads persisted GPT Live conversations and transcript items. + * The session must have used {@code store=true}. The agent and conversation must still exist. + * Like {@link GptLiveReadConversationSample}, this sample prints sensitive stored content without modifying data. + * Blocking occurs only at the standalone main method boundary. + * + *

Before running the sample, set these environment variables:

+ *
    + *
  • {@code FOUNDRY_PROJECT_ENDPOINT} - Required. The Azure AI Project endpoint. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_AGENT_NAME} - Required. The existing GPT Live agent's name. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_CONVERSATION_ID} - Required. The persisted Foundry conversation ID from + * {@code session.started.conversation_id}, not the native session ID. No default.
    • + *
+ * + *

Authenticates with {@code DefaultAzureCredential}. Run without command-line arguments. + * Model deployment settings are not needed because this sample reads an existing conversation.

+ */ +public class GptLiveReadConversationAsyncSample { + public static void main(String[] args) { + Configuration configuration = Configuration.getGlobalConfiguration(); + String endpoint = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_PROJECT_ENDPOINT"); + String agentName = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_GPT_LIVE_AGENT_NAME"); + String conversationId + = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_GPT_LIVE_CONVERSATION_ID"); + BetaVoiceAgentsLiveConversationsAsyncClient client = new AgentsClientBuilder().endpoint(endpoint) + .credential(new DefaultAzureCredentialBuilder().build()) + .addPolicy(GptLiveReadConversationSample.previewHeaders()) + .beta().buildBetaVoiceAgentsLiveConversationsAsyncClient(); + readConversation(client, agentName, conversationId).block(); + } + + static Mono readConversation(BetaVoiceAgentsLiveConversationsAsyncClient client, String agentName, + String conversationId) { + return listConversations(client, agentName, null) + .doOnNext(conversation -> System.out.printf("Available conversation: %s (%s)%n", + conversation.getId(), conversation.getStatus())) + .then(Mono.defer(() -> client.getAgentConversation(agentName, conversationId) + .doOnNext(GptLiveReadConversationSample::printConversation))) + .thenMany(listItems(client, agentName, conversationId, null)) + .concatMap(item -> client.getAgentConversationItem(agentName, conversationId, item.getId()), 0) + .doOnNext(GptLiveReadConversationSample::printItem) + .then(); + } + + private static Flux listConversations(BetaVoiceAgentsLiveConversationsAsyncClient client, + String agentName, String after) { + return client.listAgentConversations(agentName, 100, PageOrder.ASC, after, null).collectList() + .flatMapMany(page -> { + if (page.isEmpty()) { + return Flux.empty(); + } + String next = GptLiveReadConversationSample.nextCursor(page.get(page.size() - 1).getId(), after); + return Flux.fromIterable(page).concatWith(Flux.defer(() -> listConversations(client, agentName, next))); + }); + } + + private static Flux listItems(BetaVoiceAgentsLiveConversationsAsyncClient client, + String agentName, String conversationId, String after) { + return client.listAgentConversationItems(agentName, conversationId, 100, PageOrder.ASC, after, null).collectList() + .flatMapMany(page -> { + if (page.isEmpty()) { + return Flux.empty(); + } + String next = GptLiveReadConversationSample.nextCursor(page.get(page.size() - 1).getId(), after); + return Flux.fromIterable(page) + .concatWith(Flux.defer(() -> listItems(client, agentName, conversationId, next))); + }); + } +} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAudioAsyncSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAudioAsyncSample.java new file mode 100644 index 0000000000000..83325b3f132c9 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAudioAsyncSample.java @@ -0,0 +1,91 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.ai.agents.BetaVoiceAgentsLiveConversationsAsyncClient; +import com.azure.core.util.BinaryData; +import com.azure.core.util.Configuration; +import com.azure.identity.DefaultAzureCredentialBuilder; +import java.nio.file.Path; +import reactor.core.publisher.Mono; +import reactor.core.scheduler.Schedulers; + +/** + * Asynchronously downloads persisted GPT Live merged audio and optionally a speech item's WAV to new files. + * The session must have used {@code store=true}. The agent and conversation must still exist. + * Wait for recording finalization after the session ends. Like {@link GptLiveReadConversationAudioSample}, + * HTTP errors propagate and BYOS audio requires separate Azure Storage credentials. + * File writes run on boundedElastic, not an HTTP I/O thread. + * Blocking occurs only at the standalone main method boundary. Global Reactor schedulers are not shut down. + * No agent or persisted data is modified or deleted. + * + *

Before running the sample, set these environment variables:

+ *
    + *
  • {@code FOUNDRY_PROJECT_ENDPOINT} - Required. The Azure AI Project endpoint. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_AGENT_NAME} - Required. The existing GPT Live agent's name. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_CONVERSATION_ID} - Required. The persisted Foundry conversation ID from + * {@code session.started.conversation_id}, not the native session ID. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_ITEM_ID} - Optional. A speech item's ID, not a delegation item. + * Defaults to unset, downloading only the merged conversation audio. Explicitly blank values are invalid.
    • + *
+ * + *

To download an individual speech item's audio:

+ *
    + *
  1. Run {@link GptLiveReadConversationSample} with the same project endpoint, agent name, and conversation ID.
    2. + *
  2. Find the desired transcript in its output and copy the ID from the preceding + * {@code Item : type=input} or {@code Item : type=output} line. Do not select a delegation item.
    3. + *
  3. Set {@code FOUNDRY_GPT_LIVE_ITEM_ID} to that ID, then run this sample with an existing output directory + * as its sole command-line argument. It downloads both the merged call audio and the selected item's WAV.
    4. + *
+ *

For merged call audio only, skip the item lookup and leave {@code FOUNDRY_GPT_LIVE_ITEM_ID} unset.

+ * + *

Authenticates with {@code DefaultAzureCredential}. Pass one existing output directory as the sole + * command-line argument; there is no default directory. Treat downloaded audio as sensitive data. + * Model deployment settings are not needed because this sample reads an existing conversation.

+ */ +public class GptLiveReadConversationAudioAsyncSample { + public static void main(String[] args) { + Path directory = GptLiveReadConversationAudioSample.outputDirectory(args); + Configuration configuration = Configuration.getGlobalConfiguration(); + String endpoint = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_PROJECT_ENDPOINT"); + String agentName = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_GPT_LIVE_AGENT_NAME"); + String conversationId + = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_GPT_LIVE_CONVERSATION_ID"); + String itemId = configuration.get("FOUNDRY_GPT_LIVE_ITEM_ID"); + BetaVoiceAgentsLiveConversationsAsyncClient client = new AgentsClientBuilder().endpoint(endpoint) + .credential(new DefaultAzureCredentialBuilder().build()) + .addPolicy(GptLiveReadConversationSample.previewHeaders()) + .beta().buildBetaVoiceAgentsLiveConversationsAsyncClient(); + downloadAudio(client, agentName, conversationId, itemId, directory).block(); + } + + static Mono downloadAudio(BetaVoiceAgentsLiveConversationsAsyncClient client, String agentName, + String conversationId, String itemId, Path directory) { + if (itemId != null && itemId.trim().isEmpty()) { + return Mono.error(new IllegalArgumentException("FOUNDRY_GPT_LIVE_ITEM_ID must not be blank when set.")); + } + Mono merged = client.getAgentConversationAudio(agentName, conversationId).flatMap(recording -> { + GptLiveReadConversationAudioSample.printRecording(recording); + return GptLiveReadConversationAudioSample.isFoundryManaged(recording.getBlobUri()) + ? client.downloadAgentConversationAudio(agentName, conversationId) + .flatMap(audio -> saveAudio(audio, directory, "gpt-live-call-")) + : Mono.empty(); + }); + return merged.then(Mono.defer(() -> itemId == null ? Mono.empty() + : client.getAgentConversationAudioItem(agentName, conversationId, itemId).flatMap(item -> { + System.out.printf("Item %s: format=%s, rate=%d%n", itemId, item.getFormat(), item.getSampleRate()); + return GptLiveReadConversationAudioSample.isFoundryManaged(item.getBlobUri()) + ? client.downloadAgentConversationAudioItem(agentName, conversationId, itemId) + .flatMap(audio -> saveAudio(audio, directory, "gpt-live-item-")) + : Mono.empty(); + }))); + } + + private static Mono saveAudio(BinaryData audio, Path directory, String prefix) { + return Mono.fromCallable(() -> GptLiveReadConversationAudioSample.saveAudio(audio, directory, prefix)) + .subscribeOn(Schedulers.boundedElastic()) + .then(); + } +} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAudioSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAudioSample.java new file mode 100644 index 0000000000000..8a28f0dc81b38 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationAudioSample.java @@ -0,0 +1,122 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.ai.agents.BetaVoiceAgentsLiveConversationsClient; +import com.azure.ai.agents.models.VoiceAudioItem; +import com.azure.ai.agents.models.VoiceRecording; +import com.azure.core.util.BinaryData; +import com.azure.core.util.Configuration; +import com.azure.identity.DefaultAzureCredentialBuilder; +import java.io.IOException; +import java.io.OutputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +/** + * Downloads persisted GPT Live merged audio and optionally a speech item's WAV to new files. + * The session must have used {@code store=true}. The agent and conversation must still exist. + * Wait for recording finalization after the session ends. HTTP errors propagate: inspect Retry-After on + * recording_not_ready (409), and distinguish missing audio (404) from recording_unavailable (409). + * BYOS audio requires separate Azure Storage credentials; this sample does not download it. + * No agent or persisted data is modified or deleted. + * + *

Before running the sample, set these environment variables:

+ *
    + *
  • {@code FOUNDRY_PROJECT_ENDPOINT} - Required. The Azure AI Project endpoint. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_AGENT_NAME} - Required. The existing GPT Live agent's name. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_CONVERSATION_ID} - Required. The persisted Foundry conversation ID from + * {@code session.started.conversation_id}, not the native session ID. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_ITEM_ID} - Optional. A speech item's ID, not a delegation item. + * Defaults to unset, downloading only the merged conversation audio. Explicitly blank values are invalid.
    • + *
+ * + *

To download an individual speech item's audio:

+ *
    + *
  1. Run {@link GptLiveReadConversationSample} with the same project endpoint, agent name, and conversation ID.
    2. + *
  2. Find the desired transcript in its output and copy the ID from the preceding + * {@code Item : type=input} or {@code Item : type=output} line. Do not select a delegation item.
    3. + *
  3. Set {@code FOUNDRY_GPT_LIVE_ITEM_ID} to that ID, then run this sample with an existing output directory + * as its sole command-line argument. It downloads both the merged call audio and the selected item's WAV.
    4. + *
+ *

For merged call audio only, skip the item lookup and leave {@code FOUNDRY_GPT_LIVE_ITEM_ID} unset.

+ * + *

Authenticates with {@code DefaultAzureCredential}. Pass one existing output directory as the sole + * command-line argument; there is no default directory. Treat downloaded audio as sensitive data. + * Model deployment settings are not needed because this sample reads an existing conversation.

+ */ +public class GptLiveReadConversationAudioSample { + public static void main(String[] args) throws IOException { + Path directory = outputDirectory(args); + Configuration configuration = Configuration.getGlobalConfiguration(); + String endpoint = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_PROJECT_ENDPOINT"); + String agentName = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_GPT_LIVE_AGENT_NAME"); + String conversationId + = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_GPT_LIVE_CONVERSATION_ID"); + String itemId = configuration.get("FOUNDRY_GPT_LIVE_ITEM_ID"); + BetaVoiceAgentsLiveConversationsClient client = new AgentsClientBuilder().endpoint(endpoint) + .credential(new DefaultAzureCredentialBuilder().build()) + .addPolicy(GptLiveReadConversationSample.previewHeaders()) + .beta().buildBetaVoiceAgentsLiveConversationsClient(); + downloadAudio(client, agentName, conversationId, itemId, directory); + } + + static void downloadAudio(BetaVoiceAgentsLiveConversationsClient client, String agentName, + String conversationId, String itemId, Path directory) throws IOException { + if (itemId != null && itemId.trim().isEmpty()) { + throw new IllegalArgumentException("FOUNDRY_GPT_LIVE_ITEM_ID must not be blank when set."); + } + VoiceRecording recording = client.getAgentConversationAudio(agentName, conversationId); + printRecording(recording); + if (isFoundryManaged(recording.getBlobUri())) { + saveAudio(client.downloadAgentConversationAudio(agentName, conversationId), directory, "gpt-live-call-"); + } + if (itemId != null) { + VoiceAudioItem item = client.getAgentConversationAudioItem(agentName, conversationId, itemId); + System.out.printf("Item %s: format=%s, rate=%d%n", itemId, item.getFormat(), item.getSampleRate()); + if (isFoundryManaged(item.getBlobUri())) { + saveAudio(client.downloadAgentConversationAudioItem(agentName, conversationId, itemId), + directory, "gpt-live-item-"); + } + } + } + + static void printRecording(VoiceRecording recording) { + System.out.printf("Recording: format=%s, rate=%d, channels=%d, durationMs=%s%n", + recording.getFormat(), recording.getSampleRate(), recording.getChannels(), recording.getDurationMs()); + } + + static boolean isFoundryManaged(String blobUri) { + if (blobUri == null || blobUri.isEmpty()) { + return true; + } + System.out.println("BYOS audio: use metadata.getBlobUri() with your own Azure Storage credentials."); + return false; + } + + static Path saveAudio(BinaryData audio, Path directory, String prefix) throws IOException { + Path output = Files.createTempFile(directory, prefix, ".wav"); + try (OutputStream stream = Files.newOutputStream(output)) { + audio.writeTo(stream); + } catch (IOException | RuntimeException error) { + try { + Files.deleteIfExists(output); + } catch (IOException cleanupError) { + error.addSuppressed(cleanupError); + } + throw error; + } + System.out.println("Saved WAV: " + output.toAbsolutePath()); + return output; + } + + static Path outputDirectory(String[] args) { + if (args.length != 1 || !Files.isDirectory(Paths.get(args[0]))) { + throw new IllegalArgumentException("Pass one existing output directory as the command-line argument."); + } + return Paths.get(args[0]); + } +} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationSample.java new file mode 100644 index 0000000000000..00b5260e64b0d --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveReadConversationSample.java @@ -0,0 +1,113 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.ai.agents.BetaVoiceAgentsLiveConversationsClient; +import com.azure.ai.agents.models.PageOrder; +import com.azure.ai.agents.models.VoiceAgentLiveConversation; +import com.azure.ai.agents.models.VoiceAgentLiveConversationItem; +import com.azure.ai.agents.models.VoiceAgentLiveDelegationConversationItem; +import com.azure.ai.agents.models.VoiceAgentLiveInputConversationItem; +import com.azure.ai.agents.models.VoiceAgentLiveOutputConversationItem; +import com.azure.core.http.HttpHeaders; +import com.azure.core.http.policy.AddHeadersPolicy; +import com.azure.core.util.Configuration; +import com.azure.identity.DefaultAzureCredentialBuilder; +import java.util.List; +import java.util.stream.Collectors; + +/** + * Lists persisted GPT Live conversations and reads one conversation's status, usage, and transcript items. + * The session must have used {@code store=true}. Use {@code session.started.conversation_id}, not the native session ID. + * Stored speech segments are not native turns or proof of playback; delegation results are not spoken transcripts. + * This sample prints sensitive stored content. It does not create sessions, modify agents, or delete data. + * + *

Before running the sample, set these environment variables:

+ *
    + *
  • {@code FOUNDRY_PROJECT_ENDPOINT} - Required. The Azure AI Project endpoint. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_AGENT_NAME} - Required. The existing GPT Live agent's name. No default.
    • + *
  • {@code FOUNDRY_GPT_LIVE_CONVERSATION_ID} - Required. The persisted Foundry conversation ID from + * {@code session.started.conversation_id}. No default.
    • + *
+ * + *

Authenticates with {@code DefaultAzureCredential}. Run without command-line arguments. + * Model deployment settings are not needed because this sample reads an existing conversation.

+ */ +public class GptLiveReadConversationSample { + public static void main(String[] args) { + Configuration configuration = Configuration.getGlobalConfiguration(); + String endpoint = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_PROJECT_ENDPOINT"); + String agentName = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_GPT_LIVE_AGENT_NAME"); + String conversationId = GptLiveSampleUtils.requireSetting(configuration, "FOUNDRY_GPT_LIVE_CONVERSATION_ID"); + BetaVoiceAgentsLiveConversationsClient client = new AgentsClientBuilder().endpoint(endpoint) + .credential(new DefaultAzureCredentialBuilder().build()) + .addPolicy(previewHeaders()) + .beta() + .buildBetaVoiceAgentsLiveConversationsClient(); + readConversation(client, agentName, conversationId); + } + + static AddHeadersPolicy previewHeaders() { + // The generated Live REST builder methods do not automatically opt in to the voice preview. + return new AddHeadersPolicy(new HttpHeaders().set("Foundry-Features", "VoiceAgents=V1Preview")); + } + + static void readConversation(BetaVoiceAgentsLiveConversationsClient client, String agentName, + String conversationId) { + // Generated paging exposes cursor parameters but no automatic continuation token. + String after = null; + while (true) { + List page = client + .listAgentConversations(agentName, 100, PageOrder.ASC, after, null).stream().collect(Collectors.toList()); + if (page.isEmpty()) { + break; + } + for (VoiceAgentLiveConversation conversation : page) { + System.out.printf("Available conversation: %s (%s)%n", conversation.getId(), conversation.getStatus()); + } + after = nextCursor(page.get(page.size() - 1).getId(), after); + } + printConversation(client.getAgentConversation(agentName, conversationId)); + after = null; + while (true) { + List page = client + .listAgentConversationItems(agentName, conversationId, 100, PageOrder.ASC, after, null) + .stream().collect(Collectors.toList()); + if (page.isEmpty()) { + break; + } + for (VoiceAgentLiveConversationItem item : page) { + printItem(client.getAgentConversationItem(agentName, conversationId, item.getId())); + } + after = nextCursor(page.get(page.size() - 1).getId(), after); + } + } + + static String nextCursor(String id, String previous) { + if (id == null || id.isEmpty() || id.equals(previous)) { + throw new IllegalStateException("The service returned a missing or non-advancing pagination cursor."); + } + return id; + } + + static void printConversation(VoiceAgentLiveConversation conversation) { + System.out.printf("Conversation %s: status=%s, created=%s, completed=%s, usageSeconds=%s%n", + conversation.getId(), conversation.getStatus(), conversation.getCreatedAt(), conversation.getCompletedAt(), + conversation.getUsage() == null ? null : conversation.getUsage().seconds()); + } + + static void printItem(VoiceAgentLiveConversationItem item) { + System.out.printf("Item %s: type=%s%n", item.getId(), item.getType()); + if (item instanceof VoiceAgentLiveInputConversationItem) { + System.out.println("User transcript: " + ((VoiceAgentLiveInputConversationItem) item).getTranscription()); + } else if (item instanceof VoiceAgentLiveOutputConversationItem) { + System.out.println("Assistant transcript: " + ((VoiceAgentLiveOutputConversationItem) item).getTranscription()); + } else if (item instanceof VoiceAgentLiveDelegationConversationItem) { + VoiceAgentLiveDelegationConversationItem delegation = (VoiceAgentLiveDelegationConversationItem) item; + System.out.printf("Delegation target=%s, response=%s%n", delegation.getTarget(), delegation.getResponseId()); + } + } + +} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveSampleUtils.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveSampleUtils.java new file mode 100644 index 0000000000000..41697c5bd6af7 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveSampleUtils.java @@ -0,0 +1,696 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.AgentsClient; +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.ai.agents.models.CreateAgentVersionInput; +import com.azure.ai.agents.models.VoiceAgentLiveDefinition; +import com.azure.ai.agents.models.VoiceAgentLiveResponsesDelegation; +import com.azure.core.util.BinaryData; +import com.azure.core.util.Configuration; +import com.openai.core.JsonValue; +import com.openai.models.live.ClientEvent; +import com.openai.models.live.CommentaryAppendEvent; +import com.openai.models.live.InputAudioAppendEvent; +import com.openai.models.live.ResponseCreateEvent; +import com.openai.models.live.ResponseItemCreateEvent; +import com.openai.models.live.ResponsesDelegationConfig; +import com.openai.models.live.ServerEvent; +import com.openai.models.live.SessionCloseEvent; +import com.openai.models.live.SessionConfig; +import com.openai.models.responses.EasyInputMessage; +import com.openai.models.responses.ResponseInputContent; +import com.openai.models.responses.ResponseInputText; +import com.openai.models.responses.ResponseStreamEvent; +import java.io.BufferedReader; +import java.io.IOException; +import java.time.Duration; +import java.util.Arrays; +import java.util.Base64; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.UUID; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Function; +import java.util.function.Predicate; +import java.util.function.Supplier; +import javax.sound.sampled.AudioFormat; +import javax.sound.sampled.AudioSystem; +import javax.sound.sampled.LineUnavailableException; +import javax.sound.sampled.SourceDataLine; +import javax.sound.sampled.TargetDataLine; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; +import reactor.core.publisher.Sinks; +import reactor.core.scheduler.Schedulers; + +final class GptLiveSampleUtils { + static final Duration TIMEOUT = Duration.ofSeconds(60); + + private GptLiveSampleUtils() { + } + + static String requireSetting(Configuration configuration, String name) { + String value = configuration.get(name); + if (value == null || value.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable " + name + " before running this sample."); + } + return value; + } + + static String readPromptBlocking(BufferedReader reader, AudioTranscriptPrinter transcripts) throws IOException { + transcripts.beginInput(); + try { + String line = reader.readLine(); + if (line == null) { + return null; + } + String text = line.trim(); + return text.isEmpty() || "exit".equalsIgnoreCase(text) || "quit".equalsIgnoreCase(text) ? null : text; + } finally { + transcripts.endInput(); + } + } + + static Mono promptAsync(Function> sendEvent, + Function> nextPrompt, + AtomicReference> responseCompleted, boolean spoken, + AudioTranscriptPrinter transcripts, ConversationEventHandler handler) { + return nextPrompt.apply(transcripts).flatMap(text -> { + Sinks.Empty completion = Sinks.empty(); + responseCompleted.set(completion); + handler.beginResponse(); + if (!spoken) { + System.out.print("Assistant: "); + System.out.flush(); + } + return sendEvent.apply(userMessage(text)) + .then(sendEvent.apply(responseCreate())) + .then(completion.asMono()).timeout(TIMEOUT) + .thenReturn(true); + }).defaultIfEmpty(false) + .flatMap(keepGoing -> keepGoing + ? Mono.defer(() -> promptAsync(sendEvent, nextPrompt, responseCompleted, spoken, transcripts, handler)) + : Mono.empty()); + } + + static Flux silenceFramesAsync() { + // Twenty milliseconds of 24-kHz mono PCM16LE. + ClientEvent silence = audio(new byte[960]); + return Mono.just(silence) + .concatWith(Mono.just(silence).delayElement(Duration.ofMillis(20)).repeat()); + } + + static final class Greeting { + // PCM16 peak levels up to 256 (about -42 dBFS) are treated as silence for this heuristic. + static final int SILENCE_PEAK_THRESHOLD = 256; + private final Sinks.Many progress = Sinks.many().replay().latest(); + private int received; + + void onAudio(byte[] pcm) { + if (hasSpeech(pcm)) { + received |= 1; + progress.tryEmitNext(received); + } + } + + void onTranscript() { + received |= 2; + progress.tryEmitNext(received); + } + + Mono awaitPlaybackAsync(Runnable drainAudio) { + // Output can continue with silent PCM after speaking; only speech activity restarts this timer. + return quietPeriodsAsync().next() + .timeout(TIMEOUT) + .then(Mono.fromRunnable(drainAudio).subscribeOn(Schedulers.boundedElastic())); + } + + Flux quietPeriodsAsync() { + return progress.asFlux().filter(parts -> parts == 3) + .switchMap(parts -> Mono.delay(Duration.ofSeconds(1))); + } + } + + abstract static class ConversationEventHandler { + private final boolean spoken; + private final Supplier closeDetails; + private final Predicate acceptClose; + private volatile boolean started; + private volatile boolean awaitingResponse; + private volatile boolean closing; + private volatile boolean finalized; + private volatile boolean remoteHangup; + + ConversationEventHandler(boolean spoken, Supplier closeDetails, Predicate acceptClose) { + this.spoken = spoken; + this.closeDetails = closeDetails; + this.acceptClose = acceptClose; + } + + final boolean canStreamAudio() { + return spoken && started && !closing; + } + + final boolean canReadPrompt(boolean greetingComplete) { + return started && greetingComplete && !awaitingResponse && !closing; + } + + final void beginResponse() { + awaitingResponse = true; + } + + final void requestClose() { + closing = true; + } + + final boolean awaitConversationAfterReceive() { + if (!finalized) { + throw new IllegalStateException("GPT Live disconnected without session.closed."); + } + return !remoteHangup; + } + + final T process(ServerEvent event) { + if (event.isSessionStarted()) { + if (started) { + throw new IllegalStateException("Duplicate session.started event."); + } + started = true; + return onStarted(); + } + ResponseStreamEvent backend = backendEvent(event); + if (backend != null) { + if (backend.isCompleted()) { + // Spoken greetings or unsolicited responses have no pending prompt, even during closure. + if (spoken && started && !awaitingResponse) { + return ignored(); + } + // Otherwise, completion requires a pending response in a started session that is not closing. + if (!started || !awaitingResponse || closing) { + throw new IllegalStateException("Unexpected GPT Live response lifecycle event."); + } + } + return onResponse(backend); + } + if (event.isSessionClosed()) { + String reason = event.asSessionClosed().reason().toString(); + boolean remote = started && "remote_hangup".equals(reason); + boolean accepted = acceptClose.test(reason); + // Allow a remote hangup after startup, a caller-accepted reason, or acknowledgment of our close request. + if (!remote && !accepted && !(closing && "close_requested".equals(reason))) { + throw new IllegalStateException("GPT Live closed before all turns completed. Event: " + event + + "; " + closeDetails.get()); + } + remoteHangup = remote || accepted; + finalized = true; + System.out.println("Live session closed (" + reason + ")."); + return onClosed(); + } + return ignored(); + } + + final boolean completeResponse(boolean continuation) { + if (continuation) { + return false; + } + awaitingResponse = false; + if (!spoken) { + System.out.println(); + } + return true; + } + + abstract T onStarted(); + + abstract T onResponse(ResponseStreamEvent event); + + abstract T onClosed(); + + abstract T ignored(); + } + + static byte[] decodeAudio(ServerEvent event) { + byte[] pcm = Base64.getDecoder().decode(event.asSessionOutputAudioDelta().delta()); + if (pcm.length % 2 != 0) { + throw new IllegalArgumentException("GPT Live PCM16 audio must contain complete two-byte samples."); + } + return pcm; + } + + static boolean hasSpeech(byte[] pcm) { + for (int offset = 0; offset < pcm.length; offset += 2) { + int sample = (short) ((pcm[offset] & 0xff) | (pcm[offset + 1] << 8)); + if (sample < -Greeting.SILENCE_PEAK_THRESHOLD || sample > Greeting.SILENCE_PEAK_THRESHOLD) { + return true; + } + } + return false; + } + + static BinaryData event(String type, String field, Object value) { + Map event = new LinkedHashMap<>(); + event.put("type", type); + if (field != null) { + event.put(field, value); + } + return BinaryData.fromObject(event); + } + + static ClientEvent userMessage(String text) { + return ClientEvent.ofResponseItemCreate(ResponseItemCreateEvent.builder() + .item(EasyInputMessage.builder() + .role(EasyInputMessage.Role.USER) + .type(EasyInputMessage.Type.MESSAGE) + .contentOfResponseInputMessageContentList(java.util.Collections.singletonList( + ResponseInputContent.ofInputText(ResponseInputText.builder().text(text).build()))) + .build()) + .build()); + } + + static ClientEvent audio(byte[] pcm) { + return ClientEvent.ofSessionInputAudioAppend(InputAudioAppendEvent.builder() + .audio(Base64.getEncoder().encodeToString(pcm)).build()); + } + + static ClientEvent commentary(String content) { + return ClientEvent.ofSessionCommentaryAppend(CommentaryAppendEvent.builder() + .content(content) + .delegationId(java.util.Optional.empty()) + .build()); + } + + static ClientEvent responseCreate() { + return ClientEvent.ofResponseCreate(ResponseCreateEvent.builder().build()); + } + + static ClientEvent closeSession() { + return ClientEvent.ofSessionClose(SessionCloseEvent.builder().build()); + } + + static ResponseStreamEvent backendEvent(ServerEvent event) { + return event.isResponseEvent() + ? JsonValue.from(event.asResponseEvent().event()._additionalProperties()).convert(ResponseStreamEvent.class) + : null; + } + + static void checkError(ServerEvent event) { + if (event.isErrorEvent()) { + com.openai.models.live.Error error = event.asErrorEvent().error(); + boolean moderated = "content_filter".equals(error._code().asString().orElse(null)) + || "content_filter".equals(error._type().asString().orElse(null)); + throw new IllegalStateException(moderated + ? "GPT Live service error: generation stopped by service moderation " + + "(content_filter). The call has ended; review the conversation and service diagnostics. Event: " + + event + : "GPT Live service error: " + event); + } + if (event.isResponseEvent()) { + checkError(backendEvent(event)); + } else if (event._json().isPresent()) { + checkError(event._json().get().convert(ResponseStreamEvent.class)); + } + } + + static void checkError(ResponseStreamEvent event) { + if (event.isError() || event.isFailed() || event.isIncomplete()) { + Object error = JsonValue.from(event).convert(Map.class).get("error"); + if (error instanceof Map && ("content_filter".equals(((Map) error).get("code")) + || "content_filter".equals(((Map) error).get("type")))) { + throw new IllegalStateException("GPT Live service error: generation stopped by service moderation " + + "(content_filter). The call has ended; review the conversation and service diagnostics. Event: " + + event); + } + throw new IllegalStateException("GPT Live service error: " + event); + } + } + + static void printEvent(ServerEvent event) { + checkError(event); + if (event.isSessionStarted()) { + JsonValue conversationId = event.asSessionStarted()._additionalProperties().get("conversation_id"); + System.out.println(conversationId == null + || !conversationId.asString().isPresent() + ? "Live session started; conversation_id not provided (persistence may be disabled)." + : "Live session started; conversation_id=" + conversationId.asString().get()); + } else if (event.isSessionInputTranscriptDelta()) { + System.out.print(event.asSessionInputTranscriptDelta().delta()); + } else if (event.isSessionOutputTranscriptDelta()) { + System.out.print(event.asSessionOutputTranscriptDelta().delta()); + } else if (event.isResponseEvent()) { + ResponseStreamEvent backend = backendEvent(event); + if (backend.isOutputTextDelta()) { + System.out.print(backend.asOutputTextDelta().delta()); + } + } + } + + static final class AudioProcessor implements AutoCloseable { + private static final Duration SEND_TIMEOUT = Duration.ofSeconds(10); + private static final int CHUNK_BYTES = 2400; + static final int MAX_PLAYBACK_BYTES = VoiceAgentRealtimeSampleUtils.SAMPLE_RATE * 2 * 60; + private static final byte[] STOP = new byte[0]; + private final Function> sendAudio; + private final AudioFormat format = new AudioFormat(VoiceAgentRealtimeSampleUtils.SAMPLE_RATE, 16, 1, true, false); + private final BlockingQueue playback = new LinkedBlockingQueue<>(MAX_PLAYBACK_BYTES / 2); + private int queuedPlaybackBytes; + private final AtomicBoolean running = new AtomicBoolean(); + private final Sinks.Empty failure = Sinks.empty(); + private boolean closed; + private TargetDataLine microphone; + private SourceDataLine speaker; + private Thread captureThread; + private Thread playbackThread; + + AudioProcessor(TargetDataLine microphone, SourceDataLine speaker, Function> sendAudio) { + this.sendAudio = sendAudio; + this.microphone = microphone; + this.speaker = speaker; + } + + Mono failureSignalAsync() { + return failure.asMono(); + } + + synchronized void start() { + if (closed) { + throw new IllegalStateException("Audio processor is already closed."); + } + try { + if (microphone == null) { + microphone = AudioSystem.getTargetDataLine(format); + } + microphone.open(format, CHUNK_BYTES * 4); + if (speaker == null) { + speaker = AudioSystem.getSourceDataLine(format); + } + speaker.open(format); + microphone.start(); + speaker.start(); + } catch (LineUnavailableException | IllegalArgumentException error) { + close(); + throw new IllegalStateException("A 24-kHz mono PCM16 microphone and speaker are required.", error); + } + + running.set(true); + captureThread = new Thread(this::capture, "voice-agent-microphone"); + playbackThread = new Thread(this::playback, "voice-agent-speaker"); + captureThread.setDaemon(true); + playbackThread.setDaemon(true); + captureThread.start(); + playbackThread.start(); + } + + private void capture() { + byte[] buffer = new byte[CHUNK_BYTES]; + try { + while (running.get()) { + int read = microphone.read(buffer, 0, buffer.length); + if (read > 0 && running.get()) { + sendAudio.apply(BinaryData.fromBytes(Arrays.copyOf(buffer, read))).block(SEND_TIMEOUT); + } + } + } catch (RuntimeException error) { + fail(error); + } + } + + private void playback() { + try { + while (running.get()) { + byte[] pcm = playback.take(); + if (pcm == STOP) { + break; + } + synchronized (playback) { + queuedPlaybackBytes -= pcm.length; + } + speaker.write(pcm, 0, pcm.length); + } + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + fail(error); + } catch (RuntimeException error) { + fail(error); + } + } + + void queueAudio(byte[] pcm) { + if (pcm == null || pcm.length == 0) { + return; + } + if (pcm.length % 2 != 0) { + fail(new IllegalArgumentException("PCM16 audio must contain complete two-byte samples.")); + return; + } + synchronized (playback) { + if (!running.get()) { + return; + } + if (pcm.length <= MAX_PLAYBACK_BYTES - queuedPlaybackBytes && playback.offer(pcm)) { + queuedPlaybackBytes += pcm.length; + return; + } + } + fail(new IllegalStateException("Audio playback backlog exceeded 60 seconds.")); + } + + private void fail(Throwable error) { + if (running.compareAndSet(true, false)) { + failure.tryEmitError(error); + } + } + + @Override + public synchronized void close() { + if (closed) { + return; + } + closed = true; + running.set(false); + clearPlayback(); + playback.offer(STOP); + try { + closeLine(microphone); + } finally { + try { + closeLine(speaker); + } finally { + if (captureThread != null) { + captureThread.interrupt(); + } + if (playbackThread != null) { + playbackThread.interrupt(); + } + join(captureThread); + join(playbackThread); + } + } + } + + private void clearPlayback() { + synchronized (playback) { + byte[] discarded; + while ((discarded = playback.poll()) != null) { + queuedPlaybackBytes -= discarded.length; + } + } + } + + private static void closeLine(javax.sound.sampled.DataLine line) { + if (line != null) { + try { + line.stop(); + } finally { + line.close(); + } + } + } + + private static void join(Thread thread) { + if (thread != null && thread != Thread.currentThread()) { + try { + thread.join(SEND_TIMEOUT.toMillis()); + if (thread.isAlive()) { + System.err.println("Audio thread did not stop: " + thread.getName()); + } + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + } + } + } + } + + static final class AudioTranscriptPrinter { + static final int MAX_BUFFERED_TRANSCRIPT_CHARACTERS = 65536; + private String speaker; + private boolean readingInput; + private boolean lineOpen; + private final boolean inputEchoed; + private final boolean completeUtterances; + private final StringBuilder pending = new StringBuilder(); + private final StringBuilder utterance = new StringBuilder(); + + AudioTranscriptPrinter() { + this(false); + } + + AudioTranscriptPrinter(boolean inputEchoed) { + this(inputEchoed, false); + } + + AudioTranscriptPrinter(boolean inputEchoed, boolean completeUtterances) { + this.inputEchoed = inputEchoed; + this.completeUtterances = completeUtterances; + } + + synchronized void beginInput() { + if (readingInput) { + throw new IllegalStateException("A console read is already active."); + } + if (lineOpen) { + System.out.println(); + } + speaker = null; + lineOpen = false; + readingInput = true; + System.out.print("You: "); + System.out.flush(); + } + + synchronized void endInput() { + if (!readingInput) { + return; + } + readingInput = false; + if (!inputEchoed) { + System.out.println(); + } + if (pending.length() > 0) { + System.out.print(pending); + pending.setLength(0); + } + speaker = null; + } + + synchronized void beginTurn() { + speaker = null; + } + + synchronized void endUtterance() { + if (utterance.length() == 0) { + return; + } + String spokenText = utterance.toString().trim(); + if (spokenText.isEmpty()) { + utterance.setLength(0); + return; + } + String text = (lineOpen ? System.lineSeparator() : "") + "Assistant: " + spokenText + + System.lineSeparator(); + if (readingInput && text.length() > MAX_BUFFERED_TRANSCRIPT_CHARACTERS - pending.length()) { + throw new IllegalStateException("Buffered transcript exceeded 65536 characters while reading input."); + } + utterance.setLength(0); + speaker = null; + write(text); + } + + synchronized void print(ServerEvent event) { + checkError(event); + if (event.isSessionInputTranscriptDelta() || event.isSessionOutputTranscriptDelta()) { + String delta = event.isSessionInputTranscriptDelta() + ? event.asSessionInputTranscriptDelta().delta() : event.asSessionOutputTranscriptDelta().delta(); + if (completeUtterances && event.isSessionOutputTranscriptDelta()) { + if (delta.length() > MAX_BUFFERED_TRANSCRIPT_CHARACTERS - pending.length() - utterance.length()) { + throw new IllegalStateException("Buffered transcript exceeded 65536 characters."); + } + utterance.append(delta); + return; + } + String nextSpeaker = event.isSessionInputTranscriptDelta() ? "You" : "Assistant"; + if (!nextSpeaker.equals(speaker)) { + write((lineOpen ? System.lineSeparator() : "") + nextSpeaker + ": "); + speaker = nextSpeaker; + } + write(delta); + } else if (event.isSessionStarted()) { + printEvent(event); + } else if (event.isSessionClosed()) { + endUtterance(); + if (lineOpen) { + write(System.lineSeparator()); + } + speaker = null; + } + } + + private void write(String text) { + if (readingInput) { + if (text.length() > MAX_BUFFERED_TRANSCRIPT_CHARACTERS - pending.length() - utterance.length()) { + throw new IllegalStateException("Buffered transcript exceeded 65536 characters while reading input."); + } + pending.append(text); + } else { + System.out.print(text); + } + if (!text.isEmpty()) { + lineOpen = !text.endsWith("\n"); + } + } + } + + static AgentResource createAgent(AgentsClientBuilder builder, String model, String backendModel, + String scenario, boolean tools, boolean keepAgent) { + ResponsesDelegationConfig.Builder responses = ResponsesDelegationConfig.builder().model(backendModel) + .instructions(tools + ? "Use get_weather for weather requests. After receiving its result, answer briefly." + : "Answer the user briefly and helpfully."); + if (tools) { + responses.addTool(GptLiveFunctionToolSample.weatherTool()); + } + VoiceAgentLiveDefinition definition = new VoiceAgentLiveDefinition(model) + .setInstructions("You are a helpful voice assistant. Delegate requests to your Responses backend.") + .setAudio(SessionConfig.Audio.builder() + .audioPcmFormat(com.openai.models.live.AudioFormat.AudioPcm.Rate._24000).build()) + .setDelegation(new VoiceAgentLiveResponsesDelegation(responses.build())); + String agentName = createAgentName(scenario); + AgentsClient agents = builder.buildAgentsClient(); + agents.createAgentVersion(agentName, new CreateAgentVersionInput(definition)); + System.out.println("Created GPT Live agent: " + agentName); + return new AgentResource(builder, agents, agentName, keepAgent); + } + + static String createAgentName(String scenario) { + return "gpt-live-" + scenario + "-" + UUID.randomUUID(); + } + + static final class AgentResource implements AutoCloseable { + final AgentsClientBuilder builder; + final String name; + private final AgentsClient agents; + private final boolean keep; + + AgentResource(AgentsClientBuilder builder, AgentsClient agents, String name, boolean keep) { + this.builder = builder; + this.agents = agents; + this.name = name; + this.keep = keep; + } + + @Override + public void close() { + if (keep) { + System.out.println("Kept GPT Live agent: " + name); + } else { + agents.deleteAgent(name); + System.out.println("Deleted GPT Live agent: " + name); + } + } + } +} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveTextConversationAsyncSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveTextConversationAsyncSample.java new file mode 100644 index 0000000000000..d3a19df7d40ba --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveTextConversationAsyncSample.java @@ -0,0 +1,330 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.BetaVoiceAgentsLiveWebSocketSessionAsyncClient; +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.core.util.Configuration; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; +import com.azure.identity.DefaultAzureCredentialBuilder; +import com.openai.models.live.ServerEvent; +import com.openai.models.responses.ResponseStreamEvent; +import java.io.Console; +import java.io.InputStreamReader; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; +import reactor.core.publisher.Sinks; +import reactor.core.scheduler.Schedulers; +import com.openai.models.live.ClientEvent; +import java.io.BufferedReader; +import java.io.Reader; +import java.util.function.Consumer; +import java.util.function.Function; +import java.util.function.Predicate; +import java.util.function.Supplier; + + +/** + * Runs a nonblocking, multi-turn GPT Live text conversation using Responses delegation. + * Type messages interactively; a blank line, exit or quit ends the session. + * Alternatively, pass prompts as command-line arguments for a scripted conversation. + * Console lines are read on a worker, off the WebSocket thread. + * After a remote disconnect, a pending console read may require Enter to release the input worker. + * Interactive mode requests a spoken greeting, plays 24-kHz PCM16 replies, and prints the voice frontend's transcripts. + * Before the first prompt, it waits for greeting audio and text, one second of speech inactivity, and speaker drain. + * The quiet-period boundary is a heuristic, not a service speech-completion event. + * Assistant transcript fragments are assembled into utterances using the speech-inactivity heuristic. + * Completed utterances wait until Enter if input is active; incomplete text is not flushed by input submission. + * Audio continues playing while typing on the You: line. + * It streams real-time silent PCM input instead of a microphone; voice usage continues while you type. + * Backend completion allows another typed prompt; it does not mean that spoken output has finished. + * Scripted mode prints backend text without requiring a speaker. + * Agent creation/deletion run outside the asynchronous session on the main thread. + * The uniquely named agent is deleted unless FOUNDRY_KEEP_GPT_LIVE_AGENT=true. + * The standalone main method shuts down shared Reactor schedulers on exit; do not invoke it inside a shared application. + * + *

Before running the sample, set these environment variables:

+ *
    + *
  • {@code FOUNDRY_PROJECT_ENDPOINT} - The Azure AI Project endpoint.
    • + *
  • {@code FOUNDRY_GPT_LIVE_MODEL} - Optional. The GPT Live deployment name. Defaults to {@code gpt-live-1}.
    • + *
  • {@code FOUNDRY_GPT_LIVE_BACKEND_MODEL} - Optional. The Responses backend deployment name in the Live + * resource. Defaults to {@code gpt-5.6-terra}.
    • + *
  • {@code FOUNDRY_GPT_LIVE_STORE} - Optional. Set to {@code true} to persist conversation content and audio + * and request a Foundry conversation ID. Defaults to {@code false}. Also keep the agent for later REST reads.
    • + *
  • {@code FOUNDRY_KEEP_GPT_LIVE_AGENT} - Optional. Set to {@code true} to keep the agent after the sample. + * Defaults to {@code false}.
    • + *
+ */ +public class GptLiveTextConversationAsyncSample { + public static void main(String[] args) { + Configuration configuration = Configuration.getGlobalConfiguration(); + String endpoint = configuration.get("FOUNDRY_PROJECT_ENDPOINT"); + String model = configuration.get("FOUNDRY_GPT_LIVE_MODEL"); + String backendModel = configuration.get("FOUNDRY_GPT_LIVE_BACKEND_MODEL"); + boolean keepAgent = Boolean.parseBoolean(configuration.get("FOUNDRY_KEEP_GPT_LIVE_AGENT", "false")); + boolean store = Boolean.parseBoolean(configuration.get("FOUNDRY_GPT_LIVE_STORE", "false")); + + if (endpoint == null || endpoint.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_PROJECT_ENDPOINT before running this sample."); + } + if (model == null) { + model = "gpt-live-1"; + } else if (model.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_GPT_LIVE_MODEL before running this sample."); + } + if (backendModel == null) { + backendModel = "gpt-5.6-terra"; + } else if (backendModel.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_GPT_LIVE_BACKEND_MODEL before running this sample."); + } + + AgentsClientBuilder builder = new AgentsClientBuilder().endpoint(endpoint) + .credential(new DefaultAzureCredentialBuilder().build()) + .allowPreview(true); + Console console = System.console(); + try (GptLiveSampleUtils.AgentResource agent + = GptLiveSampleUtils.createAgent(builder, model, backendModel, "text-async", false, keepAgent)) { + Mono.usingWhen(agent.builder.beta().buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession(agent.name, new VoiceAgentLiveWebSocketConnectionOptions().setStoreEnabled(store)), + session -> args.length == 0 + ? runInteractiveConversationAsync(session, + console == null ? new InputStreamReader(System.in) : console.reader()) + : runConversation(session, Arrays.asList(args)), + BetaVoiceAgentsLiveWebSocketSessionAsyncClient::closeAsync, + (session, error) -> session.closeAsync(), + BetaVoiceAgentsLiveWebSocketSessionAsyncClient::closeAsync).block(); + } finally { + Schedulers.shutdownNow(); + } + } + + static Mono runConversation(BetaVoiceAgentsLiveWebSocketSessionAsyncClient session, List prompts) { + return Mono.defer(() -> { + Sinks.Empty ready = Sinks.empty(); + Sinks.Empty closed = Sinks.empty(); + AtomicBoolean started = new AtomicBoolean(); + AtomicBoolean closing = new AtomicBoolean(); + AtomicBoolean finalized = new AtomicBoolean(); + AtomicBoolean remoteHangup = new AtomicBoolean(); + AtomicReference> responseCompleted = new AtomicReference<>(); + Mono receive = session.receiveEvents() + .publishOn(Schedulers.boundedElastic(), 1) + .takeUntil(ServerEvent::isSessionClosed) + .doOnNext(event -> { + GptLiveSampleUtils.printEvent(event); + if (event.isSessionStarted()) { + if (!started.compareAndSet(false, true)) { + throw new IllegalStateException("Duplicate session.started event."); + } + ready.tryEmitEmpty(); + } else if (event.isResponseEvent()) { + ResponseStreamEvent response = GptLiveSampleUtils.backendEvent(event); + if (response.isCompleted()) { + Sinks.Empty completion = responseCompleted.getAndSet(null); + if (!started.get() || completion == null || closing.get()) { + throw new IllegalStateException("Unexpected GPT Live response lifecycle event."); + } + System.out.println(); + completion.tryEmitEmpty(); + } + } else if (event.isSessionClosed()) { + String reason = event.asSessionClosed().reason().toString(); + remoteHangup.set(started.get() && "remote_hangup".equals(reason)); + if (!remoteHangup.get() && !(closing.get() && "close_requested".equals(reason))) { + throw new IllegalStateException("GPT Live closed before all turns completed. Event: " + + event + "; transport close code=" + session.getCloseCode() + + ", reason=" + session.getCloseReason()); + } + finalized.set(true); + System.out.println("Live session closed (" + reason + ")."); + closed.tryEmitEmpty(); + } + }) + .then(Mono.defer(() -> { + if (!finalized.get()) { + return Mono.error(new IllegalStateException("GPT Live disconnected without session.closed.")); + } + return remoteHangup.get() ? Mono.empty() : Mono.never(); + })); + Mono conversation = ready.asMono().timeout(GptLiveSampleUtils.TIMEOUT) + .thenMany(Flux.fromIterable(prompts).concatMap(prompt -> Mono.defer(() -> { + Sinks.Empty completion = Sinks.empty(); + responseCompleted.set(completion); + System.out.println("You: " + prompt); + System.out.print("Assistant: "); + return session.sendEvent(GptLiveSampleUtils.userMessage(prompt)) + .then(session.sendEvent(GptLiveSampleUtils.responseCreate())) + .then(completion.asMono()).timeout(GptLiveSampleUtils.TIMEOUT); + }), 1)) + .then(Mono.defer(() -> { + closing.set(true); + return session.sendEvent(GptLiveSampleUtils.closeSession()) + .then(closed.asMono()).timeout(GptLiveSampleUtils.TIMEOUT); + })); + return Mono.firstWithSignal(receive, conversation); + }); + } + + static Mono runInteractiveConversationAsync(BetaVoiceAgentsLiveWebSocketSessionAsyncClient session, Reader input) { + return Mono.usingWhen( + Mono.fromCallable(VoiceAgentRealtimeSampleUtils.SpeakerPlayer::new) + .subscribeOn(Schedulers.boundedElastic()), + player -> runInteractiveConversationAsync(session, input, player::play, player::drain), + player -> Mono.fromRunnable(() -> { + boolean speakerEnabled = player.isEnabled(); + try { + player.close(); + } finally { + System.out.printf("(received %.2fs of reply audio%s)%n", + player.getSecondsReceived(), speakerEnabled ? " and played it" : ""); + } + }).subscribeOn(Schedulers.boundedElastic())); + } + + static Mono runInteractiveConversationAsync(BetaVoiceAgentsLiveWebSocketSessionAsyncClient session, Reader input, + Consumer playAudio, Runnable drainAudio) { + return Mono.defer(() -> runInteractiveConversationAsync(session.receiveEvents(), session::sendEvent, + () -> "transport close code=" + session.getCloseCode() + ", reason=" + session.getCloseReason(), + input, playAudio, drainAudio)); + } + + static Mono runInteractiveConversationAsync(Flux events, + Function> sendEvent, Supplier closeDetails, Reader input, + Consumer playAudio, Runnable drainAudio) { + return Mono.defer(() -> { + System.out.println("Type a message and press Enter. Blank line, 'exit' or 'quit' ends the session."); + System.out.println("Interactive voice streams silent audio while you type; voice usage is billable."); + BufferedReader console = input instanceof BufferedReader ? (BufferedReader) input : new BufferedReader(input); + return runConversationAsync(events, sendEvent, closeDetails, transcripts -> readPromptAsync(console, transcripts), + playAudio, drainAudio, true, null); + }); + } + + static Mono readPromptAsync(Reader input) { + return readPromptAsync(input, new GptLiveSampleUtils.AudioTranscriptPrinter()); + } + + private static Mono readPromptAsync(Reader input, GptLiveSampleUtils.AudioTranscriptPrinter transcripts) { + return Mono.defer(() -> { + BufferedReader reader = input instanceof BufferedReader ? (BufferedReader) input : new BufferedReader(input); + return Mono.fromCallable(() -> GptLiveSampleUtils.readPromptBlocking(reader, transcripts)) + .subscribeOn(Schedulers.boundedElastic()) + .doOnCancel(transcripts::endInput); + }); + } + + private static Mono runConversationAsync(Flux events, + Function> sendEvent, Supplier closeDetails, + Function> nextPrompt, + Consumer playAudio, Runnable drainAudio, boolean spoken, GptLiveSampleUtils.AudioProcessor microphone) { + Mono opening = microphone == null ? Mono.empty() + : Mono.defer(() -> sendEvent.apply(GptLiveSampleUtils.userMessage("Greet me briefly with: Hi, how can I help you today?")) + .then(sendEvent.apply(GptLiveSampleUtils.responseCreate())) + .then(Mono.fromRunnable(() -> System.out.println( + "Opening greeting requested. Speak into the microphone. Press Enter to end the call.")))); + return runConversationCoreAsync(events, sendEvent, closeDetails, nextPrompt, playAudio, drainAudio, spoken, + spoken && microphone == null, microphone == null && System.console() != null, + () -> Mono.fromRunnable(() -> { + if (microphone != null) { + microphone.start(); + } + }), opening, backend -> Mono.just(false), reason -> microphone != null); + } + + static Mono runConversationCoreAsync(Flux events, + Function> sendEvent, Supplier closeDetails, + Function> nextPrompt, + Consumer playAudio, Runnable drainAudio, boolean spoken, boolean interactive, boolean inputEchoed, + Supplier> onStarted, Mono opening, + Function> handleResponse, Predicate acceptClose) { + return Mono.defer(() -> { + GptLiveSampleUtils.AudioTranscriptPrinter transcripts = new GptLiveSampleUtils.AudioTranscriptPrinter(inputEchoed, interactive); + GptLiveSampleUtils.Greeting greeting = new GptLiveSampleUtils.Greeting(); + Sinks.Empty ready = Sinks.empty(); + Sinks.Empty closed = Sinks.empty(); + Sinks.Empty audioStarted = Sinks.empty(); + Sinks.Empty stopAudio = Sinks.empty(); + Sinks.Empty audioStopped = Sinks.empty(); + AtomicReference> responseCompleted = new AtomicReference<>(); + GptLiveSampleUtils.ConversationEventHandler> handler + = new GptLiveSampleUtils.ConversationEventHandler>(spoken, closeDetails, acceptClose) { + @Override + Mono onStarted() { + return Mono.defer(onStarted).doOnSuccess(ignored -> ready.tryEmitEmpty()); + } + + @Override + Mono onResponse(ResponseStreamEvent event) { + return Mono.defer(() -> handleResponse.apply(event)).defaultIfEmpty(false) + .doOnNext(continuation -> { + if (event.isCompleted() && completeResponse(continuation)) { + responseCompleted.getAndSet(null).tryEmitEmpty(); + } + }).then(); + } + + @Override + Mono onClosed() { + return Mono.fromRunnable(() -> closed.tryEmitEmpty()); + } + + @Override + Mono ignored() { + return Mono.empty(); + } + }; + Mono receive = events + .publishOn(Schedulers.boundedElastic(), 1) + .takeUntil(ServerEvent::isSessionClosed) + .concatMap(event -> Mono.defer(() -> { + if (spoken) { + transcripts.print(event); + if (event.isSessionOutputAudioDelta()) { + byte[] pcm = GptLiveSampleUtils.decodeAudio(event); + playAudio.accept(pcm); + greeting.onAudio(pcm); + } else if (event.isSessionOutputTranscriptDelta() + && !event.asSessionOutputTranscriptDelta().delta().trim().isEmpty()) { + greeting.onTranscript(); + } + } else { + GptLiveSampleUtils.printEvent(event); + } + return handler.process(event); + }), 1) + .then(Mono.defer(() -> handler.awaitConversationAfterReceive() ? Mono.never() : Mono.empty())); + Mono audio = ready.asMono() + .thenMany(GptLiveSampleUtils.silenceFramesAsync().takeUntilOther(stopAudio.asMono()) + .concatMap(frame -> sendEvent.apply(frame).doOnSuccess(ignored -> audioStarted.tryEmitEmpty()), 1)) + .then() + .doOnSuccess(ignored -> audioStopped.tryEmitEmpty()) + .then(Mono.never()); + Mono renderTranscripts = greeting.quietPeriodsAsync() + .doOnNext(ignored -> transcripts.endUtterance()).then(); + Mono conversation = ready.asMono().timeout(GptLiveSampleUtils.TIMEOUT) + .then(Mono.defer(() -> interactive ? audioStarted.asMono().timeout(GptLiveSampleUtils.TIMEOUT) : Mono.empty())) + .then(Mono.defer(() -> interactive + ? sendEvent.apply(GptLiveSampleUtils.commentary("Hi, how can I help you today?")) + .timeout(GptLiveSampleUtils.TIMEOUT) + .then(greeting.awaitPlaybackAsync(drainAudio)) + .then(Mono.fromRunnable(transcripts::endUtterance)) + : opening)) + .then(Mono.defer(() -> GptLiveSampleUtils.promptAsync(sendEvent, nextPrompt, responseCompleted, spoken, transcripts, handler))) + .then(Mono.defer(() -> { + handler.requestClose(); + stopAudio.tryEmitEmpty(); + return (interactive ? audioStopped.asMono() : Mono.empty()) + .then(sendEvent.apply(GptLiveSampleUtils.closeSession())) + .then(closed.asMono()).timeout(GptLiveSampleUtils.TIMEOUT); + })); + return interactive ? Mono.firstWithSignal(receive, audio, renderTranscripts, conversation) + : Mono.firstWithSignal(receive, conversation); + }); + } + +} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveTextConversationSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveTextConversationSample.java new file mode 100644 index 0000000000000..5f8f20bedbe41 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/GptLiveTextConversationSample.java @@ -0,0 +1,345 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.BetaVoiceAgentsLiveWebSocketSessionClient; +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.core.util.Configuration; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; +import com.azure.identity.DefaultAzureCredentialBuilder; +import com.openai.models.live.ClientEvent; +import com.openai.models.live.ServerEvent; +import com.openai.models.responses.ResponseStreamEvent; +import java.io.Console; +import java.io.InputStreamReader; +import java.util.Arrays; +import java.util.Iterator; +import java.util.List; +import java.util.function.Consumer; +import java.util.function.Supplier; +import reactor.core.scheduler.Schedulers; +import java.io.BufferedReader; +import java.io.IOException; +import java.io.Reader; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Function; +import java.util.function.Predicate; + + +/** + * Runs a synchronous, multi-turn GPT Live text conversation using Responses delegation. + * Type messages interactively; a blank line, exit or quit ends the session. + * Alternatively, pass prompts as command-line arguments for a scripted conversation. + * Interactive mode plays a spoken greeting and replies, and prints frontend transcripts instead of backend notes. + * It streams silent PCM while typing; voice usage is billable. Greeting completion uses the same speech-inactivity + * heuristic and speaker drain as the asynchronous sample. Assistant transcript fragments are assembled into + * utterances independently of typed turns; completed text waits until Enter when a You: line is active. + * Synchronous session I/O runs on workers coordinated by this sample. + * After a remote disconnect, a pending console read may require Enter to release the input worker. + * Scripted command-line prompts print backend text without speaker playback. + * A uniquely named agent is deleted on exit unless FOUNDRY_KEEP_GPT_LIVE_AGENT=true. + * The standalone main method shuts down shared Reactor schedulers on exit; do not invoke it inside a shared application. + * + *

Before running the sample, set these environment variables:

+ *
    + *
  • {@code FOUNDRY_PROJECT_ENDPOINT} - The Azure AI Project endpoint.
    • + *
  • {@code FOUNDRY_GPT_LIVE_MODEL} - Optional. The GPT Live deployment name. Defaults to {@code gpt-live-1}.
    • + *
  • {@code FOUNDRY_GPT_LIVE_BACKEND_MODEL} - Optional. The Responses backend deployment name in the Live + * resource. Defaults to {@code gpt-5.6-terra}.
    • + *
  • {@code FOUNDRY_GPT_LIVE_STORE} - Optional. Set to {@code true} to persist conversation content and audio + * and request a Foundry conversation ID. Defaults to {@code false}. Also keep the agent for later REST reads.
    • + *
  • {@code FOUNDRY_KEEP_GPT_LIVE_AGENT} - Optional. Set to {@code true} to keep the agent after the sample. + * Defaults to {@code false}.
    • + *
+ */ +public class GptLiveTextConversationSample { + public static void main(String[] args) { + Configuration configuration = Configuration.getGlobalConfiguration(); + String endpoint = configuration.get("FOUNDRY_PROJECT_ENDPOINT"); + String model = configuration.get("FOUNDRY_GPT_LIVE_MODEL"); + String backendModel = configuration.get("FOUNDRY_GPT_LIVE_BACKEND_MODEL"); + boolean keepAgent = Boolean.parseBoolean(configuration.get("FOUNDRY_KEEP_GPT_LIVE_AGENT", "false")); + boolean store = Boolean.parseBoolean(configuration.get("FOUNDRY_GPT_LIVE_STORE", "false")); + + if (endpoint == null || endpoint.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_PROJECT_ENDPOINT before running this sample."); + } + if (model == null) { + model = "gpt-live-1"; + } else if (model.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_GPT_LIVE_MODEL before running this sample."); + } + if (backendModel == null) { + backendModel = "gpt-5.6-terra"; + } else if (backendModel.trim().isEmpty()) { + throw new IllegalArgumentException("Set environment variable FOUNDRY_GPT_LIVE_BACKEND_MODEL before running this sample."); + } + + AgentsClientBuilder builder = new AgentsClientBuilder().endpoint(endpoint) + .credential(new DefaultAzureCredentialBuilder().build()) + .allowPreview(true); + try (GptLiveSampleUtils.AgentResource agent + = GptLiveSampleUtils.createAgent(builder, model, backendModel, "text", false, keepAgent); + BetaVoiceAgentsLiveWebSocketSessionClient session = agent.builder.beta() + .buildBetaVoiceAgentsLiveWebSocketClient() + .openWebSocketSession(agent.name, new VoiceAgentLiveWebSocketConnectionOptions().setStoreEnabled(store))) { + if (args.length == 0) { + Console console = System.console(); + runInteractiveConversation(session, + console == null ? new InputStreamReader(System.in) : console.reader()); + } else { + runConversation(session, Arrays.asList(args)); + } + } finally { + Schedulers.shutdownNow(); + } + } + + static void runConversation(BetaVoiceAgentsLiveWebSocketSessionClient session, List prompts) { + runConversation(session.receiveEvents(GptLiveSampleUtils.TIMEOUT), session::sendEvent, + () -> "transport close code=" + session.getCloseCode() + ", reason=" + session.getCloseReason(), prompts); + } + + static void runConversation(Iterable events, Consumer sendEvent, + Supplier closeDetails, List prompts) { + Iterator pending = prompts.iterator(); + boolean started = false; + boolean awaitingResponse = false; + boolean closing = false; + for (ServerEvent event : events) { + GptLiveSampleUtils.printEvent(event); + boolean nextTurn = false; + if (event.isSessionStarted()) { + if (started) { + throw new IllegalStateException("Duplicate session.started event."); + } + started = true; + nextTurn = true; + } else if (event.isResponseEvent()) { + ResponseStreamEvent response = GptLiveSampleUtils.backendEvent(event); + if (response.isCompleted()) { + if (!started || !awaitingResponse || closing) { + throw new IllegalStateException("Unexpected GPT Live response lifecycle event."); + } + awaitingResponse = false; + System.out.println(); + nextTurn = true; + } + } else if (event.isSessionClosed()) { + String reason = event.asSessionClosed().reason().toString(); + if (!(started && "remote_hangup".equals(reason)) + && !(closing && "close_requested".equals(reason))) { + throw new IllegalStateException("GPT Live closed before all turns completed. Event: " + event + + "; " + closeDetails.get()); + } + System.out.println("Live session closed (" + reason + ")."); + return; + } + if (nextTurn) { + if (pending.hasNext()) { + String prompt = pending.next(); + System.out.println("You: " + prompt); + System.out.print("Assistant: "); + awaitingResponse = true; + sendEvent.accept(GptLiveSampleUtils.userMessage(prompt)); + sendEvent.accept(GptLiveSampleUtils.responseCreate()); + } else { + closing = true; + sendEvent.accept(GptLiveSampleUtils.closeSession()); + } + } + } + throw new IllegalStateException("GPT Live disconnected without session.closed."); + } + + static void runInteractiveConversation(BetaVoiceAgentsLiveWebSocketSessionClient session, Reader input) { + try (VoiceAgentRealtimeSampleUtils.SpeakerPlayer player = new VoiceAgentRealtimeSampleUtils.SpeakerPlayer()) { + runInteractiveConversation(session, input, player::play, player::drain); + player.drain(); + System.out.printf("(received %.2fs of reply audio%s)%n", player.getSecondsReceived(), + player.isEnabled() ? " and played it" : ""); + } + } + + static void runInteractiveConversation(BetaVoiceAgentsLiveWebSocketSessionClient session, Reader input, + Consumer playAudio, Runnable drainAudio) { + System.out.println("Type a message and press Enter. Blank line, 'exit' or 'quit' ends the session."); + System.out.println("Interactive voice streams silent audio while you type; voice usage is billable."); + BufferedReader console = input instanceof BufferedReader ? (BufferedReader) input : new BufferedReader(input); + runConversationCore(session.receiveEvents(GptLiveSampleUtils.TIMEOUT), session::sendEvent, + () -> "transport close code=" + session.getCloseCode() + ", reason=" + session.getCloseReason(), transcripts -> { + try { + return GptLiveSampleUtils.readPromptBlocking(console, transcripts); + } catch (IOException error) { + throw new IllegalStateException("Unable to read console input.", error); + } + }, playAudio, drainAudio, true, () -> { }, backend -> false, reason -> false); + } + + static void runConversationCore(Iterable events, Consumer sendEvent, + Supplier closeDetails, Function nextPrompt, + Consumer playAudio, Runnable drainAudio, boolean spoken, + Runnable onStarted, Predicate handleResponse, Predicate acceptClose) { + GptLiveSampleUtils.ConversationEventHandler handler + = new GptLiveSampleUtils.ConversationEventHandler(spoken, closeDetails, acceptClose) { + @Override + Boolean onStarted() { + onStarted.run(); + return false; + } + + @Override + Boolean onResponse(ResponseStreamEvent event) { + boolean continuation = handleResponse.test(event); + if (event.isCompleted()) { + completeResponse(continuation); + } + return false; + } + + @Override + Boolean onClosed() { + return true; + } + + @Override + Boolean ignored() { + return false; + } + }; + ExecutorService workers = Executors.newFixedThreadPool(2, task -> { + Thread thread = new Thread(task, "gpt-live-sync-conversation"); + thread.setDaemon(true); + return thread; + }); + BlockingQueue pending = new LinkedBlockingQueue<>(256); + AtomicReference receiveFailure = new AtomicReference<>(); + AtomicBoolean receiveEnded = new AtomicBoolean(); + GptLiveSampleUtils.AudioTranscriptPrinter transcripts = new GptLiveSampleUtils.AudioTranscriptPrinter(System.console() != null, spoken); + Future receiver = workers.submit(() -> { + try { + for (ServerEvent event : events) { + pending.put(event); + if (event.isSessionClosed()) { + break; + } + } + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + } catch (RuntimeException error) { + receiveFailure.set(error); + } finally { + receiveEnded.set(true); + } + }); + Future prompt = null; + boolean greetingComplete = !spoken; + boolean greetingAudio = false; + boolean greetingTranscript = false; + long deadline = System.nanoTime() + GptLiveSampleUtils.TIMEOUT.toNanos(); + long nextAudio = 0; + long quietSince = 0; + ClientEvent silence = GptLiveSampleUtils.audio(new byte[960]); + try { + while (true) { + ServerEvent event = pending.poll(20, TimeUnit.MILLISECONDS); + long now = System.nanoTime(); + if (event != null) { + if (spoken) { + transcripts.print(event); + if (event.isSessionOutputAudioDelta()) { + byte[] pcm = GptLiveSampleUtils.decodeAudio(event); + playAudio.accept(pcm); + if (GptLiveSampleUtils.hasSpeech(pcm)) { + greetingAudio = true; + quietSince = now; + } + } else if (event.isSessionOutputTranscriptDelta() + && !event.asSessionOutputTranscriptDelta().delta().trim().isEmpty()) { + greetingTranscript = true; + quietSince = now; + } + } else { + GptLiveSampleUtils.printEvent(event); + } + if (handler.process(event)) { + return; + } + if (event.isSessionStarted()) { + if (spoken) { + sendEvent.accept(silence); + nextAudio = now + TimeUnit.MILLISECONDS.toNanos(20); + sendEvent.accept(GptLiveSampleUtils.commentary("Hi, how can I help you today?")); + } + } + } else if (receiveEnded.get()) { + if (receiveFailure.get() != null) { + throw receiveFailure.get(); + } + throw new IllegalStateException("GPT Live disconnected without session.closed."); + } + + if (handler.canStreamAudio() && now >= nextAudio) { + sendEvent.accept(silence); + nextAudio = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(20); + } + if (spoken && greetingAudio && greetingTranscript && quietSince != 0 + && now - quietSince >= TimeUnit.SECONDS.toNanos(1)) { + transcripts.endUtterance(); + quietSince = 0; + if (!greetingComplete) { + drainAudio.run(); + greetingComplete = true; + } + } + if (handler.canReadPrompt(greetingComplete) && prompt == null) { + prompt = workers.submit(() -> nextPrompt.apply(transcripts)); + } + if (prompt != null && prompt.isDone()) { + String text = prompt.get(); + prompt = null; + deadline = System.nanoTime() + GptLiveSampleUtils.TIMEOUT.toNanos(); + if (text == null) { + handler.requestClose(); + sendEvent.accept(GptLiveSampleUtils.closeSession()); + } else { + handler.beginResponse(); + if (!spoken) { + System.out.print("Assistant: "); + } + sendEvent.accept(GptLiveSampleUtils.userMessage(text)); + sendEvent.accept(GptLiveSampleUtils.responseCreate()); + } + } + if (!handler.canReadPrompt(greetingComplete) && now >= deadline) { + throw new IllegalStateException("Timed out waiting for the GPT Live conversation."); + } + } + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("GPT Live conversation interrupted.", error); + } catch (ExecutionException error) { + if (error.getCause() instanceof RuntimeException) { + throw (RuntimeException) error.getCause(); + } + throw new IllegalStateException("Unable to read console input.", error.getCause()); + } finally { + receiver.cancel(true); + if (prompt != null) { + prompt.cancel(true); + } + transcripts.endInput(); + workers.shutdownNow(); + } + } + +} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentBasicAsyncSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentBasicAsyncSample.java index d5ee1ea3e8e86..849da1d764423 100644 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentBasicAsyncSample.java +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentBasicAsyncSample.java @@ -39,7 +39,7 @@ public static void main(String[] args) { .buildAgentsAsyncClient(); client.createAgentVersion(agentName, - new CreateAgentVersionInput(VoiceAgentSampleUtils.createDefinition(modelType, model, + new CreateAgentVersionInput(VoiceAgentRealtimeSampleUtils.createDefinition(modelType, model, "You are a friendly voice assistant. Keep replies short and natural."))) .doOnNext(created -> System.out.printf("Created voice agent %s, version %s%n", created.getName(), created.getVersion())) @@ -49,7 +49,7 @@ public static void main(String[] args) { .thenMany(client.listAgents(AgentKind.VOICE, null, null, null, null)) .doOnNext(agent -> System.out.println("Voice agent: " + agent.getName())) .then(client.createAgentVersion(agentName, - new CreateAgentVersionInput(VoiceAgentSampleUtils.createDefinition(modelType, model, + new CreateAgentVersionInput(VoiceAgentRealtimeSampleUtils.createDefinition(modelType, model, "You are a friendly voice assistant. Always greet the caller warmly.")) .setDescription("Updated voice-agent instructions."))) .doOnNext(updated -> System.out.println("Created updated version: " + updated.getVersion())) diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentBasicSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentBasicSample.java index e03ab19962386..e5128424c3660 100644 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentBasicSample.java +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentBasicSample.java @@ -40,7 +40,7 @@ public static void main(String[] args) { .buildAgentsClient(); try { AgentVersionDetails created = client.createAgentVersion(agentName, - new CreateAgentVersionInput(VoiceAgentSampleUtils.createDefinition(modelType, model, + new CreateAgentVersionInput(VoiceAgentRealtimeSampleUtils.createDefinition(modelType, model, "You are a friendly voice assistant. Keep replies short and natural."))); System.out.printf("Created voice agent %s, version %s%n", created.getName(), created.getVersion()); @@ -51,7 +51,7 @@ public static void main(String[] args) { } AgentVersionDetails updated = client.createAgentVersion(agentName, - new CreateAgentVersionInput(VoiceAgentSampleUtils.createDefinition(modelType, model, + new CreateAgentVersionInput(VoiceAgentRealtimeSampleUtils.createDefinition(modelType, model, "You are a friendly voice assistant. Always greet the caller warmly.")) .setDescription("Updated voice-agent instructions.")); System.out.println("Created updated version: " + updated.getVersion()); diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveAudioConversationAsyncSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveAudioConversationAsyncSample.java index 2a80cc060fbae..b0c48a48d127c 100644 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveAudioConversationAsyncSample.java +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveAudioConversationAsyncSample.java @@ -7,8 +7,8 @@ import com.azure.ai.agents.AgentsClientBuilder; import com.azure.ai.agents.BetaVoiceAgentsConversationsAsyncClient; import com.azure.ai.agents.BetaAgentsAsyncClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketAsyncClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionAsyncClient; import com.azure.ai.agents.models.CreateAgentVersionInput; import com.azure.ai.agents.models.RealtimeConversationItemInputAudioTranscriptionCompletedEvent; import com.azure.ai.agents.models.RealtimeInputAudioBufferSpeechStartedEvent; @@ -42,6 +42,7 @@ import java.util.concurrent.LinkedBlockingQueue; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Function; /** * Demonstrates an asynchronous hands-free voice conversation using Java Sound and server-side VAD. @@ -81,7 +82,7 @@ public static void main(String[] args) { .allowPreview(true); AgentsAsyncClient agents = builder.buildAgentsAsyncClient(); BetaAgentsAsyncClient betaAgents = builder.beta().buildBetaAgentsAsyncClient(); - BetaVoiceAgentWebSocketAsyncClient realtime = builder.beta().buildBetaVoiceAgentWebSocketAsyncClient(); + BetaVoiceAgentsWebSocketAsyncClient realtime = builder.beta().buildBetaVoiceAgentsWebSocketAsyncClient(); BetaVoiceAgentsConversationsAsyncClient conversations = builder.beta().buildBetaVoiceAgentsConversationsAsyncClient(); @@ -99,9 +100,9 @@ public static void main(String[] args) { }) .then(Mono.usingWhen(realtime.openWebSocketSession(agentName), session -> runConversation(session, conversationId), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync, + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync, (session, error) -> session.closeAsync(), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync)) + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync)) .then(Mono.defer(() -> conversationId.get() == null ? Mono.fromRunnable(() -> System.out.println("No persisted conversation ID was returned.")) : VoiceAgentRealtimeSampleUtils.readConversation(conversations, agentName, conversationId.get()))) @@ -120,7 +121,7 @@ private static Mono cleanupAgent(AgentsAsyncClient agents, String agentNam .doOnSuccess(ignored -> System.out.println("Deleted voice agent: " + agentName)); } - private static Mono runConversation(BetaVoiceAgentWebSocketSessionAsyncClient session, + private static Mono runConversation(BetaVoiceAgentsWebSocketSessionAsyncClient session, AtomicReference conversationId) { AudioProcessor processor = new AudioProcessor(session); AtomicBoolean responseActive = new AtomicBoolean(); @@ -195,7 +196,7 @@ static final class AudioProcessor implements AutoCloseable { private static final int CHUNK_BYTES = 2400; static final int MAX_PLAYBACK_BYTES = VoiceAgentRealtimeSampleUtils.SAMPLE_RATE * 2 * 60; private static final byte[] STOP = new byte[0]; - private final BetaVoiceAgentWebSocketSessionAsyncClient session; + private final Function> sendAudio; private final AudioFormat format = new AudioFormat(VoiceAgentRealtimeSampleUtils.SAMPLE_RATE, 16, 1, true, false); private final BlockingQueue playback = new LinkedBlockingQueue<>(MAX_PLAYBACK_BYTES / 2); private int queuedPlaybackBytes; @@ -207,16 +208,24 @@ static final class AudioProcessor implements AutoCloseable { private Thread captureThread; private Thread playbackThread; - AudioProcessor(BetaVoiceAgentWebSocketSessionAsyncClient session) { + AudioProcessor(BetaVoiceAgentsWebSocketSessionAsyncClient session) { this(session, null, null); } - AudioProcessor(BetaVoiceAgentWebSocketSessionAsyncClient session, TargetDataLine microphone, SourceDataLine speaker) { - this.session = session; + AudioProcessor(BetaVoiceAgentsWebSocketSessionAsyncClient session, TargetDataLine microphone, SourceDataLine speaker) { + this(microphone, speaker, pcm -> session.appendInputAudio(pcm)); + } + + AudioProcessor(TargetDataLine microphone, SourceDataLine speaker, Function> sendAudio) { + this.sendAudio = sendAudio; this.microphone = microphone; this.speaker = speaker; } + Mono failureSignal() { + return failure.asMono(); + } + synchronized void start() { if (closed) { throw new IllegalStateException("Audio processor is already closed."); @@ -252,7 +261,7 @@ private void capture() { while (running.get()) { int read = microphone.read(buffer, 0, buffer.length); if (read > 0 && running.get()) { - session.appendInputAudio(BinaryData.fromBytes(Arrays.copyOf(buffer, read))).block(SEND_TIMEOUT); + sendAudio.apply(BinaryData.fromBytes(Arrays.copyOf(buffer, read))).block(SEND_TIMEOUT); } } } catch (RuntimeException error) { diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveFunctionToolSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveFunctionToolSample.java index f51801608fb3d..9ed04a93c7d21 100644 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveFunctionToolSample.java +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveFunctionToolSample.java @@ -5,8 +5,8 @@ import com.azure.ai.agents.AgentsClient; import com.azure.ai.agents.AgentsClientBuilder; -import com.azure.ai.agents.BetaVoiceAgentWebSocketClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionClient; import com.azure.ai.agents.models.CreateAgentVersionInput; import com.azure.ai.agents.models.RealtimeConversationItem; import com.azure.ai.agents.models.RealtimeConversationItemType; @@ -64,7 +64,7 @@ public static void main(String[] args) { .endpoint(endpoint) .allowPreview(true); AgentsClient agents = builder.buildAgentsClient(); - BetaVoiceAgentWebSocketClient realtime = builder.beta().buildBetaVoiceAgentWebSocketClient(); + BetaVoiceAgentsWebSocketClient realtime = builder.beta().buildBetaVoiceAgentsWebSocketClient(); Map cityProperty = new LinkedHashMap<>(); cityProperty.put("type", "string"); @@ -89,7 +89,7 @@ public static void main(String[] args) { try { agents.createAgentVersion(agentName, new CreateAgentVersionInput(definition)); System.out.println("Created voice agent: " + agentName); - try (BetaVoiceAgentWebSocketSessionClient session = realtime.openWebSocketSession(agentName)) { + try (BetaVoiceAgentsWebSocketSessionClient session = realtime.openWebSocketSession(agentName)) { ExecutorService receiver = Executors.newSingleThreadExecutor(); Future response = receiver.submit(() -> receiveResponse(session)); try { @@ -114,7 +114,7 @@ public static void main(String[] args) { } } - private static void receiveResponse(BetaVoiceAgentWebSocketSessionClient session) { + private static void receiveResponse(BetaVoiceAgentsWebSocketSessionClient session) { for (RealtimeServerEvent event : session.receiveEvents()) { if (event instanceof RealtimeResponseFunctionCallArgumentsDoneEvent) { RealtimeResponseFunctionCallArgumentsDoneEvent call diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationAsyncSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationAsyncSample.java index c054427a5cc3a..caaf3e36a0c2d 100644 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationAsyncSample.java +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationAsyncSample.java @@ -7,8 +7,8 @@ import com.azure.ai.agents.AgentsClientBuilder; import com.azure.ai.agents.BetaVoiceAgentsConversationsAsyncClient; import com.azure.ai.agents.BetaAgentsAsyncClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketAsyncClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionAsyncClient; import com.azure.ai.agents.models.CreateAgentVersionInput; import com.azure.ai.agents.models.VoiceAgentDefinition; import com.azure.core.util.BinaryData; @@ -59,7 +59,7 @@ public static void main(String[] args) { .allowPreview(true); AgentsAsyncClient agents = builder.buildAgentsAsyncClient(); BetaAgentsAsyncClient betaAgents = builder.beta().buildBetaAgentsAsyncClient(); - BetaVoiceAgentWebSocketAsyncClient realtime = builder.beta().buildBetaVoiceAgentWebSocketAsyncClient(); + BetaVoiceAgentsWebSocketAsyncClient realtime = builder.beta().buildBetaVoiceAgentsWebSocketAsyncClient(); BetaVoiceAgentsConversationsAsyncClient conversations = builder.beta().buildBetaVoiceAgentsConversationsAsyncClient(); @@ -80,9 +80,9 @@ public static void main(String[] args) { }) .then(Mono.usingWhen(realtime.openWebSocketSession(agentName), session -> runConversation(session, scanner, conversationId, player), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync, + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync, (session, error) -> session.closeAsync(), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync)) + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync)) .then(Mono.defer(() -> conversationId.get() == null ? Mono.fromRunnable(() -> System.out.println("No persisted conversation ID was returned.")) : VoiceAgentRealtimeSampleUtils.readConversation(conversations, agentName, conversationId.get()))) @@ -105,7 +105,7 @@ private static Mono cleanupAgent(AgentsAsyncClient agents, String agentNam .doOnSuccess(ignored -> System.out.println("Deleted voice agent: " + agentName)); } - private static Mono runConversation(BetaVoiceAgentWebSocketSessionAsyncClient session, Scanner scanner, + private static Mono runConversation(BetaVoiceAgentsWebSocketSessionAsyncClient session, Scanner scanner, AtomicReference conversationId, VoiceAgentRealtimeSampleUtils.SpeakerPlayer player) { AtomicReference> responseCompleted = new AtomicReference<>(); Disposable receiver = session.receiveEvents().subscribe(event -> { @@ -131,7 +131,7 @@ private static Mono runConversation(BetaVoiceAgentWebSocketSessionAsyncCli }); } - private static Mono prompt(BetaVoiceAgentWebSocketSessionAsyncClient session, Scanner scanner, + private static Mono prompt(BetaVoiceAgentsWebSocketSessionAsyncClient session, Scanner scanner, AtomicReference> responseCompleted) { return Mono.fromCallable(() -> { System.out.print("You: "); diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationSample.java index 6b9ce05b1c00c..97ac549ca56c0 100644 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationSample.java +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationSample.java @@ -7,8 +7,8 @@ import com.azure.ai.agents.AgentsClientBuilder; import com.azure.ai.agents.BetaVoiceAgentsConversationsClient; import com.azure.ai.agents.BetaAgentsClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionClient; import com.azure.ai.agents.models.AgentDetails; import com.azure.ai.agents.models.AgentVersionDetails; import com.azure.ai.agents.models.CreateAgentVersionInput; @@ -16,6 +16,7 @@ import com.azure.ai.agents.models.VoiceAgentDefinition; import com.azure.core.util.BinaryData; import com.azure.core.util.Configuration; +import com.azure.core.exception.ResourceModifiedException; import com.azure.identity.DefaultAzureCredentialBuilder; import java.time.Duration; @@ -40,6 +41,8 @@ *
  • {@code FOUNDRY_VOICE_AGENT_NAME} - Optional. The voice agent name. Defaults to * {@code sample-live-text-conversation-agent-java}.
    • * + *

    This sample deletes and replaces an existing agent with the selected name on a creation conflict. + * Use a name reserved for this sample. The replacement is deleted on exit, including on failure.

    */ public class VoiceAgentLiveTextConversationSample { private static final Duration RESPONSE_TIMEOUT = Duration.ofSeconds(45); @@ -56,23 +59,18 @@ public static void main(String[] args) { .allowPreview(true); AgentsClient agents = builder.buildAgentsClient(); BetaAgentsClient betaAgents = builder.beta().buildBetaAgentsClient(); - BetaVoiceAgentWebSocketClient realtime = builder.beta().buildBetaVoiceAgentWebSocketClient(); + BetaVoiceAgentsWebSocketClient realtime = builder.beta().buildBetaVoiceAgentsWebSocketClient(); BetaVoiceAgentsConversationsClient conversations = builder.beta().buildBetaVoiceAgentsConversationsClient(); - Map request = new LinkedHashMap<>(); - request.put("kind", "voice"); - request.put("name", agentName); - AgentDetails generated = betaAgents.createAgentFromPrompt(BinaryData.fromObject(request)); - - try { - AgentVersionDetails latest = generated.getVersions().getLatest(); + try (SampleAgent agent = createAgent(agents, betaAgents, agentName)) { + AgentVersionDetails latest = agent.details.getVersions().getLatest(); VoiceAgentDefinition definition = (VoiceAgentDefinition) latest.getDefinition(); agents.createAgentVersion(agentName, new CreateAgentVersionInput(definition.setStore(true))); AtomicReference conversationId = new AtomicReference<>(); try (VoiceAgentRealtimeSampleUtils.SpeakerPlayer player = new VoiceAgentRealtimeSampleUtils.SpeakerPlayer(); - BetaVoiceAgentWebSocketSessionClient session = realtime.openWebSocketSession(agentName); + BetaVoiceAgentsWebSocketSessionClient session = realtime.openWebSocketSession(agentName); Scanner scanner = new Scanner(System.in)) { AtomicReference> responseCompleted = new AtomicReference<>(); ExecutorService receiver = Executors.newSingleThreadExecutor(); @@ -124,13 +122,47 @@ public static void main(String[] args) { } else { System.out.println("No persisted conversation ID was returned."); } - } finally { + } + } + + static SampleAgent createAgent(AgentsClient agents, BetaAgentsClient betaAgents, String agentName) { + Map request = new LinkedHashMap<>(); + request.put("kind", "voice"); + request.put("name", agentName); + BinaryData body = BinaryData.fromObject(request); + AgentDetails details; + try { + details = betaAgents.createAgentFromPrompt(body); + } catch (ResourceModifiedException error) { + if (error.getResponse() == null || error.getResponse().getStatusCode() != 409) { + throw error; + } + System.err.println("Deleting existing voice agent before replacement: " + agentName); agents.deleteAgent(agentName); - System.out.println("Deleted voice agent: " + agentName); + details = betaAgents.createAgentFromPrompt(body); + } + return new SampleAgent(agents, agentName, details); + } + + static final class SampleAgent implements AutoCloseable { + final AgentDetails details; + private final AgentsClient agents; + private final String name; + + private SampleAgent(AgentsClient agents, String name, AgentDetails details) { + this.agents = agents; + this.name = name; + this.details = details; + } + + @Override + public void close() { + agents.deleteAgent(name); + System.out.println("Deleted voice agent: " + name); } } - private static boolean awaitResponse(BetaVoiceAgentWebSocketSessionClient session, + private static boolean awaitResponse(BetaVoiceAgentsWebSocketSessionClient session, CompletableFuture completion) { try { completion.get(RESPONSE_TIMEOUT.toMillis(), TimeUnit.MILLISECONDS); diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentRealtimeSampleUtils.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentRealtimeSampleUtils.java index 6ce3f33a22d51..aac7dea9530af 100644 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentRealtimeSampleUtils.java +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentRealtimeSampleUtils.java @@ -11,7 +11,13 @@ import com.azure.ai.agents.models.RealtimeResponseAudioTranscriptDoneEvent; import com.azure.ai.agents.models.RealtimeResponseDoneEvent; import com.azure.ai.agents.models.RealtimeSessionCreatedEvent; +import com.azure.ai.agents.models.VoiceAgentAudioConfiguration; +import com.azure.ai.agents.models.VoiceAgentAudioOutputConfiguration; +import com.azure.ai.agents.models.VoiceAgentDefinition; import com.azure.ai.agents.models.VoiceConversation; +import com.azure.ai.agents.models.VoiceModelType; +import com.azure.ai.agents.models.VoiceOutputModality; +import com.azure.ai.agents.models.VoiceType; import com.azure.core.http.rest.RequestOptions; import com.azure.core.util.BinaryData; import reactor.core.publisher.Mono; @@ -20,6 +26,7 @@ import javax.sound.sampled.AudioSystem; import javax.sound.sampled.LineUnavailableException; import javax.sound.sampled.SourceDataLine; +import java.util.Collections; import java.util.List; import java.util.Map; import java.util.concurrent.atomic.AtomicReference; @@ -30,6 +37,19 @@ final class VoiceAgentRealtimeSampleUtils { private VoiceAgentRealtimeSampleUtils() { } + static VoiceAgentDefinition createDefinition(VoiceModelType modelType, String model, String instructions) { + VoiceAgentAudioOutputConfiguration output = new VoiceAgentAudioOutputConfiguration() + .setVoice("en-US-AvaNeural") + .setVoiceType(VoiceType.AZURE_STANDARD); + return new VoiceAgentDefinition() + .setModelType(modelType) + .setModel(model) + .setInstructions(instructions) + .setAudio(new VoiceAgentAudioConfiguration().setOutput(output)) + .setOutputModalities(Collections.singletonList(VoiceOutputModality.AUDIO)) + .setStore(true); + } + static boolean handleResponseEvent(RealtimeServerEvent event, AtomicReference conversationId, SpeakerPlayer player) { if (event instanceof RealtimeSessionCreatedEvent) { @@ -139,6 +159,12 @@ boolean isEnabled() { return line != null; } + synchronized void drain() { + if (line != null) { + line.drain(); + } + } + @Override public synchronized void close() { if (line != null) { diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentSampleUtils.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentSampleUtils.java deleted file mode 100644 index 2b37d1c0b1e02..0000000000000 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentSampleUtils.java +++ /dev/null @@ -1,31 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT License. - -package com.azure.ai.agents.voice; - -import com.azure.ai.agents.models.VoiceAgentAudioConfiguration; -import com.azure.ai.agents.models.VoiceAgentAudioOutputConfiguration; -import com.azure.ai.agents.models.VoiceAgentDefinition; -import com.azure.ai.agents.models.VoiceModelType; -import com.azure.ai.agents.models.VoiceOutputModality; -import com.azure.ai.agents.models.VoiceType; - -import java.util.Collections; - -final class VoiceAgentSampleUtils { - private VoiceAgentSampleUtils() { - } - - static VoiceAgentDefinition createDefinition(VoiceModelType modelType, String model, String instructions) { - VoiceAgentAudioOutputConfiguration output = new VoiceAgentAudioOutputConfiguration() - .setVoice("en-US-AvaNeural") - .setVoiceType(VoiceType.AZURE_STANDARD); - return new VoiceAgentDefinition() - .setModelType(modelType) - .setModel(model) - .setInstructions(instructions) - .setAudio(new VoiceAgentAudioConfiguration().setOutput(output)) - .setOutputModalities(Collections.singletonList(VoiceOutputModality.AUDIO)) - .setStore(true); - } -} diff --git a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentVersionsSample.java b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentVersionsSample.java index 724b553e30756..9e9e991c73459 100644 --- a/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentVersionsSample.java +++ b/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentVersionsSample.java @@ -38,14 +38,14 @@ public static void main(String[] args) { .buildAgentsClient(); try { AgentVersionDetails first = client.createAgentVersion(agentName, - new CreateAgentVersionInput(VoiceAgentSampleUtils.createDefinition(modelType, model, + new CreateAgentVersionInput(VoiceAgentRealtimeSampleUtils.createDefinition(modelType, model, "You are a helpful voice assistant."))); AgentVersionDetails released = client.createAgentVersion(agentName, - new CreateAgentVersionInput(VoiceAgentSampleUtils.createDefinition(modelType, model, + new CreateAgentVersionInput(VoiceAgentRealtimeSampleUtils.createDefinition(modelType, model, "You are a helpful voice assistant. Greet the caller by name.")) .setDescription("Added a personalized greeting.")); AgentVersionDetails draft = client.createAgentVersion(agentName, - new CreateAgentVersionInput(VoiceAgentSampleUtils.createDefinition(modelType, model, + new CreateAgentVersionInput(VoiceAgentRealtimeSampleUtils.createDefinition(modelType, model, "You are an experimental voice assistant.")) .setDescription("Candidate persona under review.") .setDraft(true)); diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionClientTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionClientTests.java deleted file mode 100644 index 3db2e46948de0..0000000000000 --- a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionClientTests.java +++ /dev/null @@ -1,71 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT License. - -package com.azure.ai.agents; - -import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketClientConfiguration; -import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; -import com.azure.ai.agents.models.VoiceAgentTransport; -import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; -import com.azure.core.http.ProxyOptions; -import java.net.InetSocketAddress; -import java.net.Proxy; -import java.net.URI; -import java.time.OffsetDateTime; -import java.util.List; -import okhttp3.OkHttpClient; -import org.junit.jupiter.api.Test; -import reactor.core.publisher.Mono; - -import static org.junit.jupiter.api.Assertions.assertEquals; - -public class BetaVoiceAgentWebSocketSessionClientTests { - @Test - public void connectionUriUsesSelectedTransport() { - VoiceAgentWebSocketClientConfiguration configuration = new VoiceAgentWebSocketClientConfiguration( - URI.create("https://localhost"), - request -> Mono.just(new com.azure.core.credential.AccessToken("token", OffsetDateTime.now().plusHours(1))), - "v1", "test-user-agent", null, null); - - URI defaultUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, "agent", - new VoiceAgentWebSocketConnectionOptions()); - URI websocketUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, "agent", - new VoiceAgentWebSocketConnectionOptions().setTransportType(VoiceAgentTransport.WEBSOCKET)); - URI webRtcUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, "agent", - new VoiceAgentWebSocketConnectionOptions( - new VoiceAgentWebSocketConnectionOptions().setTransportType(VoiceAgentTransport.WEBRTC))); - - assertEquals("websocket", queryParameter(defaultUri, "transport")); - assertEquals("websocket", queryParameter(websocketUri, "transport")); - assertEquals("webrtc", queryParameter(webRtcUri, "transport")); - } - - @Test - public void syncTransportHonorsNonProxyHosts() { - InetSocketAddress proxyAddress = new InetSocketAddress("localhost", 8080); - ProxyOptions proxyOptions - = new ProxyOptions(ProxyOptions.Type.HTTP, proxyAddress).setNonProxyHosts("localhost"); - VoiceAgentWebSocketClientConfiguration configuration = new VoiceAgentWebSocketClientConfiguration( - URI.create("https://localhost"), - request -> Mono.just(new com.azure.core.credential.AccessToken("token", OffsetDateTime.now().plusHours(1))), - "v1", "test-user-agent", null, proxyOptions); - OkHttpClient client = BetaVoiceAgentWebSocketSessionClient.createHttpClient(configuration, - new VoiceAgentWebSocketConnectionOptions()); - - List bypassed = client.proxySelector().select(URI.create("https://LOCALHOST/session")); - List proxied = client.proxySelector().select(URI.create("https://example.com/session")); - - assertEquals(Proxy.NO_PROXY, bypassed.get(0)); - assertEquals(new Proxy(Proxy.Type.HTTP, proxyAddress), proxied.get(0)); - } - - private static String queryParameter(URI uri, String name) { - for (String parameter : uri.getQuery().split("&")) { - String[] pair = parameter.split("=", 2); - if (name.equals(pair[0])) { - return pair.length == 2 ? pair[1] : ""; - } - } - return null; - } -} diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketSessionClientTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketSessionClientTests.java new file mode 100644 index 0000000000000..5fbdb1144b488 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/BetaVoiceAgentsWebSocketSessionClientTests.java @@ -0,0 +1,221 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents; + +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketClientConfiguration; +import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketUtils; +import com.azure.ai.agents.implementation.realtime.WebSocketSessionTransport; +import com.azure.ai.agents.implementation.realtime.WebSocketTransportOptions; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; +import com.azure.ai.agents.models.VoiceAgentTransport; +import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; +import com.azure.ai.agents.models.VoiceAgentWebSocketOverflowStrategy; +import com.azure.core.http.ProxyOptions; +import com.azure.core.http.HttpHeaders; +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.net.URI; +import java.net.URLDecoder; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.OffsetDateTime; +import java.util.List; +import java.util.function.Consumer; +import okhttp3.OkHttpClient; +import org.junit.jupiter.api.Test; +import reactor.core.publisher.Mono; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; + +public class BetaVoiceAgentsWebSocketSessionClientTests { + @Test + public void connectionUriUsesSelectedTransport() throws Exception { + VoiceAgentWebSocketClientConfiguration configuration = new VoiceAgentWebSocketClientConfiguration( + URI.create("https://localhost"), + request -> Mono.just(new com.azure.core.credential.AccessToken("token", OffsetDateTime.now().plusHours(1))), + "v1", "test-user-agent", null, null); + + URI defaultUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, "agent", + new VoiceAgentWebSocketConnectionOptions()); + URI websocketUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, "agent", + new VoiceAgentWebSocketConnectionOptions().setTransportType(VoiceAgentTransport.WEBSOCKET)); + URI webRtcUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, "agent", + new VoiceAgentWebSocketConnectionOptions( + new VoiceAgentWebSocketConnectionOptions().setTransportType(VoiceAgentTransport.WEBRTC))); + + assertEquals("websocket", queryParameter(defaultUri, "transport")); + assertEquals("websocket", queryParameter(websocketUri, "transport")); + assertEquals("webrtc", queryParameter(webRtcUri, "transport")); + } + + @Test + public void connectionUrisPreserveEncodedPathsAndReplaceEndpointQuery() throws Exception { + VoiceAgentWebSocketClientConfiguration configuration = new VoiceAgentWebSocketClientConfiguration( + URI.create("https://localhost:8443/project%20name/?discard=old&api-version=old"), + request -> Mono.just(new com.azure.core.credential.AccessToken("token", OffsetDateTime.now().plusHours(1))), + "v1", "test-user-agent", null, null); + + URI realtimeUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, "agent/name", + new VoiceAgentWebSocketConnectionOptions().setAgentSessionId("session & id")); + URI liveUri = VoiceAgentWebSocketUtils.buildLiveWebSocketUri(configuration, "agent/name", + new VoiceAgentLiveWebSocketConnectionOptions().setStructuredInputs("{\"name\":\"a & b\"}")); + + for (URI uri : new URI[] { realtimeUri, liveUri }) { + assertEquals("wss", uri.getScheme()); + assertEquals("localhost", uri.getHost()); + assertEquals(8443, uri.getPort()); + assertEquals("v1", queryParameter(uri, "api-version")); + assertEquals(null, queryParameter(uri, "discard")); + } + assertEquals("/project%20name/agents/agent%2Fname/endpoint/protocols/voice", realtimeUri.getRawPath()); + assertEquals("/project%20name/agents/agent%2Fname/endpoint/protocols/voice/live/sessions", + liveUri.getRawPath()); + assertEquals("session & id", queryParameter(realtimeUri, "agent_session_id")); + assertEquals("{\"name\":\"a & b\"}", queryParameter(liveUri, "structured_inputs")); + } + + @Test + public void syncTransportHonorsNonProxyHosts() { + InetSocketAddress proxyAddress = new InetSocketAddress("localhost", 8080); + ProxyOptions proxyOptions + = new ProxyOptions(ProxyOptions.Type.HTTP, proxyAddress).setNonProxyHosts("localhost"); + VoiceAgentWebSocketClientConfiguration configuration = new VoiceAgentWebSocketClientConfiguration( + URI.create("https://localhost"), + request -> Mono.just(new com.azure.core.credential.AccessToken("token", OffsetDateTime.now().plusHours(1))), + "v1", "test-user-agent", null, proxyOptions); + OkHttpClient client = WebSocketSessionTransport.createHttpClient(configuration, + new WebSocketTransportOptions(new VoiceAgentWebSocketConnectionOptions())); + + List bypassed = client.proxySelector().select(URI.create("https://LOCALHOST/session")); + List proxied = client.proxySelector().select(URI.create("https://example.com/session")); + + assertEquals(Proxy.NO_PROXY, bypassed.get(0)); + assertEquals(new Proxy(Proxy.Type.HTTP, proxyAddress), proxied.get(0)); + } + + @Test + public void syncConnectionPreservesTokenFailure() { + IllegalStateException failure = new IllegalStateException("Token unavailable"); + VoiceAgentWebSocketClientConfiguration configuration = new VoiceAgentWebSocketClientConfiguration( + URI.create("https://localhost"), request -> Mono.error(failure), "v1", "test-user-agent", null, null); + + IllegalStateException actual = assertThrows(IllegalStateException.class, + () -> WebSocketSessionTransport.connect(configuration, URI.create("wss://localhost/session"), + new WebSocketTransportOptions(new VoiceAgentWebSocketConnectionOptions()), null, + token -> new HttpHeaders())); + + assertSame(failure, actual); + } + + @Test + public void syncConnectionPreservesHeaderInitializationFailure() { + IllegalStateException failure = new IllegalStateException("Headers unavailable"); + VoiceAgentWebSocketClientConfiguration configuration = new VoiceAgentWebSocketClientConfiguration( + URI.create("https://localhost"), + request -> Mono.just(new com.azure.core.credential.AccessToken("token", OffsetDateTime.now().plusHours(1))), + "v1", "test-user-agent", null, null); + + IllegalStateException actual = assertThrows(IllegalStateException.class, + () -> WebSocketSessionTransport.connect(configuration, URI.create("wss://localhost/session"), + new WebSocketTransportOptions(new VoiceAgentWebSocketConnectionOptions()), null, token -> { + throw failure; + })); + + assertSame(failure, actual); + } + + @Test + public void liveOptionsAndTransportSnapshotAreIndependent() { + Consumer handler = error -> { + }; + VoiceAgentLiveWebSocketConnectionOptions options + = new VoiceAgentLiveWebSocketConnectionOptions().setReceiveBufferCapacity(17) + .setMaxMessageSize(4096) + .setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.DROP_OLDEST) + .setMalformedEventHandler(handler) + .setHandshakeTimeout(Duration.ofSeconds(2)) + .setCloseTimeout(Duration.ofSeconds(3)) + .setStoreEnabled(true) + .setAgentVersionOverride("version-1") + .setStructuredInputs("{\"name\":\"value\"}"); + VoiceAgentLiveWebSocketConnectionOptions copy = new VoiceAgentLiveWebSocketConnectionOptions(options); + WebSocketTransportOptions snapshot = new WebSocketTransportOptions(options); + options.setReceiveBufferCapacity(1) + .setMaxMessageSize(1) + .setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.ERROR) + .setMalformedEventHandler(null) + .setHandshakeTimeout(Duration.ofSeconds(10)) + .setCloseTimeout(Duration.ofSeconds(10)) + .setStoreEnabled(false) + .setAgentVersionOverride(null) + .setStructuredInputs(null); + + assertTransportSnapshot(snapshot, handler); + assertTransportSnapshot(new WebSocketTransportOptions(copy), handler); + assertEquals(true, copy.isStoreEnabled()); + assertEquals("version-1", copy.getAgentVersionOverride()); + assertEquals("{\"name\":\"value\"}", copy.getStructuredInputs()); + } + + @Test + public void realtimeTransportSnapshotIsIndependent() { + Consumer handler = error -> { + }; + VoiceAgentWebSocketConnectionOptions options + = new VoiceAgentWebSocketConnectionOptions().setReceiveBufferCapacity(17) + .setMaxMessageSize(4096) + .setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.DROP_OLDEST) + .setMalformedEventHandler(handler) + .setHandshakeTimeout(Duration.ofSeconds(2)) + .setCloseTimeout(Duration.ofSeconds(3)); + WebSocketTransportOptions snapshot = new WebSocketTransportOptions(options); + options.setReceiveBufferCapacity(1) + .setMaxMessageSize(1) + .setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.ERROR) + .setMalformedEventHandler(null) + .setHandshakeTimeout(Duration.ofSeconds(10)) + .setCloseTimeout(Duration.ofSeconds(10)); + + assertTransportSnapshot(snapshot, handler); + } + + @Test + public void liveOptionsValidateTransportSettings() { + VoiceAgentLiveWebSocketConnectionOptions options = new VoiceAgentLiveWebSocketConnectionOptions(); + assertEquals(256, options.getReceiveBufferCapacity()); + assertEquals(32 * 1024 * 1024, options.getMaxMessageSize()); + assertEquals(VoiceAgentWebSocketOverflowStrategy.ERROR, options.getOverflowStrategy()); + assertEquals(Duration.ofSeconds(30), options.getHandshakeTimeout()); + assertEquals(Duration.ofSeconds(10), options.getCloseTimeout()); + assertThrows(IllegalArgumentException.class, () -> options.setReceiveBufferCapacity(0)); + assertThrows(IllegalArgumentException.class, () -> options.setReceiveBufferCapacity(65537)); + assertThrows(IllegalArgumentException.class, () -> options.setMaxMessageSize(0)); + assertThrows(NullPointerException.class, () -> options.setOverflowStrategy(null)); + assertThrows(IllegalArgumentException.class, () -> options.setHandshakeTimeout(null)); + assertThrows(IllegalArgumentException.class, () -> options.setHandshakeTimeout(Duration.ZERO)); + assertThrows(IllegalArgumentException.class, () -> options.setCloseTimeout(Duration.ofSeconds(-1))); + assertThrows(NullPointerException.class, () -> new VoiceAgentLiveWebSocketConnectionOptions(null)); + } + + private static void assertTransportSnapshot(WebSocketTransportOptions snapshot, Consumer handler) { + assertEquals(17, snapshot.getReceiveBufferCapacity()); + assertEquals(4096, snapshot.getMaxMessageSize()); + assertEquals(VoiceAgentWebSocketOverflowStrategy.DROP_OLDEST, snapshot.getOverflowStrategy()); + assertSame(handler, snapshot.getMalformedEventHandler()); + assertEquals(Duration.ofSeconds(2), snapshot.getHandshakeTimeout()); + assertEquals(Duration.ofSeconds(3), snapshot.getCloseTimeout()); + } + + private static String queryParameter(URI uri, String name) throws Exception { + for (String parameter : uri.getRawQuery().split("&")) { + String[] pair = parameter.split("=", 2); + if (name.equals(pair[0])) { + return pair.length == 2 ? URLDecoder.decode(pair[1], StandardCharsets.UTF_8.name()) : ""; + } + } + return null; + } +} diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/FoundryFeaturesHeaderVerificationTest.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/FoundryFeaturesHeaderVerificationTest.java index 20c80de494c65..87ce4147d12fe 100644 --- a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/FoundryFeaturesHeaderVerificationTest.java +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/FoundryFeaturesHeaderVerificationTest.java @@ -442,11 +442,11 @@ public void webSocketClientsRejectUnsupportedHttpConfiguration() { .retryPolicy(new RetryPolicy()); IllegalStateException syncError - = assertThrows(IllegalStateException.class, () -> builder.beta().buildBetaVoiceAgentWebSocketClient()); + = assertThrows(IllegalStateException.class, () -> builder.beta().buildBetaVoiceAgentsWebSocketClient()); assertTrue(syncError.getMessage().contains("httpClient, pipeline, addPolicy, retryOptions, retryPolicy")); - IllegalStateException asyncError - = assertThrows(IllegalStateException.class, () -> builder.beta().buildBetaVoiceAgentWebSocketAsyncClient()); + IllegalStateException asyncError = assertThrows(IllegalStateException.class, + () -> builder.beta().buildBetaVoiceAgentsWebSocketAsyncClient()); assertEquals(syncError.getMessage(), asyncError.getMessage()); } diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/GptLiveConversationSamplesTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/GptLiveConversationSamplesTests.java new file mode 100644 index 0000000000000..d49e1f4b292ac --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/GptLiveConversationSamplesTests.java @@ -0,0 +1,325 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.core.exception.HttpResponseException; +import com.azure.core.http.HttpClient; +import com.azure.core.http.HttpHeaderName; +import com.azure.core.http.HttpHeaders; +import com.azure.core.http.HttpMethod; +import com.azure.core.http.HttpRequest; +import com.azure.core.http.HttpResponse; +import com.azure.core.http.policy.ExponentialBackoffOptions; +import com.azure.core.http.policy.RetryOptions; +import com.azure.core.test.http.MockHttpResponse; +import com.azure.core.test.utils.MockTokenCredential; +import com.azure.core.util.BinaryData; +import com.azure.core.util.Configuration; +import com.azure.core.util.Context; +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.ArrayDeque; +import java.util.Deque; +import java.util.List; +import java.util.function.Function; +import java.util.stream.Collectors; +import java.util.stream.Stream; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import org.junit.jupiter.params.provider.ValueSource; +import reactor.core.publisher.Mono; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class GptLiveConversationSamplesTests { + private static final String ROOT = "/agents/live-agent/endpoint/protocols/voice/live/conversations"; + private static final String PATH = ROOT + "/conversation-1"; + private static final String CONVERSATION + = "{\"id\":\"conversation-1\",\"status\":\"completed\",\"created_at\":1700000000,\"usage\":{\"seconds\":2.5}}"; + private static final String INPUT = "{\"id\":\"input-1\",\"type\":\"input\",\"transcription\":\"Hello\"}"; + private static final String OUTPUT = "{\"id\":\"output-1\",\"type\":\"output\",\"transcription\":\"Hi\"}"; + private static final String DELEGATION + = "{\"id\":\"delegation-1\",\"type\":\"delegation\",\"target\":\"responses\",\"response_id\":\"response-1\"}"; + private static final byte[] WAV = { 82, 73, 70, 70, 0, 0, 0, 0, 87, 65, 86, 69 }; + private static final Duration TIMEOUT = Duration.ofSeconds(10); + + @TempDir + Path directory; + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void readsCursorPagesAndFetchesSpeechAndDelegationDetails(boolean async) { + ScriptedTransport transport = new ScriptedTransport(async); + transport.json(ROOT, null, 200, page(CONVERSATION)); + transport.json(ROOT, "conversation-1", 200, page()); + transport.json(PATH, null, 200, CONVERSATION); + transport.json(PATH + "/items", null, 200, page(INPUT)); + transport.json(PATH + "/items/input-1", null, 200, INPUT); + transport.json(PATH + "/items", "input-1", 200, page(OUTPUT, DELEGATION)); + transport.json(PATH + "/items/output-1", null, 200, OUTPUT); + transport.json(PATH + "/items/delegation-1", null, 200, DELEGATION); + transport.json(PATH + "/items", "delegation-1", 200, page()); + readConversation(builder(transport), async); + transport.assertConsumed(); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void missingConversationPropagates(boolean async) { + ScriptedTransport transport = new ScriptedTransport(async); + transport.json(ROOT, null, 200, page()); + transport.json(PATH, null, 404, error("not_found")); + HttpResponseException failure + = assertThrows(HttpResponseException.class, () -> readConversation(builder(transport), async)); + assertEquals(404, failure.getResponse().getStatusCode()); + transport.assertConsumed(); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void repeatedCursorFailsRatherThanLooping(boolean async) { + ScriptedTransport transport = new ScriptedTransport(async); + transport.json(ROOT, null, 200, page(CONVERSATION)); + transport.json(ROOT, "conversation-1", 200, page(CONVERSATION)); + assertThrows(IllegalStateException.class, () -> readConversation(builder(transport), async)); + transport.assertConsumed(); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void downloadsMergedAndItemBytesWithoutOverwritingFiles(boolean async) throws IOException { + Path existing = directory.resolve("existing.wav"); + byte[] original = { 1, 2, 3 }; + Files.write(existing, original); + ScriptedTransport transport = audioTransport(async, false); + downloadAudio(builder(transport), async, "input-1"); + transport.assertConsumed(); + assertEquals(3, files().size()); + assertArrayEquals(original, Files.readAllBytes(existing)); + for (Path file : files()) { + if (!file.equals(existing)) { + assertArrayEquals(WAV, Files.readAllBytes(file)); + assertTrue(file.getFileName().toString().endsWith(".wav")); + } + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void byosMetadataDoesNotCallFoundryContentEndpoints(boolean async) throws IOException { + ScriptedTransport transport = audioTransport(async, true); + downloadAudio(builder(transport), async, "input-1"); + transport.assertConsumed(); + assertTrue(files().isEmpty()); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void optionalItemCanBeOmitted(boolean async) throws IOException { + ScriptedTransport transport = new ScriptedTransport(async); + transport.json(PATH + "/audio", null, 200, recording(false)); + transport.audio(PATH + "/audio/content"); + downloadAudio(builder(transport), async, null); + transport.assertConsumed(); + assertEquals(1, files().size()); + } + + @ParameterizedTest + @MethodSource("audioFailures") + public void audioErrorsPreserveStatusAndRetryAfter(boolean async, int status, boolean content) throws IOException { + ScriptedTransport transport = new ScriptedTransport(async); + if (content) { + transport.json(PATH + "/audio", null, 200, recording(false)); + } + transport.json(PATH + (content ? "/audio/content" : "/audio"), null, status, error("recording_not_ready")); + HttpResponseException failure + = assertThrows(HttpResponseException.class, () -> downloadAudio(builder(transport), async, null)); + assertEquals(status, failure.getResponse().getStatusCode()); + assertEquals("3", failure.getResponse().getHeaderValue("Retry-After")); + assertTrue(files().isEmpty()); + transport.assertConsumed(); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void blankItemFailsBeforeNetworkAccess(boolean async) { + ScriptedTransport transport = new ScriptedTransport(async); + assertThrows(IllegalArgumentException.class, () -> downloadAudio(builder(transport), async, " ")); + transport.assertConsumed(); + } + + @Test + public void failedFileWriteRemovesPartialOutput() throws IOException { + InputStream failingStream = new InputStream() { + @Override + public int read() throws IOException { + throw new IOException("Simulated read failure"); + } + }; + assertThrows(IOException.class, () -> GptLiveReadConversationAudioSample + .saveAudio(BinaryData.fromStream(failingStream), directory, "failure-")); + assertTrue(files().isEmpty()); + } + + @Test + public void configurationDirectoryAndCursorValidation() { + assertThrows(IllegalArgumentException.class, + () -> GptLiveSampleUtils.requireSetting(Configuration.NONE, "FOUNDRY_PROJECT_ENDPOINT")); + assertThrows(IllegalArgumentException.class, + () -> GptLiveReadConversationAudioSample.outputDirectory(new String[0])); + assertThrows(IllegalArgumentException.class, () -> GptLiveReadConversationAudioSample + .outputDirectory(new String[] { directory.resolve("missing").toString() })); + assertEquals(directory, + GptLiveReadConversationAudioSample.outputDirectory(new String[] { directory.toString() })); + assertThrows(IllegalStateException.class, () -> GptLiveReadConversationSample.nextCursor(null, null)); + assertThrows(IllegalStateException.class, () -> GptLiveReadConversationSample.nextCursor("", null)); + } + + private static Stream audioFailures() { + return Stream.of(false, true) + .flatMap(async -> Stream.of(403, 404, 409) + .flatMap(status -> Stream.of(false, true).map(content -> Arguments.of(async, status, content)))); + } + + private static void readConversation(AgentsClientBuilder builder, boolean async) { + if (async) { + GptLiveReadConversationAsyncSample + .readConversation(builder.beta().buildBetaVoiceAgentsLiveConversationsAsyncClient(), "live-agent", + "conversation-1") + .block(TIMEOUT); + } else { + GptLiveReadConversationSample.readConversation(builder.beta().buildBetaVoiceAgentsLiveConversationsClient(), + "live-agent", "conversation-1"); + } + } + + private void downloadAudio(AgentsClientBuilder builder, boolean async, String itemId) throws IOException { + if (async) { + GptLiveReadConversationAudioAsyncSample + .downloadAudio(builder.beta().buildBetaVoiceAgentsLiveConversationsAsyncClient(), "live-agent", + "conversation-1", itemId, directory) + .block(TIMEOUT); + } else { + GptLiveReadConversationAudioSample.downloadAudio( + builder.beta().buildBetaVoiceAgentsLiveConversationsClient(), "live-agent", "conversation-1", itemId, + directory); + } + } + + private static ScriptedTransport audioTransport(boolean async, boolean byos) { + ScriptedTransport transport = new ScriptedTransport(async); + transport.json(PATH + "/audio", null, 200, recording(byos)); + if (!byos) { + transport.audio(PATH + "/audio/content"); + } + transport.json(PATH + "/items/input-1/audio", null, 200, + "{\"conversation_id\":\"conversation-1\",\"item_id\":\"input-1\",\"format\":\"wav\",\"sample_rate\":24000" + + (byos ? ",\"blob_uri\":\"https://storage.example/item.wav\"" : "") + "}"); + if (!byos) { + transport.audio(PATH + "/items/input-1/audio/content"); + } + return transport; + } + + private List files() throws IOException { + try (Stream paths = Files.list(directory)) { + return paths.collect(Collectors.toList()); + } + } + + private static String recording(boolean byos) { + return "{\"conversation_id\":\"conversation-1\",\"format\":\"wav\",\"sample_rate\":24000,\"channels\":2," + + "\"channel_layout\":{\"left\":\"user\",\"right\":\"assistant\"},\"duration_ms\":2000" + + (byos ? ",\"blob_uri\":\"https://storage.example/call.wav\"" : "") + "}"; + } + + private static String page(String... entries) { + return "{\"data\":[" + String.join(",", entries) + "]}"; + } + + private static String error(String code) { + return "{\"error\":{\"code\":\"" + code + "\",\"message\":\"Sample service error\"}}"; + } + + private static AgentsClientBuilder builder(HttpClient transport) { + return new AgentsClientBuilder().endpoint("https://localhost") + .credential(new MockTokenCredential()) + .configuration(Configuration.NONE) + .addPolicy(GptLiveReadConversationSample.previewHeaders()) + .httpClient(transport) + .retryOptions(new RetryOptions(new ExponentialBackoffOptions().setMaxRetries(0))); + } + + private static final class ScriptedTransport implements HttpClient { + private final Deque> requests = new ArrayDeque<>(); + private final boolean async; + + ScriptedTransport(boolean async) { + this.async = async; + } + + void json(String path, String after, int status, String body) { + expect(path, after, status, "application/json", BinaryData.fromString(body).toBytes()); + } + + void audio(String path) { + expect(path, null, 200, "audio/wav", WAV); + } + + private void expect(String path, String after, int status, String contentType, byte[] body) { + requests.add(request -> { + assertEquals(HttpMethod.GET, request.getHttpMethod(), "Samples must not modify or delete data"); + assertEquals(path, request.getUrl().getPath()); + assertEquals("VoiceAgents=V1Preview", request.getHeaders().getValue("Foundry-Features")); + String query = request.getUrl().getQuery(); + assertTrue(query.contains("api-version=")); + if (path.equals(ROOT) || path.endsWith("/items")) { + assertTrue(query.contains("limit=100")); + assertTrue(query.contains("order=asc")); + } + if (after != null) { + assertTrue(query.contains("after=" + after), "Missing cursor: " + query); + } else { + assertFalse(query.contains("after="), "Unexpected cursor: " + query); + } + return new MockHttpResponse(request, status, + new HttpHeaders().set(HttpHeaderName.CONTENT_TYPE, contentType) + .set(HttpHeaderName.fromString("Retry-After"), "3"), + body); + }); + } + + @Override + public Mono send(HttpRequest request) { + assertTrue(async, "Synchronous examples must use synchronous HTTP"); + return Mono.fromSupplier(() -> respond(request)); + } + + @Override + public HttpResponse sendSync(HttpRequest request, Context context) { + assertFalse(async, "Asynchronous examples must use asynchronous HTTP"); + return respond(request); + } + + private HttpResponse respond(HttpRequest request) { + assertFalse(requests.isEmpty(), "Unexpected request: " + request.getUrl()); + return requests.removeFirst().apply(request); + } + + void assertConsumed() { + assertTrue(requests.isEmpty(), "Not all expected requests were made"); + } + } +} diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/GptLiveSamplesTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/GptLiveSamplesTests.java new file mode 100644 index 0000000000000..67e4d354a3d48 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/GptLiveSamplesTests.java @@ -0,0 +1,1505 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.ai.agents.BetaVoiceAgentsLiveWebSocketSessionAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsLiveWebSocketSessionClient; +import com.azure.ai.agents.implementation.OpenAIJsonHelper; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; +import com.azure.core.credential.AccessToken; +import com.azure.core.util.BinaryData; +import com.azure.core.util.Configuration; +import com.openai.models.live.ClientEvent; +import com.openai.models.live.ServerEvent; +import com.openai.models.responses.ResponseStreamEvent; +import com.openai.models.responses.ResponseFunctionToolCall; +import java.io.ByteArrayOutputStream; +import java.io.FilterReader; +import java.io.InputStream; +import java.io.IOException; +import java.io.PrintStream; +import java.io.Reader; +import java.io.PipedReader; +import java.io.PipedWriter; +import java.io.StringReader; +import java.lang.reflect.Proxy; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.OffsetDateTime; +import java.util.Arrays; +import java.util.Base64; +import java.util.Collections; +import java.util.List; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.UUID; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Consumer; +import java.util.function.Predicate; +import java.util.stream.Stream; +import javax.sound.sampled.SourceDataLine; +import javax.sound.sampled.TargetDataLine; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.api.parallel.Isolated; +import org.junit.jupiter.api.parallel.ResourceLock; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import org.junit.jupiter.params.provider.ValueSource; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; +import reactor.core.publisher.Sinks; +import reactor.core.scheduler.Schedulers; +import reactor.netty.DisposableServer; +import reactor.test.StepVerifier; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +@Isolated("Virtual-time tests replace the global Reactor schedulers") +@ResourceLock("voice-agent-websocket-tls") +@Timeout(20) +public class GptLiveSamplesTests { + private static final Duration TIMEOUT = Duration.ofSeconds(5); + private static final String STARTED = "{\"type\":\"session.started\",\"session\":{\"id\":\"live-1\"}}"; + private static final String CLOSED = "{\"type\":\"session.closed\",\"reason\":\"close_requested\"}"; + private static final String COMPLETED = "{\"type\":\"response.event\",\"delegation_id\":\"del-1\",\"event\":" + + "{\"type\":\"response.completed\",\"response\":{\"id\":\"resp-1\",\"output\":[]}}}"; + private static final String MODERATED = "{\"type\":\"error\",\"event_id\":\"moderated-1\"," + + "\"error\":{\"type\":\"content_filter\",\"code\":\"content_filter\"," + + "\"message\":\"The generation was stopped due to moderation.\"}}"; + private final List> commands = new CopyOnWriteArrayList<>(); + private DisposableServer server; + + @ParameterizedTest + @ValueSource(strings = { "text", "text-async", "audio-async", "tools" }) + public void agentNamesStayBelowServiceLimit(String scenario) { + String prefix = "gpt-live-" + scenario + "-"; + String name = GptLiveSampleUtils.createAgentName(scenario); + assertTrue(name.startsWith(prefix)); + assertTrue(name.length() <= 57, "Sample agent names must leave headroom below the 63-character limit"); + assertTrue(name.matches("[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9]")); + assertEquals(36, name.substring(prefix.length()).length()); + assertEquals(4, UUID.fromString(name.substring(prefix.length())).version()); + assertFalse(name.equals(GptLiveSampleUtils.createAgentName(scenario))); + } + + @Test + @ResourceLock("java.lang.System.out") + public void audioTranscriptsLabelSpeakersWithoutPrintingBackendText() { + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + GptLiveSampleUtils.AudioTranscriptPrinter printer = new GptLiveSampleUtils.AudioTranscriptPrinter(); + printer.print(typedEvent(BinaryData.fromString(STARTED))); + printer.print(typedEvent(GptLiveSampleUtils.event("session.input_transcript.delta", "delta", "Hello"))); + printer.print(typedEvent(GptLiveSampleUtils.event("session.input_transcript.delta", "delta", " there"))); + printer.print(typedEvent(BinaryData.fromString("{\"type\":\"response.event\"," + + "\"event\":{\"type\":\"response.output_text.delta\",\"delta\":\"backend-only text\"}}"))); + printer.print(typedEvent(GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "Hi!"))); + printer.beginTurn(); + printer.print( + typedEvent(GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "How can I help?"))); + } finally { + System.setOut(original); + } + String transcript = new String(output.toByteArray(), StandardCharsets.UTF_8); + assertEquals("Live session started; conversation_id not provided (persistence may be disabled)." + + System.lineSeparator() + "You: Hello there" + System.lineSeparator() + "Assistant: Hi!" + + System.lineSeparator() + "Assistant: How can I help?", transcript); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + @ResourceLock("java.lang.System.out") + public void startupPrintsFoundryConversationIdInScriptedAndInteractiveModes(boolean interactive) { + ServerEvent event = typedEvent(BinaryData.fromString( + "{\"type\":\"session.started\",\"conversation_id\":\"conversation-1\",\"session\":{\"id\":\"native-1\"}}")); + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + if (interactive) { + new GptLiveSampleUtils.AudioTranscriptPrinter(false, true).print(event); + } else { + GptLiveSampleUtils.printEvent(event); + } + } finally { + System.setOut(original); + } + assertEquals("Live session started; conversation_id=conversation-1" + System.lineSeparator(), + new String(output.toByteArray(), StandardCharsets.UTF_8)); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + @ResourceLock("java.lang.System.out") + public void startupWithoutPersistenceDoesNotSubstituteNativeSessionId(boolean interactive) { + ServerEvent event = typedEvent(BinaryData.fromString(STARTED)); + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + if (interactive) { + new GptLiveSampleUtils.AudioTranscriptPrinter(false, true).print(event); + } else { + GptLiveSampleUtils.printEvent(event); + } + } finally { + System.setOut(original); + } + assertEquals("Live session started; conversation_id not provided (persistence may be disabled)." + + System.lineSeparator(), new String(output.toByteArray(), StandardCharsets.UTF_8)); + } + + @Test + @ResourceLock("java.lang.System.out") + public void transcriptBufferIsBoundedWhileReadingInput() { + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + GptLiveSampleUtils.AudioTranscriptPrinter printer = new GptLiveSampleUtils.AudioTranscriptPrinter(); + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + printer.print( + typedEvent(GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "Previous response."))); + printer.beginInput(); + assertTrue(new String(output.toByteArray(), StandardCharsets.UTF_8) + .endsWith("Previous response." + System.lineSeparator() + "You: ")); + int promptLength = output.size(); + char[] characters = new char[GptLiveSampleUtils.AudioTranscriptPrinter.MAX_BUFFERED_TRANSCRIPT_CHARACTERS + - "Assistant: ".length()]; + Arrays.fill(characters, 'a'); + printer.print(typedEvent( + GptLiveSampleUtils.event("session.output_transcript.delta", "delta", new String(characters)))); + assertEquals(promptLength, output.size(), "Transcript output must not overwrite the input line"); + IllegalStateException error = assertThrows(IllegalStateException.class, () -> printer + .print(typedEvent(GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "x")))); + assertTrue(error.getMessage().contains("65536")); + printer.endInput(); + assertTrue(output.size() > promptLength, "Buffered text must be flushed after input ends"); + } finally { + System.setOut(original); + } + } + + @ParameterizedTest + @ValueSource(booleans = { true, false }) + @ResourceLock("java.lang.System.out") + public void consoleTranscriptUsesSingleLineBreaks(boolean inputEchoed) { + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + GptLiveSampleUtils.AudioTranscriptPrinter printer + = new GptLiveSampleUtils.AudioTranscriptPrinter(inputEchoed); + printer.print(typedEvent(GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "Welcome."))); + printer.beginInput(); + if (inputEchoed) { + System.out.println("testing"); + } + printer.print(typedEvent(GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "Reply."))); + printer.endInput(); + printer.beginInput(); + if (inputEchoed) { + System.out.println("exit"); + } + printer.endInput(); + printer.print(typedEvent(BinaryData.fromString(CLOSED))); + System.out.println("Live session closed."); + } finally { + System.setOut(original); + } + String newline = System.lineSeparator(); + assertEquals( + "Assistant: Welcome." + newline + "You: " + (inputEchoed ? "testing" : "") + newline + "Assistant: Reply." + + newline + "You: " + (inputEchoed ? "exit" : "") + newline + "Live session closed." + newline, + new String(output.toByteArray(), StandardCharsets.UTF_8)); + } + + @Test + @ResourceLock("java.lang.System.out") + public void partialUtteranceSurvivesInputSubmissionWithoutBeingPrinted() { + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + GptLiveSampleUtils.AudioTranscriptPrinter printer + = new GptLiveSampleUtils.AudioTranscriptPrinter(false, true); + printer.beginInput(); + printer.print( + typedEvent(GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "This sentence "))); + printer.endInput(); + assertFalse(new String(output.toByteArray(), StandardCharsets.UTF_8).contains("This sentence")); + printer.beginTurn(); + printer.beginInput(); + printer.print(typedEvent( + GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "continues after Enter."))); + printer.endUtterance(); + assertTrue(new String(output.toByteArray(), StandardCharsets.UTF_8).endsWith("You: ")); + printer.endInput(); + } finally { + System.setOut(original); + } + String newline = System.lineSeparator(); + assertEquals( + "You: " + newline + "You: " + newline + "Assistant: This sentence continues after Enter." + newline, + new String(output.toByteArray(), StandardCharsets.UTF_8)); + } + + @Test + @ResourceLock("java.lang.System.out") + public void completedAndPartialUtterancesShareTheTranscriptBufferLimit() { + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + GptLiveSampleUtils.AudioTranscriptPrinter printer + = new GptLiveSampleUtils.AudioTranscriptPrinter(false, true); + printer.beginInput(); + printer + .print(typedEvent(GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "Completed."))); + printer.endUtterance(); + int queuedLength = ("Assistant: Completed." + System.lineSeparator()).length(); + char[] characters + = new char[GptLiveSampleUtils.AudioTranscriptPrinter.MAX_BUFFERED_TRANSCRIPT_CHARACTERS - queuedLength]; + Arrays.fill(characters, 'a'); + printer.print(typedEvent( + GptLiveSampleUtils.event("session.output_transcript.delta", "delta", new String(characters)))); + assertThrows(IllegalStateException.class, () -> printer + .print(typedEvent(GptLiveSampleUtils.event("session.output_transcript.delta", "delta", "x")))); + printer.endInput(); + assertTrue(new String(output.toByteArray(), StandardCharsets.UTF_8).contains("Assistant: Completed.")); + assertFalse(new String(output.toByteArray(), StandardCharsets.UTF_8).contains("aaaa")); + } finally { + System.setOut(original); + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + @ResourceLock("java.lang.System.out") + public void spokenSentenceIsNotSplitByBackendCompletionOrAnotherTypedTurn(boolean async) { + AtomicInteger responses = new AtomicInteger(); + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.sendWebsocket((inbound, outbound) -> outbound + .sendString(Flux.concat(Mono.just(STARTED), inbound.receive().asString().concatMap(json -> { + Map command = GptLiveFunctionToolSample.object(BinaryData.fromString(json)); + commands.add(command); + if ("session.commentary.append".equals(command.get("type"))) { + return Flux.just("{\"type\":\"session.output_transcript.delta\",\"delta\":\"Welcome.\"}", + "{\"type\":\"session.output_audio.delta\",\"delta\":\"AQI=\"}"); + } + if ("response.create".equals(command.get("type"))) { + String fragment + = responses.incrementAndGet() == 1 ? "This sentence " : "continues across typed turns."; + return Flux.just( + GptLiveSampleUtils.event("session.output_transcript.delta", "delta", fragment).toString(), + "{\"type\":\"session.output_audio.delta\",\"delta\":\"AQI=\"}", COMPLETED); + } + return "session.close".equals(command.get("type")) ? Flux.just(CLOSED) : Flux.empty(); + }))) + .then())) + .bindNow(); + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + runInteractiveConversation(async, new StringReader("one\ntwo\nexit\n")); + } finally { + System.setOut(original); + } + assertEquals(2, responses.get()); + String transcript = new String(output.toByteArray(), StandardCharsets.UTF_8); + assertTrue(transcript.contains("Assistant: This sentence continues across typed turns.")); + assertFalse(transcript.contains("Assistant: continues")); + assertEquals(2, transcript.split("Assistant: ", -1).length - 1); + assertEquals(3, transcript.split("You: ", -1).length - 1, "Input must remain available between backend turns"); + } + + @Test + @ResourceLock("java.lang.System.out") + public void unknownLiveEventsDoNotInterruptTranscriptHandlers() { + ServerEvent event = typedEvent(GptLiveSampleUtils.event("session.future_event", "value", "future")); + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + GptLiveSampleUtils.printEvent(event); + new GptLiveSampleUtils.AudioTranscriptPrinter().print(event); + } finally { + System.setOut(original); + } + assertEquals(0, output.size()); + } + + @ParameterizedTest + @ValueSource(strings = { "response.failed", "response.incomplete" }) + public void responseFailuresAreReportedForUnknownAndDelegatedEvents(String type) { + BinaryData failure = GptLiveSampleUtils.event(type, "event_id", "failure-1"); + for (boolean delegated : new boolean[] { false, true }) { + Map wrapper = new LinkedHashMap<>(); + wrapper.put("type", "response.event"); + wrapper.put("event", GptLiveFunctionToolSample.object(failure)); + ServerEvent event = typedEvent(delegated ? BinaryData.fromObject(wrapper) : failure); + IllegalStateException error + = assertThrows(IllegalStateException.class, () -> GptLiveSampleUtils.printEvent(event)); + assertTrue(error.getMessage().contains(type)); + assertTrue(error.getMessage().contains("failure-1")); + assertThrows(IllegalStateException.class, + () -> new GptLiveSampleUtils.AudioTranscriptPrinter().print(event)); + } + } + + @Test + public void moderationIsReportedExplicitlyAndPreservesServiceDiagnostics() { + Map event = GptLiveFunctionToolSample.object(BinaryData.fromString(MODERATED)); + IllegalStateException error = assertThrows(IllegalStateException.class, + () -> new GptLiveSampleUtils.AudioTranscriptPrinter().print(typedEvent(BinaryData.fromString(MODERATED)))); + assertTrue(error.getMessage().contains("generation stopped by service moderation")); + assertTrue(error.getMessage().contains("content_filter")); + assertTrue(error.getMessage().contains("moderated-1")); + Map wrapper = new java.util.LinkedHashMap<>(); + wrapper.put("type", "response.event"); + wrapper.put("event", event); + assertThrows(IllegalStateException.class, + () -> new GptLiveSampleUtils.AudioTranscriptPrinter().print(typedEvent(BinaryData.fromObject(wrapper)))); + } + + @ParameterizedTest + @MethodSource("invalidSampleSettingCases") + @ResourceLock("com.azure.core.util.Configuration") + public void examplesValidateSettingsBeforeCreatingClients(String sampleName, Consumer main, + String setting, String value) { + Configuration configuration = Configuration.getGlobalConfiguration(); + Map original = new LinkedHashMap<>(); + for (String name : Arrays.asList("FOUNDRY_PROJECT_ENDPOINT", "FOUNDRY_GPT_LIVE_MODEL", + "FOUNDRY_GPT_LIVE_BACKEND_MODEL")) { + original.put(name, configuration.get(name)); + } + try { + configuration.put("FOUNDRY_PROJECT_ENDPOINT", "not-an-endpoint"); + configuration.put("FOUNDRY_GPT_LIVE_MODEL", "gpt-live-1"); + configuration.put("FOUNDRY_GPT_LIVE_BACKEND_MODEL", "gpt-5.6-terra"); + configuration.put(setting, value); + IllegalArgumentException error + = assertThrows(IllegalArgumentException.class, () -> main.accept(new String[0]), sampleName); + assertTrue(error.getMessage().contains(setting)); + } finally { + original.forEach((name, previous) -> { + if (previous == null) { + configuration.remove(name); + } else { + configuration.put(name, previous); + } + }); + } + } + + private static Stream invalidSampleSettingCases() { + Map> samples = new LinkedHashMap<>(); + samples.put("audio-async", GptLiveAudioConversationAsyncSample::main); + samples.put("text", GptLiveTextConversationSample::main); + samples.put("text-async", GptLiveTextConversationAsyncSample::main); + samples.put("tools", GptLiveFunctionToolSample::main); + return samples.entrySet() + .stream() + .flatMap(sample -> Stream + .of("FOUNDRY_PROJECT_ENDPOINT", "FOUNDRY_GPT_LIVE_MODEL", "FOUNDRY_GPT_LIVE_BACKEND_MODEL") + .flatMap(setting -> Stream.of("", " ") + .map(value -> Arguments.of(sample.getKey(), sample.getValue(), setting, value)))); + } + + @BeforeAll + static void installTlsCertificate() { + VoiceAgentWebSocketSessionTests.installTlsCertificate(); + } + + @AfterAll + static void restoreTlsConfiguration() { + VoiceAgentWebSocketSessionTests.restoreTlsConfiguration(); + } + + @AfterEach + void closeServer() { + if (server != null) { + server.disposeNow(); + } + } + + @Test + public void blockingConversationConsumesEventsWithoutSession() { + java.util.concurrent.BlockingQueue incoming = new java.util.concurrent.LinkedBlockingQueue<>(); + incoming.add(typedEvent(BinaryData.fromString(STARTED))); + Iterable events = () -> Stream.generate(() -> { + try { + return incoming.take(); + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("Event producer interrupted.", error); + } + }).iterator(); + GptLiveTextConversationSample.runConversation(events, command -> { + commands.add(GptLiveFunctionToolSample.object(OpenAIJsonHelper.toBinaryData(command))); + if (command.isResponseCreate()) { + incoming.add(typedEvent(BinaryData.fromString(COMPLETED))); + } else if (command.isSessionClose()) { + incoming.add(typedEvent(BinaryData.fromString(CLOSED))); + } + }, () -> "test close details", Arrays.asList("one", "two")); + assertEquals(Arrays.asList("response.item.create", "response.create", "response.item.create", "response.create", + "session.close"), commandTypes()); + } + + @Test + public void blockingConversationReportsCloseDetailsWithoutSession() { + IllegalStateException error = assertThrows(IllegalStateException.class, () -> GptLiveTextConversationSample + .runConversation(Collections.singletonList(typedEvent(BinaryData.fromString(CLOSED))), command -> { + }, () -> "transport close code=1006, reason=test disconnect", Collections.emptyList())); + assertTrue(error.getMessage().contains("transport close code=1006, reason=test disconnect")); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void conversationCoresUseTheSameLifecycleHooks(boolean async) { + List callbacks = new CopyOnWriteArrayList<>(); + runCoreWithCallbacks(async, () -> callbacks.add("started"), event -> { + assertTrue(event.isCompleted()); + callbacks.add("response"); + return false; + }, reason -> { + callbacks.add("close:" + reason); + return "expired".equals(reason); + }); + assertEquals(Arrays.asList("started", "response", "close:expired"), callbacks); + assertEquals(Arrays.asList("response.item.create", "response.create", "session.close"), commandTypes()); + } + + @ParameterizedTest + @MethodSource("coreCallbackFailureCases") + public void conversationCoresPropagateHookFailures(boolean async, String callback) { + IllegalStateException failure = new IllegalStateException("Failed " + callback); + IllegalStateException actual + = assertThrows(IllegalStateException.class, () -> runCoreWithCallbacks(async, () -> { + if ("started".equals(callback)) { + throw failure; + } + }, event -> { + if ("response".equals(callback)) { + throw failure; + } + return false; + }, reason -> { + throw failure; + })); + assertEquals(failure, actual); + assertEquals("started".equals(callback) + ? Collections.emptyList() + : "response".equals(callback) + ? Arrays.asList("response.item.create", "response.create") + : Arrays.asList("response.item.create", "response.create", "session.close"), + commandTypes()); + } + + private static Stream coreCallbackFailureCases() { + return Stream.of(false, true) + .flatMap( + async -> Stream.of("started", "response", "closed").map(callback -> Arguments.of(async, callback))); + } + + private void runCoreWithCallbacks(boolean async, Runnable onStarted, Predicate handleResponse, + Predicate acceptClose) { + java.util.concurrent.BlockingQueue blocking = new java.util.concurrent.LinkedBlockingQueue<>(); + Sinks.Many reactive = Sinks.many().unicast().onBackpressureBuffer(); + Consumer enqueue + = async ? event -> reactive.emitNext(event, Sinks.EmitFailureHandler.FAIL_FAST) : blocking::add; + enqueue.accept(typedEvent(BinaryData.fromString(STARTED))); + Consumer sendEvent = command -> { + commands.add(GptLiveFunctionToolSample.object(OpenAIJsonHelper.toBinaryData(command))); + if (command.isResponseCreate()) { + enqueue.accept(typedEvent(BinaryData.fromString(COMPLETED))); + } else if (command.isSessionClose()) { + enqueue + .accept(typedEvent(BinaryData.fromString("{\"type\":\"session.closed\",\"reason\":\"expired\"}"))); + } + }; + AtomicInteger prompts = new AtomicInteger(); + if (async) { + GptLiveTextConversationAsyncSample.runConversationCoreAsync(reactive.asFlux(), + command -> Mono.fromRunnable(() -> sendEvent.accept(command)), () -> "test close details", + transcripts -> Mono.justOrEmpty(prompts.getAndIncrement() == 0 ? "one" : null), pcm -> { + }, () -> { + }, false, false, false, () -> Mono.fromRunnable(onStarted), Mono.empty(), + event -> Mono.fromSupplier(() -> handleResponse.test(event)), acceptClose).block(TIMEOUT); + } else { + Iterable events = () -> Stream.generate(() -> { + try { + return blocking.take(); + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("Event producer interrupted.", error); + } + }).iterator(); + GptLiveTextConversationSample.runConversationCore(events, sendEvent, () -> "test close details", + transcripts -> prompts.getAndIncrement() == 0 ? "one" : null, pcm -> { + }, () -> { + }, false, onStarted, handleResponse, acceptClose); + } + } + + @Test + public void asyncResponseContinuationWaitsForHandlerBeforeProcessingNextEvent() { + Sinks.Many incoming = Sinks.many().unicast().onBackpressureBuffer(); + Sinks.Empty continuation = Sinks.empty(); + CountDownLatch handling = new CountDownLatch(1); + AtomicInteger responses = new AtomicInteger(); + AtomicInteger prompts = new AtomicInteger(); + incoming.emitNext(typedEvent(BinaryData.fromString(STARTED)), Sinks.EmitFailureHandler.FAIL_FAST); + Mono conversation + = GptLiveTextConversationAsyncSample.runConversationCoreAsync(incoming.asFlux(), command -> { + commands.add(GptLiveFunctionToolSample.object(OpenAIJsonHelper.toBinaryData(command))); + if (command.isResponseCreate()) { + incoming.emitNext(typedEvent(BinaryData.fromString(COMPLETED)), Sinks.EmitFailureHandler.FAIL_FAST); + } else if (command.isSessionClose()) { + incoming.emitNext(typedEvent(BinaryData.fromString(CLOSED)), Sinks.EmitFailureHandler.FAIL_FAST); + } + return Mono.empty(); + }, () -> "test close details", + transcripts -> Mono.justOrEmpty(prompts.getAndIncrement() == 0 ? "one" : null), pcm -> { + }, () -> { + }, false, false, false, Mono::empty, Mono.empty(), event -> { + if (responses.incrementAndGet() == 1) { + incoming.emitNext(typedEvent(BinaryData.fromString(COMPLETED)), + Sinks.EmitFailureHandler.FAIL_FAST); + handling.countDown(); + return continuation.asMono().thenReturn(true); + } + assertEquals(1, prompts.get(), "The next prompt must wait until the continuation completes"); + return Mono.just(false); + }, reason -> false); + StepVerifier.create(conversation).then(() -> { + try { + assertTrue(handling.await(TIMEOUT.toMillis(), TimeUnit.MILLISECONDS)); + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + throw new AssertionError(error); + } + assertEquals(1, responses.get(), "Later events must wait for the asynchronous handler"); + assertEquals(1, prompts.get()); + continuation.emitEmpty(Sinks.EmitFailureHandler.FAIL_FAST); + }).verifyComplete(); + assertEquals(2, responses.get()); + assertEquals(2, prompts.get()); + assertEquals(Arrays.asList("response.item.create", "response.create", "session.close"), commandTypes()); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void textSamplesWaitForStartupAndCompleteMultipleTurns(boolean async) { + startTextServer(false, false); + if (async) { + Mono.usingWhen( + builder().beta().buildBetaVoiceAgentsLiveWebSocketAsyncClient().openWebSocketSession("agent"), + session -> GptLiveTextConversationAsyncSample.runConversation(session, Arrays.asList("one", "two")), + BetaVoiceAgentsLiveWebSocketSessionAsyncClient::closeAsync).block(TIMEOUT); + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + GptLiveTextConversationSample.runConversation(session, Arrays.asList("one", "two")); + } + } + List expected = Arrays.asList("response.item.create", "response.create", "response.item.create", + "response.create", "session.close"); + assertEquals(expected, commandTypes()); + Map item = GptLiveFunctionToolSample.object(commands.get(0).get("item")); + assertEquals("user", item.get("role")); + assertEquals("message", item.get("type")); + assertEquals("one", GptLiveFunctionToolSample.object(((List) item.get("content")).get(0)).get("text")); + } + + @ParameterizedTest + @MethodSource("interactiveExitCases") + public void interactiveTextSamplesReadMultipleTurnsAndCloseOnExit(boolean async, String exit) { + startTextServer(false, false); + AtomicBoolean inputClosed = new AtomicBoolean(); + Reader input = new StringReader("one\r\ntwo\n" + exit) { + @Override + public void close() { + inputClosed.set(true); + super.close(); + } + }; + runInteractiveConversation(async, input); + assertEquals(Arrays.asList("session.commentary.append", "response.item.create", "response.create", + "response.item.create", "response.create", "session.close"), commandsWithoutInputAudio()); + List sentPrompts = commands.stream() + .filter(command -> "response.item.create".equals(command.get("type"))) + .map(command -> GptLiveFunctionToolSample.object(command.get("item"))) + .map(item -> GptLiveFunctionToolSample.object(((List) item.get("content")).get(0))) + .map(part -> GptLiveFunctionToolSample.string(part, "text")) + .collect(java.util.stream.Collectors.toList()); + assertEquals(Arrays.asList("one", "two"), sentPrompts); + assertFalse(inputClosed.get(), "Conversation cleanup must not close standard input"); + } + + @ParameterizedTest + @MethodSource("interactiveExitCases") + public void interactiveTextSamplesCanExitWithoutSendingAPrompt(boolean async, String text) { + startTextServer(false, false); + runInteractiveConversation(async, new StringReader(text)); + assertEquals(Arrays.asList("session.commentary.append", "session.close"), commandsWithoutInputAudio()); + } + + private static Stream interactiveExitCases() { + return Stream.of(false, true) + .flatMap(async -> Stream.of("", "\n", "exit\n", " QuIt \r\n").map(exit -> Arguments.of(async, exit))); + } + + private List commandsWithoutInputAudio() { + return commandTypes().stream() + .filter(type -> !"session.input_audio.append".equals(type)) + .collect(java.util.stream.Collectors.toList()); + } + + private void runInteractiveConversation(boolean async, Reader input) { + runInteractiveConversation(async, input, pcm -> { + }, () -> { + }); + } + + private void runInteractiveConversation(boolean async, Reader input, Consumer playAudio, + Runnable drainAudio) { + if (async) { + Mono.usingWhen( + builder().beta().buildBetaVoiceAgentsLiveWebSocketAsyncClient().openWebSocketSession("agent"), + session -> GptLiveTextConversationAsyncSample.runInteractiveConversationAsync(session, input, playAudio, + drainAudio), + BetaVoiceAgentsLiveWebSocketSessionAsyncClient::closeAsync).block(TIMEOUT); + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + GptLiveTextConversationSample.runInteractiveConversation(session, input, playAudio, drainAudio); + } + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void interactiveTextSampleReceivesServiceErrorsWhileWaitingForInput(boolean async) { + Sinks.Empty inputStarted = Sinks.empty(); + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.sendWebsocket((inbound, outbound) -> outbound + .sendString(Flux.concat(Mono.just(STARTED), inbound.receive().asString().concatMap(json -> { + Map command = GptLiveFunctionToolSample.object(BinaryData.fromString(json)); + return "session.commentary.append".equals(command.get("type")) + ? Flux.just("{\"type\":\"session.output_transcript.delta\",\"delta\":\"Welcome.\"}", + "{\"type\":\"session.output_audio.delta\",\"delta\":\"AQI=\"}") + : Flux.empty(); + })).mergeWith(inputStarted.asMono().thenReturn(MODERATED))) + .then())) + .bindNow(); + CountDownLatch releaseRead = new CountDownLatch(1); + Reader idleInput = new Reader() { + @Override + public boolean ready() { + throw new AssertionError("Console input must use a blocking line read, not readiness polling"); + } + + @Override + public int read(char[] buffer, int offset, int length) throws IOException { + inputStarted.tryEmitEmpty(); + try { + releaseRead.await(); + return -1; + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + throw new IOException("Console read interrupted", error); + } + } + + @Override + public void close() { + throw new AssertionError("Standard input must not be closed"); + } + }; + try { + StepVerifier + .create(Mono.fromRunnable(() -> runInteractiveConversation(async, idleInput)) + .subscribeOn(Schedulers.boundedElastic())) + .expectErrorMatches( + error -> error instanceof IllegalStateException && error.getMessage().contains("content_filter")) + .verify(TIMEOUT); + } finally { + releaseRead.countDown(); + } + } + + @Test + public void consoleReadFailuresPropagate() { + Reader input = new StringReader("") { + @Override + public int read(char[] buffer, int offset, int length) throws IOException { + throw new IOException("Console unavailable"); + } + }; + StepVerifier.create(GptLiveTextConversationAsyncSample.readPromptAsync(input)) + .expectErrorMatches( + error -> error instanceof IOException && "Console unavailable".equals(error.getMessage())) + .verify(TIMEOUT); + } + + @Test + public void consoleInputIsSubmittedOnlyAfterEnterWithoutLosingCharacters() throws Exception { + try (PipedReader input = new PipedReader(); PipedWriter keyboard = new PipedWriter(input)) { + CompletableFuture prompt = GptLiveTextConversationAsyncSample.readPromptAsync(input).toFuture(); + try { + keyboard.write("testtesttes testing"); + keyboard.flush(); + assertThrows(TimeoutException.class, () -> prompt.get(200, TimeUnit.MILLISECONDS)); + keyboard.write("\r\n"); + keyboard.flush(); + assertEquals("testtesttes testing", prompt.get(5, TimeUnit.SECONDS)); + } finally { + prompt.cancel(true); + } + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + @ResourceLock("java.lang.System.out") + public void greetingIsPrintedAndPlayedBeforeReadingInput(boolean async) throws Exception { + Sinks.Empty allowGreeting = Sinks.empty(); + CountDownLatch greetingRequested = new CountDownLatch(1); + CountDownLatch inputStarted = new CountDownLatch(1); + startTextServer(false, false, Mono.fromRunnable(greetingRequested::countDown).then(allowGreeting.asMono())); + CountDownLatch greetingPlayed = new CountDownLatch(1); + CountDownLatch replyPlayed = new CountDownLatch(1); + CountDownLatch drainStarted = new CountDownLatch(1); + CountDownLatch allowDrain = new CountDownLatch(1); + AtomicBoolean drained = new AtomicBoolean(); + AtomicInteger played = new AtomicInteger(); + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PipedReader pipe = new PipedReader(); + PipedWriter keyboard = new PipedWriter(pipe); + PrintStream capture = new PrintStream(output)) { + Reader input = new FilterReader(pipe) { + @Override + public int read(char[] buffer, int offset, int length) throws IOException { + assertTrue(drained.get(), "Queued greeting audio must finish before input begins"); + inputStarted.countDown(); + return super.read(buffer, offset, length); + } + }; + System.setOut(capture); + CompletableFuture conversation + = Mono.fromRunnable(() -> runInteractiveConversation(async, input, pcm -> { + if (played.incrementAndGet() == 1) { + greetingPlayed.countDown(); + } else { + replyPlayed.countDown(); + } + }, () -> { + drainStarted.countDown(); + try { + assertTrue(allowDrain.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + throw new AssertionError(error); + } + drained.set(true); + })).subscribeOn(Schedulers.boundedElastic()).toFuture(); + try { + assertTrue(greetingRequested.await(5, TimeUnit.SECONDS)); + assertFalse(inputStarted.await(200, TimeUnit.MILLISECONDS)); + allowGreeting.tryEmitEmpty(); + assertTrue(greetingPlayed.await(5, TimeUnit.SECONDS)); + assertFalse(inputStarted.await(200, TimeUnit.MILLISECONDS), + "The quiet period must elapse before prompting"); + assertTrue(drainStarted.await(5, TimeUnit.SECONDS)); + assertFalse(inputStarted.await(200, TimeUnit.MILLISECONDS), + "The first prompt must wait for queued speaker playback"); + allowDrain.countDown(); + assertTrue(inputStarted.await(5, TimeUnit.SECONDS)); + String whileTyping = new String(output.toByteArray(), StandardCharsets.UTF_8); + assertTrue(whileTyping.endsWith("You: ")); + assertTrue(whileTyping.contains("Assistant: Welcome." + System.lineSeparator() + "You: ")); + keyboard.write("testing\n"); + keyboard.flush(); + assertTrue(replyPlayed.await(5, TimeUnit.SECONDS)); + keyboard.write("exit\n"); + keyboard.flush(); + conversation.get(5, TimeUnit.SECONDS); + String transcript = new String(output.toByteArray(), StandardCharsets.UTF_8); + assertTrue(transcript.indexOf("Assistant: Welcome.") < transcript.indexOf("You: ")); + assertTrue(transcript.contains("Spoken reply.")); + assertFalse(transcript.contains(System.lineSeparator() + System.lineSeparator())); + } finally { + allowDrain.countDown(); + conversation.cancel(true); + } + } finally { + System.setOut(original); + } + } + + @Test + public void greetingWaitsForBothOutputsAndRestartsQuietPeriodForEachChunk() { + AtomicReference greeting = new AtomicReference<>(); + AtomicBoolean drained = new AtomicBoolean(); + StepVerifier.withVirtualTime(() -> { + GptLiveSampleUtils.Greeting pending = new GptLiveSampleUtils.Greeting(); + greeting.set(pending); + return pending.awaitPlaybackAsync(() -> drained.set(true)); + }) + .expectSubscription() + .then(() -> greeting.get().onTranscript()) + .expectNoEvent(Duration.ofSeconds(2)) + .then(() -> greeting.get().onAudio(new byte[] { 1, 2 })) + .expectNoEvent(Duration.ofMillis(750)) + .then(() -> greeting.get().onAudio(new byte[] { 1, 2 })) + .expectNoEvent(Duration.ofMillis(750)) + .then(() -> greeting.get().onTranscript()) + .expectNoEvent(Duration.ofMillis(999)) + .then(() -> assertFalse(drained.get())) + .thenAwait(Duration.ofMillis(1)) + .verifyComplete(); + assertTrue(drained.get()); + } + + @ParameterizedTest + @ValueSource(strings = { "session.output_transcript.delta", "session.output_audio.delta" }) + public void missingGreetingOutputTimesOutInsteadOfStartingInput(String type) { + AtomicBoolean drained = new AtomicBoolean(); + StepVerifier.withVirtualTime(() -> { + GptLiveSampleUtils.Greeting greeting = new GptLiveSampleUtils.Greeting(); + if ("session.output_audio.delta".equals(type)) { + greeting.onAudio(new byte[] { 1, 2 }); + } else { + greeting.onTranscript(); + } + return greeting.awaitPlaybackAsync(() -> drained.set(true)); + }).expectSubscription().thenAwait(GptLiveSampleUtils.TIMEOUT).expectError(TimeoutException.class).verify(); + assertFalse(drained.get()); + } + + @Test + public void greetingDrainFailureIsReported() { + StepVerifier.withVirtualTime(() -> { + GptLiveSampleUtils.Greeting greeting = new GptLiveSampleUtils.Greeting(); + greeting.onAudio(new byte[] { 1, 2 }); + greeting.onTranscript(); + return greeting.awaitPlaybackAsync(() -> { + throw new IllegalStateException("Speaker drain failed"); + }); + }).expectSubscription().thenAwait(Duration.ofSeconds(1)).expectErrorMessage("Speaker drain failed").verify(); + } + + @Test + public void continuousSilentOutputDoesNotKeepTheGreetingOpen() { + AtomicBoolean drained = new AtomicBoolean(); + StepVerifier.withVirtualTime(() -> { + GptLiveSampleUtils.Greeting greeting = new GptLiveSampleUtils.Greeting(); + greeting.onAudio(new byte[] { 1, 2 }); + greeting.onTranscript(); + Mono silence = Flux.interval(Duration.ofMillis(20)) + .take(50) + .doOnNext(tick -> greeting.onAudio(new byte[960])) + .then(); + return Mono.when(silence, greeting.awaitPlaybackAsync(() -> drained.set(true))); + }).expectSubscription().thenAwait(Duration.ofSeconds(1)).verifyComplete(); + assertTrue(drained.get()); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + @ResourceLock("java.lang.System.out") + public void continuousSilentAudioStillAllowsInputAfterTheGreeting(boolean async) { + startTextServer(false, false, Mono.empty(), true); + AtomicInteger silentFrames = new AtomicInteger(); + AtomicBoolean readingInput = new AtomicBoolean(); + CountDownLatch silenceWhileReading = new CountDownLatch(1); + Reader input = new StringReader("exit\n") { + @Override + public int read(char[] buffer, int offset, int length) throws IOException { + assertTrue(silentFrames.get() > 0, "Silent output must arrive before input begins"); + readingInput.set(true); + try { + assertTrue(silenceWhileReading.await(5, TimeUnit.SECONDS), + "Silent output must continue while reading input"); + return super.read(buffer, offset, length); + } catch (InterruptedException error) { + Thread.currentThread().interrupt(); + throw new IOException("Console read interrupted", error); + } finally { + readingInput.set(false); + } + } + }; + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + runInteractiveConversation(async, input, pcm -> { + if (pcm.length == 960) { + assertArrayEquals(new byte[960], pcm); + silentFrames.incrementAndGet(); + if (readingInput.get()) { + silenceWhileReading.countDown(); + } + } + }, () -> { + }); + } finally { + System.setOut(original); + } + String transcript = new String(output.toByteArray(), StandardCharsets.UTF_8); + assertTrue(transcript.contains("Assistant: Welcome." + System.lineSeparator() + "You: "), + () -> "Expected greeting followed by input prompt (async=" + async + "). Actual transcript: " + + transcript.replace("\r", "\\r").replace("\n", "\\n")); + assertEquals(Arrays.asList("session.commentary.append", "session.close"), commandsWithoutInputAudio()); + } + + @ParameterizedTest + @ValueSource(ints = { 0, 256, -256, 257, -257, 32767, -32768 }) + public void greetingSpeechDetectionHonorsSignedPcm16PeakThreshold(int peak) { + boolean audible = Math.abs(peak) > GptLiveSampleUtils.Greeting.SILENCE_PEAK_THRESHOLD; + AtomicBoolean drained = new AtomicBoolean(); + StepVerifier.Step verification = StepVerifier.withVirtualTime(() -> { + GptLiveSampleUtils.Greeting greeting = new GptLiveSampleUtils.Greeting(); + greeting.onTranscript(); + greeting.onAudio(new byte[] { (byte) peak, (byte) (peak >> 8) }); + return greeting.awaitPlaybackAsync(() -> drained.set(true)); + }).expectSubscription().thenAwait(audible ? Duration.ofSeconds(1) : GptLiveSampleUtils.TIMEOUT); + if (audible) { + verification.verifyComplete(); + } else { + verification.expectError(TimeoutException.class).verify(); + } + assertEquals(audible, drained.get()); + } + + @ParameterizedTest + @ValueSource(strings = { "", "\r\n", "exit\n", "quit\n" }) + public void consoleEofBlankAndExitProduceNoPrompt(String input) { + StepVerifier.create(GptLiveTextConversationAsyncSample.readPromptAsync(new StringReader(input))) + .verifyComplete(); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void interactiveTextPlaybackFailureTerminatesTheConversation(boolean async) { + startTextServer(false, false); + StepVerifier.create( + Mono.fromRunnable(() -> runInteractiveConversation(async, new StringReader("testing\n"), pcm -> { + throw new IllegalStateException("Speaker failed"); + }, () -> { + })).subscribeOn(Schedulers.boundedElastic())) + .expectErrorMatches( + error -> error instanceof IllegalStateException && "Speaker failed".equals(error.getMessage())) + .verify(TIMEOUT); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + @ResourceLock("java.lang.System.out") + public void interactiveTextPlaysGreetingAndRepliesWithoutPrintingBackendNotes(boolean async) { + startTextServer(false, false); + List played = new CopyOnWriteArrayList<>(); + ByteArrayOutputStream output = new ByteArrayOutputStream(); + PrintStream original = System.out; + try (PrintStream capture = new PrintStream(output)) { + System.setOut(capture); + runInteractiveConversation(async, new StringReader("testing\nexit\n"), played::add, () -> { + }); + } finally { + System.setOut(original); + } + assertEquals(2, played.size()); + assertArrayEquals(new byte[] { 1, 2 }, played.get(0)); + assertArrayEquals(new byte[] { 1, 2 }, played.get(1)); + String transcript = new String(output.toByteArray(), StandardCharsets.UTF_8); + assertTrue(transcript.contains("Assistant: Welcome.")); + assertTrue(transcript.contains("Spoken reply.")); + assertFalse(transcript.contains("Hello"), "Backend text must not be printed as the spoken answer"); + assertFalse(transcript.contains("[GPT Live diagnostics]")); + assertFalse(transcript.contains("\"type\":\"session.closed\"")); + Map greeting = commands.stream() + .filter(command -> "session.commentary.append".equals(command.get("type"))) + .findFirst() + .get(); + assertEquals("session.commentary.append", greeting.get("type")); + assertEquals("Hi, how can I help you today?", greeting.get("content")); + assertTrue(greeting.containsKey("delegation_id")); + assertEquals(null, greeting.get("delegation_id")); + assertEquals("session.input_audio.append", commands.get(0).get("type")); + assertEquals("session.close", commands.get(commands.size() - 1).get("type")); + } + + @Test + public void silentAudioFramesArePacedAtRealTimePcmRate() { + StepVerifier.withVirtualTime(() -> GptLiveSampleUtils.silenceFramesAsync().take(3)) + .expectSubscription() + .thenAwait(Duration.ZERO) + .assertNext(this::assertSilentAudioFrame) + .expectNoEvent(Duration.ofMillis(19)) + .thenAwait(Duration.ofMillis(1)) + .assertNext(this::assertSilentAudioFrame) + .expectNoEvent(Duration.ofMillis(19)) + .thenAwait(Duration.ofMillis(1)) + .assertNext(this::assertSilentAudioFrame) + .verifyComplete(); + } + + @Test + public void silentAudioFramesWaitForDemandWithoutTimerOverflow() { + StepVerifier.withVirtualTime(() -> GptLiveSampleUtils.silenceFramesAsync().take(3), 0) + .expectSubscription() + .expectNoEvent(Duration.ofSeconds(40)) + .thenRequest(1) + .assertNext(this::assertSilentAudioFrame) + .expectNoEvent(Duration.ofSeconds(40)) + .thenRequest(1) + .thenAwait(Duration.ofMillis(20)) + .assertNext(this::assertSilentAudioFrame) + .expectNoEvent(Duration.ofSeconds(40)) + .thenRequest(1) + .thenAwait(Duration.ofMillis(20)) + .assertNext(this::assertSilentAudioFrame) + .verifyComplete(); + } + + @Test + public void silentAudioToleratesSendsSlowerThanItsFrameRate() { + StepVerifier + .withVirtualTime(() -> GptLiveSampleUtils.silenceFramesAsync() + .concatMap(frame -> Mono.delay(Duration.ofMillis(75)).thenReturn(frame), 1) + .take(500)) + .expectSubscription() + .thenAwait(Duration.ofSeconds(40)) + .expectNextCount(500) + .verifyComplete(); + } + + private void assertSilentAudioFrame(ClientEvent frame) { + assertTrue(frame.isSessionInputAudioAppend()); + assertArrayEquals(new byte[960], Base64.getDecoder().decode(frame.asSessionInputAudioAppend().audio())); + } + + @ParameterizedTest + @MethodSource("interactiveClosureCases") + public void interactiveTextPreservesServerClosureReason(boolean async, String reason) { + String closed = "{\"type\":\"session.closed\",\"reason\":\"" + reason + "\",\"event_id\":\"closed-1\"}"; + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.sendWebsocket((inbound, outbound) -> outbound + .sendString(Flux.concat(Mono.just(STARTED), inbound.receive().asString().concatMap(json -> { + Map command = GptLiveFunctionToolSample.object(BinaryData.fromString(json)); + commands.add(command); + if ("session.commentary.append".equals(command.get("type"))) { + return Flux.just("{\"type\":\"session.output_transcript.delta\",\"delta\":\"Welcome.\"}", + "{\"type\":\"session.output_audio.delta\",\"delta\":\"AQI=\"}"); + } + return "response.create".equals(command.get("type")) ? Mono.just(closed) : Mono.empty(); + }))) + .then())) + .bindNow(); + if ("remote_hangup".equals(reason)) { + runInteractiveConversation(async, new StringReader("testing\n")); + } else { + IllegalStateException error = assertThrows(IllegalStateException.class, + () -> runInteractiveConversation(async, new StringReader("testing\n"))); + assertTrue(error.getMessage().contains(reason)); + assertTrue(error.getMessage().contains("closed-1")); + } + assertFalse(commandTypes().contains("session.close")); + } + + private static Stream interactiveClosureCases() { + return Stream.of(false, true) + .flatMap(async -> Stream.of("remote_hangup", "expired", "content", "connection_lost") + .map(reason -> Arguments.of(async, reason))); + } + + @Test + public void synchronousConsoleReadFailuresPropagateAndReleaseTheSession() { + startTextServer(false, false); + IOException failure = new IOException("Console unavailable"); + Reader input = new Reader() { + @Override + public int read(char[] buffer, int offset, int length) throws IOException { + throw failure; + } + + @Override + public void close() { + throw new AssertionError("Standard input must not be closed"); + } + }; + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + IllegalStateException error = assertThrows(IllegalStateException.class, + () -> GptLiveTextConversationSample.runInteractiveConversation(session, input, pcm -> { + }, () -> { + })); + assertEquals("Unable to read console input.", error.getMessage()); + assertEquals(failure, error.getCause()); + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void serviceErrorsFailBothTextSamples(boolean async) { + startTextServer(false, true); + if (async) { + StepVerifier + .create(Mono.usingWhen( + builder().beta().buildBetaVoiceAgentsLiveWebSocketAsyncClient().openWebSocketSession("agent"), + session -> GptLiveTextConversationAsyncSample.runConversation(session, + Collections.singletonList("one")), + BetaVoiceAgentsLiveWebSocketSessionAsyncClient::closeAsync)) + .expectErrorMatches(error -> error instanceof IllegalStateException + && error.getMessage().contains("GPT Live service error")) + .verify(TIMEOUT); + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + assertThrows(IllegalStateException.class, + () -> GptLiveTextConversationSample.runConversation(session, Collections.singletonList("one"))); + } + } + assertFalse(commandTypes().contains("session.close")); + } + + @Test + public void toolsUseGranularEventsAndReturnResultsBeforeContinuation() { + startTextServer(true, false); + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + GptLiveFunctionToolSample.runConversation(session, Collections.singletonList("Weather in Seattle?"), + GptLiveFunctionToolSample::toolResult); + } + assertEquals(Arrays.asList("response.item.create", "response.create", "response.item.create", "response.create", + "session.close"), commandTypes()); + Map item = GptLiveFunctionToolSample.object(commands.get(2).get("item")); + assertEquals("function_call_output", item.get("type")); + assertEquals("call-1", item.get("call_id")); + Map result = GptLiveFunctionToolSample.object(BinaryData.fromString((String) item.get("output"))); + assertEquals("Seattle", result.get("city")); + assertEquals(true, result.get("simulated")); + assertFalse(item.containsKey("delegation_id")); + } + + @Test + public void toolHandlerFailureStopsConversationBeforeContinuation() { + startTextServer(true, false); + IllegalStateException failure = new IllegalStateException("Tool execution failed"); + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + IllegalStateException error = assertThrows(IllegalStateException.class, () -> GptLiveFunctionToolSample + .runConversation(session, Collections.singletonList("Weather in Seattle?"), call -> { + throw failure; + })); + assertEquals(failure, error); + } + assertEquals(Arrays.asList("response.item.create", "response.create"), commandTypes()); + } + + @Test + public void toolSampleHandlesMultipleTurns() { + startTextServer(true, false); + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + GptLiveFunctionToolSample.runConversation(session, Arrays.asList("Weather in Seattle?", "Thanks!"), + GptLiveFunctionToolSample::toolResult); + } + assertEquals(Arrays.asList("response.item.create", "response.create", "response.item.create", "response.create", + "response.item.create", "response.create", "session.close"), commandTypes()); + Map followUp = GptLiveFunctionToolSample.object(commands.get(4).get("item")); + assertEquals("user", followUp.get("role")); + assertEquals("Thanks!", + GptLiveFunctionToolSample.object(((List) followUp.get("content")).get(0)).get("text")); + } + + @Test + public void toolSchemaAndArgumentsAreValidated() { + Map tool + = GptLiveFunctionToolSample.object(OpenAIJsonHelper.toBinaryData(GptLiveFunctionToolSample.weatherTool())); + assertEquals("function", tool.get("type")); + assertEquals("get_weather", tool.get("name")); + assertEquals(true, tool.get("strict")); + assertEquals(false, GptLiveFunctionToolSample.object(tool.get("parameters")).get("additionalProperties")); + ResponseFunctionToolCall unknown + = ResponseFunctionToolCall.builder().name("unknown").callId("c").arguments("{}").build(); + assertThrows(IllegalArgumentException.class, () -> GptLiveFunctionToolSample.toolResult(unknown)); + ResponseFunctionToolCall invalid + = ResponseFunctionToolCall.builder().name("get_weather").callId("c").arguments("{}").build(); + assertThrows(IllegalArgumentException.class, () -> GptLiveFunctionToolSample.toolResult(invalid)); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void prematureClosureDoesNotReportSuccessfulConversation(boolean async) { + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.sendWebsocket( + (inbound, outbound) -> outbound.sendString(Flux.just(STARTED, CLOSED)).then(inbound.receive().then()))) + .bindNow(); + if (async) { + StepVerifier + .create(Mono.usingWhen( + builder().beta().buildBetaVoiceAgentsLiveWebSocketAsyncClient().openWebSocketSession("agent"), + session -> GptLiveTextConversationAsyncSample.runConversation(session, + Collections.singletonList("one")), + BetaVoiceAgentsLiveWebSocketSessionAsyncClient::closeAsync)) + .expectError(IllegalStateException.class) + .verify(TIMEOUT); + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + assertThrows(IllegalStateException.class, + () -> GptLiveTextConversationSample.runConversation(session, Collections.singletonList("one"))); + } + } + } + + @Test + public void typedCommentaryPreservesExplicitNullDelegationId() throws IOException { + ClientEvent event = GptLiveSampleUtils.commentary("Hello"); + assertTrue(event.isSessionCommentaryAppend()); + BinaryData data = OpenAIJsonHelper.toBinaryData(event); + Map command = GptLiveFunctionToolSample.object(data); + assertEquals("session.commentary.append", command.get("type")); + assertEquals("Hello", command.get("content")); + assertTrue(command.containsKey("delegation_id")); + assertEquals(null, command.get("delegation_id")); + assertEquals(command, data.toObject(Map.class)); + assertEquals(command, GptLiveFunctionToolSample.object(BinaryData.fromBytes(data.toBytes()))); + ByteArrayOutputStream output = new ByteArrayOutputStream(); + try (com.azure.json.JsonWriter writer = com.azure.json.JsonProviders.createWriter(output)) { + data.writeTo(writer); + } + assertEquals(command, GptLiveFunctionToolSample.object(BinaryData.fromBytes(output.toByteArray()))); + } + + @Test + public void nativeAudioEncodingAndDecodingPreservePcmBytes() { + byte[] pcm = new byte[] { 0, 1, -1, 127 }; + ClientEvent command = GptLiveSampleUtils.audio(pcm); + assertTrue(command.isSessionInputAudioAppend()); + assertArrayEquals(pcm, Base64.getDecoder().decode(command.asSessionInputAudioAppend().audio())); + ServerEvent event = typedEvent( + GptLiveSampleUtils.event("session.output_audio.delta", "delta", Base64.getEncoder().encodeToString(pcm))); + assertArrayEquals(pcm, GptLiveSampleUtils.decodeAudio(event)); + assertThrows(IllegalArgumentException.class, () -> GptLiveSampleUtils + .decodeAudio(typedEvent(GptLiveSampleUtils.event("session.output_audio.delta", "delta", "%%%")))); + assertThrows(IllegalArgumentException.class, () -> GptLiveSampleUtils + .decodeAudio(typedEvent(GptLiveSampleUtils.event("session.output_audio.delta", "delta", "AA==")))); + } + + private static ServerEvent typedEvent(BinaryData data) { + return OpenAIJsonHelper.fromBinaryData(data, ServerEvent.class); + } + + private static Stream audioSessionCases() { + return Stream.of(false, true) + .flatMap(moderated -> Stream.of(false, true).map(store -> Arguments.of(moderated, store))); + } + + @ParameterizedTest + @MethodSource("audioSessionCases") + public void audioSampleGreetsCapturesPlaysAndReleasesDevices(boolean moderated, boolean store) { + AtomicBoolean captured = new AtomicBoolean(); + AtomicInteger closed = new AtomicInteger(); + CountDownLatch unblockRead = new CountDownLatch(1); + AtomicReference played = new AtomicReference<>(); + AtomicBoolean firstRead = new AtomicBoolean(true); + TargetDataLine microphone = (TargetDataLine) Proxy.newProxyInstance(getClass().getClassLoader(), + new Class[] { TargetDataLine.class }, (proxy, method, args) -> { + if ("read".equals(method.getName())) { + if (firstRead.compareAndSet(true, false)) { + byte[] bytes = (byte[]) args[0]; + bytes[0] = 1; + bytes[1] = 2; + return 2; + } + unblockRead.await(5, TimeUnit.SECONDS); + return 0; + } else if ("close".equals(method.getName())) { + closed.incrementAndGet(); + unblockRead.countDown(); + } + return null; + }); + SourceDataLine speaker = (SourceDataLine) Proxy.newProxyInstance(getClass().getClassLoader(), + new Class[] { SourceDataLine.class }, (proxy, method, args) -> { + if ("write".equals(method.getName())) { + played.set(((byte[]) args[0]).clone()); + return args[2]; + } else if ("close".equals(method.getName())) { + closed.incrementAndGet(); + } + return null; + }); + AtomicReference requestUri = new AtomicReference<>(); + String started = store + ? "{\"type\":\"session.started\",\"conversation_id\":\"conversation-1\",\"session\":{\"id\":\"live-1\"}}" + : STARTED; + server = VoiceAgentWebSocketSessionTests.tlsServer().host("localhost").port(0).handle((request, response) -> { + requestUri.set(request.uri()); + return response + .sendWebsocket( + (inbound, + outbound) -> outbound.sendString(Flux.concat( + Flux.just(started, "{\"type\":\"session.output_audio.delta\",\"delta\":\"AQI=\"}"), + inbound.receive().asString().concatMap(json -> { + Map command = GptLiveFunctionToolSample.object(BinaryData.fromString(json)); + commands.add(command); + if ("session.input_audio.append".equals(command.get("type"))) { + captured.set(true); + } + if (moderated && "response.create".equals(command.get("type"))) { + return Mono.just(MODERATED); + } + return "session.close".equals(command.get("type")) ? Mono.just(CLOSED) : Mono.empty(); + }))).then()); + }).bindNow(); + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = builder().beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent", new VoiceAgentLiveWebSocketConnectionOptions().setStoreEnabled(store)) + .block(TIMEOUT); + assertNotNull(session); + assertTrue(requestUri.get().contains("store=" + store), requestUri.get()); + InputStream enterAfterAudio = new InputStream() { + @Override + public int available() { + return captured.get() && played.get() != null && commandTypes().contains("response.create") ? 1 : 0; + } + + @Override + public int read() { + return '\n'; + } + }; + try { + GptLiveSampleUtils.AudioProcessor processor = new GptLiveSampleUtils.AudioProcessor(microphone, speaker, + pcm -> session.sendEvent(GptLiveSampleUtils.audio(pcm.toBytes()))); + Mono conversation + = GptLiveAudioConversationAsyncSample.runAudioConversationAsync(session, processor, enterAfterAudio); + if (moderated) { + IllegalStateException error + = assertThrows(IllegalStateException.class, () -> conversation.block(TIMEOUT)); + assertTrue(error.getMessage().contains("content_filter")); + assertTrue(error.getMessage().contains("moderated-1")); + } else { + conversation.block(TIMEOUT); + assertTrue(captured.get()); + assertArrayEquals(new byte[] { 1, 2 }, played.get()); + assertTrue(commandTypes().contains("session.close")); + } + assertEquals(2, closed.get()); + List greetingCommands = commandTypes().stream() + .filter(type -> "response.item.create".equals(type) || "response.create".equals(type)) + .collect(java.util.stream.Collectors.toList()); + assertEquals(Arrays.asList("response.item.create", "response.create"), greetingCommands); + Map greeting = commands.stream() + .filter(command -> "response.item.create".equals(command.get("type"))) + .findFirst() + .get(); + Map item = GptLiveFunctionToolSample.object(greeting.get("item")); + assertEquals("Greet me briefly with: Hi, how can I help you today?", + GptLiveFunctionToolSample.object(((List) item.get("content")).get(0)).get("text")); + } finally { + session.closeAsync().block(TIMEOUT); + } + } + + private List commandTypes() { + return commands.stream() + .map(command -> GptLiveFunctionToolSample.string(command, "type")) + .collect(java.util.stream.Collectors.toList()); + } + + private AgentsClientBuilder builder() { + return new AgentsClientBuilder().endpoint("https://localhost:" + server.port()) + .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))) + .configuration(Configuration.NONE); + } + + private void startTextServer(boolean tools, boolean fail) { + startTextServer(tools, fail, Mono.empty()); + } + + private void startTextServer(boolean tools, boolean fail, Mono beforeGreeting) { + startTextServer(tools, fail, beforeGreeting, false); + } + + private void startTextServer(boolean tools, boolean fail, Mono beforeGreeting, boolean continuousSilence) { + AtomicInteger responses = new AtomicInteger(); + AtomicInteger silentInputs = new AtomicInteger(); + String silentOutput = "{\"type\":\"session.output_audio.delta\",\"delta\":\"" + + Base64.getEncoder().encodeToString(new byte[960]) + "\"}"; + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, + response) -> response.sendWebsocket((inbound, outbound) -> outbound + .sendString(Flux.concat(Mono.just(STARTED).delayElement(Duration.ofMillis(50)), + inbound.receive().asString().concatMap(json -> { + Map command = GptLiveFunctionToolSample.object(BinaryData.fromString(json)); + commands.add(command); + if ("session.commentary.append".equals(command.get("type"))) { + return beforeGreeting.thenMany( + Flux.just("{\"type\":\"session.output_transcript.delta\",\"delta\":\"Welcome.\"}", + "{\"type\":\"session.output_audio.delta\",\"delta\":\"AQI=\"}")); + } + if ("response.create".equals(command.get("type"))) { + int responseIndex = responses.incrementAndGet(); + if (fail) { + return Flux.just("{\"type\":\"error\",\"error\":{\"message\":\"failed\"}}"); + } + if (tools && responseIndex == 1) { + return Flux.just("{\"type\":\"response.event\",\"delegation_id\":\"del-1\"," + + "\"event\":{\"type\":\"response.output_item.done\",\"item\":" + + "{\"type\":\"function_call\",\"call_id\":\"call-1\",\"name\":\"get_weather\"," + + "\"arguments\":\"{\\\"city\\\":\\\"Seattle\\\"}\"}}}", COMPLETED); + } + return Flux.just( + "{\"type\":\"response.event\",\"event\":" + + "{\"type\":\"response.output_text.delta\",\"delta\":\"Hello\"}}", + "{\"type\":\"session.output_transcript.delta\",\"delta\":\"Spoken reply.\"}", + "{\"type\":\"session.output_audio.delta\",\"delta\":\"AQI=\"}", COMPLETED); + } + if (continuousSilence && "session.input_audio.append".equals(command.get("type"))) { + return silentInputs.getAndIncrement() % 5 == 0 ? Flux.just(silentOutput) : Flux.empty(); + } + return "session.close".equals(command.get("type")) ? Flux.just(CLOSED) : Flux.empty(); + }))) + .then())) + .bindNow(); + } +} diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentConversationsAsyncTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentConversationsAsyncTests.java index 6d684658132ab..c16daa0a8a0f1 100644 --- a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentConversationsAsyncTests.java +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentConversationsAsyncTests.java @@ -7,7 +7,7 @@ import com.azure.ai.agents.AgentsAsyncClient; import com.azure.ai.agents.AgentsClientBuilder; import com.azure.ai.agents.BetaVoiceAgentsConversationsAsyncClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionAsyncClient; import com.azure.ai.agents.models.CreateAgentVersionInput; import com.azure.ai.agents.models.RealtimeResponseDoneEvent; import com.azure.ai.agents.models.RealtimeSessionCreatedEvent; @@ -105,7 +105,7 @@ public void readLivePersistedConversation() { try { agents.createAgentVersion(agentName, new CreateAgentVersionInput(definition)).block(TIMEOUT); created = true; - Mono.usingWhen(builder.beta().buildBetaVoiceAgentWebSocketAsyncClient().openWebSocketSession(agentName), + Mono.usingWhen(builder.beta().buildBetaVoiceAgentsWebSocketAsyncClient().openWebSocketSession(agentName), session -> session.receiveEvents().index().concatMap(indexed -> { if (indexed.getT1() == 0) { assertTrue(indexed.getT2() instanceof RealtimeSessionCreatedEvent, @@ -123,8 +123,8 @@ public void readLivePersistedConversation() { .switchIfEmpty(Mono.error(new AssertionError("Session ended without response.done."))) .timeout(Duration.ofSeconds(45)) .then(), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync, (session, error) -> session.closeAsync(), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync).block(Duration.ofSeconds(90)); + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync, (session, error) -> session.closeAsync(), + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync).block(Duration.ofSeconds(90)); Mono.delay(Duration.ofSeconds(30)).block(Duration.ofSeconds(35)); reading = true; assertPersistedConversation(conversations, agentName, conversationId.get()); diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentConversationsTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentConversationsTests.java index 590ed1b4cceaf..d80518c0c5a52 100644 --- a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentConversationsTests.java +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentConversationsTests.java @@ -6,7 +6,7 @@ import com.azure.ai.agents.AgentsClient; import com.azure.ai.agents.AgentsClientBuilder; import com.azure.ai.agents.BetaVoiceAgentsConversationsClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionClient; import com.azure.ai.agents.models.CreateAgentVersionInput; import com.azure.ai.agents.models.RealtimeServerEvent; import com.azure.ai.agents.models.RealtimeResponseDoneEvent; @@ -107,8 +107,8 @@ public void readLivePersistedConversation() throws InterruptedException { try { agents.createAgentVersion(agentName, new CreateAgentVersionInput(definition)); created = true; - try (BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession(agentName)) { + try (BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession(agentName)) { Iterator events = session.receiveEvents(TIMEOUT).iterator(); assertTrue(events.hasNext(), "Expected session.created."); RealtimeServerEvent first = events.next(); diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationSampleTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationSampleTests.java new file mode 100644 index 0000000000000..65068289eb980 --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentLiveTextConversationSampleTests.java @@ -0,0 +1,185 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.core.exception.HttpResponseException; +import com.azure.core.http.HttpClient; +import com.azure.core.http.HttpHeaderName; +import com.azure.core.http.HttpHeaders; +import com.azure.core.http.HttpMethod; +import com.azure.core.http.HttpRequest; +import com.azure.core.http.HttpResponse; +import com.azure.core.http.policy.ExponentialBackoffOptions; +import com.azure.core.http.policy.RetryOptions; +import com.azure.core.test.http.MockHttpResponse; +import com.azure.core.test.utils.MockTokenCredential; +import com.azure.core.util.BinaryData; +import com.azure.core.util.Configuration; +import com.azure.core.util.Context; +import java.util.ArrayDeque; +import java.util.Deque; +import java.util.function.Function; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import reactor.core.publisher.Mono; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class VoiceAgentLiveTextConversationSampleTests { + private static final String NAME = "sample-live-text-conversation-agent-java"; + private static final String GENERATE = "/agents:generate"; + private static final String AGENT = "/agents/" + NAME; + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void createdAgentIsDeletedOnNormalExitAndFailure(boolean failConversation) { + ScriptedTransport transport = new ScriptedTransport(); + transport.expect(HttpMethod.POST, GENERATE, 200); + transport.expect(HttpMethod.DELETE, AGENT, 200); + AgentsClientBuilder builder = builder(transport); + IllegalStateException failure = new IllegalStateException("Conversation failed"); + Runnable conversation = () -> { + try (VoiceAgentLiveTextConversationSample.SampleAgent agent = VoiceAgentLiveTextConversationSample + .createAgent(builder.buildAgentsClient(), builder.beta().buildBetaAgentsClient(), NAME)) { + assertNotNull(agent.details); + if (failConversation) { + throw failure; + } + } + }; + if (failConversation) { + assertSame(failure, assertThrows(IllegalStateException.class, conversation::run)); + } else { + conversation.run(); + } + transport.assertConsumed(); + } + + @Test + public void conflictDeletesExistingAgentRetriesOnceAndDeletesReplacement() { + ScriptedTransport transport = new ScriptedTransport(); + transport.expect(HttpMethod.POST, GENERATE, 409); + transport.expect(HttpMethod.DELETE, AGENT, 200); + transport.expect(HttpMethod.POST, GENERATE, 200); + transport.expect(HttpMethod.DELETE, AGENT, 200); + AgentsClientBuilder builder = builder(transport); + try (VoiceAgentLiveTextConversationSample.SampleAgent agent = VoiceAgentLiveTextConversationSample + .createAgent(builder.buildAgentsClient(), builder.beta().buildBetaAgentsClient(), NAME)) { + assertNotNull(agent.details); + } + transport.assertConsumed(); + } + + @ParameterizedTest + @ValueSource(ints = { 400, 403, 412 }) + public void nonConflictCreationFailureDoesNotDeleteAnAgent(int status) { + ScriptedTransport transport = new ScriptedTransport(); + transport.expect(HttpMethod.POST, GENERATE, status); + AgentsClientBuilder builder = builder(transport); + HttpResponseException error + = assertThrows(HttpResponseException.class, () -> VoiceAgentLiveTextConversationSample + .createAgent(builder.buildAgentsClient(), builder.beta().buildBetaAgentsClient(), NAME)); + assertEquals(status, error.getResponse().getStatusCode()); + transport.assertConsumed(); + } + + @Test + public void replacementDeletionFailureStopsWithoutRetryingCreation() { + ScriptedTransport transport = new ScriptedTransport(); + transport.expect(HttpMethod.POST, GENERATE, 409); + transport.expect(HttpMethod.DELETE, AGENT, 403); + AgentsClientBuilder builder = builder(transport); + HttpResponseException error + = assertThrows(HttpResponseException.class, () -> VoiceAgentLiveTextConversationSample + .createAgent(builder.buildAgentsClient(), builder.beta().buildBetaAgentsClient(), NAME)); + assertEquals(403, error.getResponse().getStatusCode()); + transport.assertConsumed(); + } + + @Test + public void repeatedConflictIsNotRetriedOrDeletedAgain() { + ScriptedTransport transport = new ScriptedTransport(); + transport.expect(HttpMethod.POST, GENERATE, 409); + transport.expect(HttpMethod.DELETE, AGENT, 200); + transport.expect(HttpMethod.POST, GENERATE, 409); + AgentsClientBuilder builder = builder(transport); + HttpResponseException error + = assertThrows(HttpResponseException.class, () -> VoiceAgentLiveTextConversationSample + .createAgent(builder.buildAgentsClient(), builder.beta().buildBetaAgentsClient(), NAME)); + assertEquals(409, error.getResponse().getStatusCode()); + transport.assertConsumed(); + } + + @Test + public void cleanupFailureIsPreservedAlongsideConversationFailure() { + ScriptedTransport transport = new ScriptedTransport(); + transport.expect(HttpMethod.POST, GENERATE, 200); + transport.expect(HttpMethod.DELETE, AGENT, 403); + AgentsClientBuilder builder = builder(transport); + IllegalStateException failure = new IllegalStateException("Conversation failed"); + IllegalStateException error = assertThrows(IllegalStateException.class, () -> { + try (VoiceAgentLiveTextConversationSample.SampleAgent agent = VoiceAgentLiveTextConversationSample + .createAgent(builder.buildAgentsClient(), builder.beta().buildBetaAgentsClient(), NAME)) { + assertNotNull(agent.details); + throw failure; + } + }); + assertSame(failure, error); + assertEquals(1, error.getSuppressed().length); + assertTrue(error.getSuppressed()[0] instanceof HttpResponseException); + transport.assertConsumed(); + } + + private static AgentsClientBuilder builder(HttpClient transport) { + return new AgentsClientBuilder().endpoint("https://localhost") + .credential(new MockTokenCredential()) + .configuration(Configuration.NONE) + .httpClient(transport) + .retryOptions(new RetryOptions(new ExponentialBackoffOptions().setMaxRetries(0))) + .allowPreview(true); + } + + private static final class ScriptedTransport implements HttpClient { + private final Deque> requests = new ArrayDeque<>(); + + void expect(HttpMethod method, String path, int status) { + requests.add(request -> { + assertEquals(method, request.getHttpMethod()); + assertEquals(path, request.getUrl().getPath()); + if (method == HttpMethod.POST) { + assertEquals(NAME, request.getBodyAsBinaryData().toObject(java.util.Map.class).get("name")); + } + String body = status >= 400 + ? "{\"error\":{\"code\":\"" + (status == 409 ? "conflict" : "failed") + + "\",\"message\":\"failed\"}}" + : "{}"; + return new MockHttpResponse(request, status, + new HttpHeaders().set(HttpHeaderName.CONTENT_TYPE, "application/json"), + BinaryData.fromString(body).toBytes()); + }); + } + + @Override + public Mono send(HttpRequest request) { + throw new AssertionError("The sync sample must not use asynchronous HTTP."); + } + + @Override + public HttpResponse sendSync(HttpRequest request, Context context) { + assertFalse(requests.isEmpty(), "Unexpected request: " + request.getUrl()); + return requests.removeFirst().apply(request); + } + + void assertConsumed() { + assertTrue(requests.isEmpty(), "Expected HTTP requests were not sent"); + } + } +} diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentLiveWebSocketTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentLiveWebSocketTests.java new file mode 100644 index 0000000000000..88526d703823c --- /dev/null +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentLiveWebSocketTests.java @@ -0,0 +1,644 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package com.azure.ai.agents.voice; + +import com.azure.ai.agents.AgentsClientBuilder; +import com.azure.ai.agents.BetaVoiceAgentsLiveWebSocketSessionAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsLiveWebSocketSessionClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionClient; +import com.azure.ai.agents.implementation.OpenAIJsonHelper; +import com.azure.ai.agents.models.VoiceAgentLiveWebSocketConnectionOptions; +import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions; +import com.azure.core.credential.AccessToken; +import com.azure.core.exception.ResourceNotFoundException; +import com.azure.core.util.BinaryData; +import com.azure.core.util.ClientOptions; +import com.azure.core.util.Configuration; +import com.azure.core.util.Header; +import com.openai.core.JsonValue; +import com.openai.models.live.ClientEvent; +import com.openai.models.live.ResponseCreateEvent; +import com.openai.models.live.ServerEvent; +import io.netty.handler.codec.http.HttpHeaderNames; +import java.net.URI; +import java.net.URLDecoder; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.OffsetDateTime; +import java.util.Arrays; +import java.util.Iterator; +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.api.parallel.ResourceLock; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; +import reactor.netty.DisposableServer; +import reactor.netty.http.server.WebsocketServerSpec; +import reactor.test.StepVerifier; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +@ResourceLock("voice-agent-websocket-tls") +@Timeout(20) +public class VoiceAgentLiveWebSocketTests { + private static final Duration TIMEOUT = Duration.ofSeconds(5); + private static final String STARTED + = "{\"type\":\"session.started\",\"conversation_id\":\"conversation-1\",\"session\":{\"id\":\"live-1\"}}"; + private static final String RESPONSE = "{\"type\":\"response.done\",\"future_field\":{\"native\":true}}"; + private static final String FUTURE = "{\"type\":\"future.live.event\",\"unknown\":[1,2,3]}"; + private static final String ERROR + = "{\"type\":\"error\",\"error\":{\"type\":\"server_error\",\"message\":\"native Live error\"}}"; + private static final String COMMAND = "{\"type\":\"session.update\",\"session\":{\"future_setting\":true}}"; + private DisposableServer server; + + @BeforeAll + static void installTlsCertificate() { + VoiceAgentWebSocketSessionTests.installTlsCertificate(); + } + + @AfterAll + static void restoreTlsConfiguration() { + VoiceAgentWebSocketSessionTests.restoreTlsConfiguration(); + } + + @AfterEach + void stopServer() { + if (server != null) { + server.disposeNow(); + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void handshakeUsesLiveRouteWithoutRealtimeSubprotocol(boolean async) throws Exception { + AtomicReference requestUri = new AtomicReference<>(); + AtomicReference headers = new AtomicReference<>(); + server = VoiceAgentWebSocketSessionTests.tlsServer().host("localhost").port(0).handle((request, response) -> { + requestUri.set(request.uri()); + headers.set(request.requestHeaders().copy()); + return response.sendWebsocket((inbound, outbound) -> inbound.receive().then()); + }).bindNow(); + AgentsClientBuilder builder + = builder().endpoint("https://localhost:" + server.port() + "/api/projects/project/") + .clientOptions(new ClientOptions().setApplicationId("live-app") + .setHeaders(Arrays.asList(new Header("X-Custom", "value"), new Header("Authorization", "override"), + new Header("Foundry-Features", "override"), new Header("Sec-WebSocket-Protocol", "realtime")))); + String inputs = "{\"name\":\"a & b/\u4f60\u597d\"}"; + VoiceAgentLiveWebSocketConnectionOptions options + = new VoiceAgentLiveWebSocketConnectionOptions().setStoreEnabled(true) + .setAgentVersionOverride("version 1") + .setStructuredInputs(inputs); + if (async) { + Mono connection = builder.beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent /name", options); + options.setStoreEnabled(false).setAgentVersionOverride("changed").setStructuredInputs("{}"); + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = connection.block(TIMEOUT); + assertNotNull(session); + assertTrue(session.isOpen()); + session.closeAsync().block(TIMEOUT); + assertFalse(session.isOpen()); + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session = builder.beta() + .buildBetaVoiceAgentsLiveWebSocketClient() + .openWebSocketSession("agent /name", options)) { + assertTrue(session.isOpen()); + } + } + + URI uri = URI.create(requestUri.get()); + assertEquals("/api/projects/project/agents/agent%20%2Fname/endpoint/protocols/voice/live/sessions", + uri.getRawPath()); + String query = URLDecoder.decode(uri.getRawQuery(), StandardCharsets.UTF_8.name()); + assertTrue(query.contains("api-version=v1"), query); + assertTrue(query.contains("store=true"), query); + assertTrue(query.contains("x-agent-version-override=version 1"), query); + assertTrue(query.contains("structured_inputs=" + inputs), query); + assertFalse(query.contains("transport="), query); + assertFalse(query.contains("agent_session_id="), query); + assertNull(headers.get().get(HttpHeaderNames.SEC_WEBSOCKET_PROTOCOL)); + assertNull(headers.get().get("x-ms-voice-structured-inputs")); + assertEquals("VoiceAgents=V1Preview", headers.get().get("Foundry-Features")); + assertEquals("value", headers.get().get("X-Custom")); + assertTrue(headers.get().get(HttpHeaderNames.USER_AGENT).startsWith("live-app azsdk-java-")); + assertFalse("override".equals(headers.get().get(HttpHeaderNames.AUTHORIZATION))); + } + + @Test + public void syncNativeEventsAreLosslessAndSessionIsBidirectional() { + List sent = new CopyOnWriteArrayList<>(); + startEchoServer(sent); + BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent"); + try { + Iterator events = session.receiveRawEvents(TIMEOUT).iterator(); + assertEquals(STARTED, events.next().toString()); + assertEquals(RESPONSE, events.next().toString()); + assertEquals(FUTURE, events.next().toString()); + assertEquals(ERROR, events.next().toString()); + assertThrows(IllegalStateException.class, session::receiveRawEvents); + assertThrows(IllegalStateException.class, session::receiveEvents); + assertThrows(IllegalArgumentException.class, () -> session.sendEvent(BinaryData.fromString("[]"))); + session.sendEvent(BinaryData.fromString(COMMAND)); + assertEquals(COMMAND, events.next().toString()); + assertEquals(Arrays.asList(COMMAND), sent); + session.close(1000, "done"); + assertFalse(session.isOpen()); + assertThrows(IllegalStateException.class, () -> session.sendEvent(BinaryData.fromString(COMMAND))); + } finally { + session.close(); + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void rawReceptionDoesNotRequireTypedDecoding(boolean async) { + String message = "not-json"; + String unusualPayload = "{\"type\":\"session.started\",\"session\":7}"; + AtomicInteger malformed = new AtomicInteger(); + VoiceAgentLiveWebSocketConnectionOptions options = new VoiceAgentLiveWebSocketConnectionOptions() + .setMalformedEventHandler(error -> malformed.incrementAndGet()); + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.sendWebsocket((inbound, + outbound) -> outbound.sendString(Flux.just(message, unusualPayload)).then(inbound.receive().then()))) + .bindNow(); + if (async) { + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = builder().beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent", options) + .block(TIMEOUT); + assertNotNull(session); + try { + StepVerifier.create(session.receiveRawEvents().take(2)) + .assertNext(event -> assertEquals(message, event.toString())) + .assertNext(event -> assertEquals(unusualPayload, event.toString())) + .verifyComplete(); + } finally { + session.closeAsync().block(TIMEOUT); + } + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent", options)) { + Iterator events = session.receiveRawEvents(TIMEOUT).iterator(); + assertEquals(message, events.next().toString()); + assertEquals(unusualPayload, events.next().toString()); + } + } + assertEquals(0, malformed.get()); + } + + @Test + public void asyncNativeEventsAreLosslessAndSessionIsBidirectional() { + List sent = new CopyOnWriteArrayList<>(); + startEchoServer(sent); + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = builder().beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent") + .block(TIMEOUT); + assertNotNull(session); + try { + StepVerifier.create(session.sendEvent(BinaryData.fromString("[]"))) + .expectError(IllegalArgumentException.class) + .verify(TIMEOUT); + StepVerifier + .create(session.sendEvent(BinaryData.fromString(COMMAND)) + .thenMany(session.receiveRawEvents()) + .map(BinaryData::toString)) + .expectNext(STARTED, RESPONSE, FUTURE, ERROR) + .expectNext(COMMAND) + .thenCancel() + .verify(TIMEOUT); + assertEquals(Arrays.asList(COMMAND), sent); + StepVerifier.create(session.receiveRawEvents()).expectError(IllegalStateException.class).verify(TIMEOUT); + StepVerifier.create(session.receiveEvents()).expectError(IllegalStateException.class).verify(TIMEOUT); + session.closeAsync(1000, "done").block(TIMEOUT); + assertFalse(session.isOpen()); + StepVerifier.create(session.sendEvent(BinaryData.fromString(COMMAND))) + .expectError(IllegalStateException.class) + .verify(TIMEOUT); + } finally { + session.closeAsync().block(TIMEOUT); + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void sameBuilderKeepsLiveAndRealtimeHandshakesIndependent(boolean async) { + List requestUris = new CopyOnWriteArrayList<>(); + List headers = new CopyOnWriteArrayList<>(); + server = VoiceAgentWebSocketSessionTests.tlsServer().host("localhost").port(0).handle((request, response) -> { + requestUris.add(request.uri()); + headers.add(request.requestHeaders().copy()); + return response.sendWebsocket((inbound, outbound) -> inbound.receive().then(), + WebsocketServerSpec.builder().protocols("realtime").build()); + }).bindNow(); + AgentsClientBuilder builder = builder(); + String inputs = "{\"name\":\"value\"}"; + VoiceAgentLiveWebSocketConnectionOptions liveOptions + = new VoiceAgentLiveWebSocketConnectionOptions().setStructuredInputs(inputs); + VoiceAgentWebSocketConnectionOptions realtimeOptions + = new VoiceAgentWebSocketConnectionOptions().setStructuredInputs(inputs); + if (async) { + try (BetaVoiceAgentsLiveWebSocketSessionAsyncClient live = builder.beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent", liveOptions) + .block(TIMEOUT)) { + assertNotNull(live); + assertTrue(live.isOpen()); + } + try (BetaVoiceAgentsWebSocketSessionAsyncClient realtime = builder.beta() + .buildBetaVoiceAgentsWebSocketAsyncClient() + .openWebSocketSession("agent", realtimeOptions) + .block(TIMEOUT)) { + assertNotNull(realtime); + assertTrue(realtime.isOpen()); + } + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient live + = builder.beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent", liveOptions)) { + assertTrue(live.isOpen()); + } + try (BetaVoiceAgentsWebSocketSessionClient realtime + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent", realtimeOptions)) { + assertTrue(realtime.isOpen()); + } + } + assertEquals(2, requestUris.size()); + assertEquals("/agents/agent/endpoint/protocols/voice/live/sessions", URI.create(requestUris.get(0)).getPath()); + assertTrue(requestUris.get(0).contains("structured_inputs=")); + assertFalse(requestUris.get(0).contains("transport=")); + assertNull(headers.get(0).get(HttpHeaderNames.SEC_WEBSOCKET_PROTOCOL)); + assertNull(headers.get(0).get("x-ms-voice-structured-inputs")); + assertEquals("/agents/agent/endpoint/protocols/voice", URI.create(requestUris.get(1)).getPath()); + assertTrue(requestUris.get(1).contains("transport=websocket")); + assertFalse(requestUris.get(1).contains("structured_inputs=")); + assertEquals("realtime", headers.get(1).get(HttpHeaderNames.SEC_WEBSOCKET_PROTOCOL)); + assertEquals(inputs, headers.get(1).get("x-ms-voice-structured-inputs")); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void typedEventsRoundTripWithoutLosingUnknownFields(boolean async) { + List sent = new CopyOnWriteArrayList<>(); + String audio = "{\"type\":\"session.output_audio.delta\",\"delta\":\"AAA=\",\"future_field\":true}"; + String backend = "{\"type\":\"response.event\",\"event_id\":\"backend-1\"," + + "\"event\":{\"type\":\"response.completed\",\"response\":{\"id\":\"response-1\",\"future_field\":true}}}"; + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, + response) -> response.sendWebsocket((inbound, + outbound) -> outbound.sendString(Flux.concat(Flux.just(STARTED, audio, backend, FUTURE, ERROR), + inbound.receive().asString().doOnNext(sent::add))).then())) + .bindNow(); + ClientEvent command = ClientEvent.ofResponseCreate(ResponseCreateEvent.builder() + .eventId("command-1") + .putAdditionalProperty("future_setting", JsonValue.from(true)) + .build()); + String commandJson = OpenAIJsonHelper.toBinaryData(command).toString(); + if (async) { + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = builder().beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent") + .block(TIMEOUT); + assertNotNull(session); + try { + assertThrows(NullPointerException.class, () -> session.sendEvent((ClientEvent) null)); + StepVerifier.create(session.sendEvent(command).thenMany(session.receiveEvents())) + .assertNext(event -> assertStartedEvent(event)) + .assertNext(event -> assertAudioEvent(event, audio)) + .assertNext(event -> assertBackendEvent(event, backend)) + .assertNext(event -> assertEventJson(event, FUTURE)) + .assertNext(event -> { + assertTrue(event.isErrorEvent()); + assertEventJson(event, ERROR); + }) + .assertNext(event -> assertEventJson(event, commandJson)) + .thenCancel() + .verify(TIMEOUT); + StepVerifier.create(session.receiveRawEvents()) + .expectError(IllegalStateException.class) + .verify(TIMEOUT); + } finally { + session.closeAsync().block(TIMEOUT); + } + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + assertThrows(NullPointerException.class, () -> session.sendEvent((ClientEvent) null)); + assertThrows(IllegalArgumentException.class, () -> session.receiveEvents(Duration.ZERO)); + Iterator events = session.receiveEvents(TIMEOUT).iterator(); + assertStartedEvent(events.next()); + assertAudioEvent(events.next(), audio); + assertBackendEvent(events.next(), backend); + assertEventJson(events.next(), FUTURE); + ServerEvent error = events.next(); + assertTrue(error.isErrorEvent()); + assertEventJson(error, ERROR); + session.sendEvent(command); + assertEventJson(events.next(), commandJson); + assertThrows(IllegalStateException.class, session::receiveRawEvents); + } + } + assertEquals(Arrays.asList(commandJson), sent); + } + + private static void assertStartedEvent(ServerEvent event) { + assertTrue(event.isSessionStarted()); + assertEquals(JsonValue.from("conversation-1"), + event.asSessionStarted()._additionalProperties().get("conversation_id")); + assertEventJson(event, STARTED); + } + + private static void assertAudioEvent(ServerEvent event, String json) { + assertTrue(event.isSessionOutputAudioDelta()); + assertEquals("AAA=", event.asSessionOutputAudioDelta().delta()); + assertEventJson(event, json); + } + + private static void assertBackendEvent(ServerEvent event, String json) { + assertTrue(event.isResponseEvent()); + assertEquals("backend-1", event.asResponseEvent().eventId()); + assertEventJson(event, json); + } + + private static void assertEventJson(ServerEvent event, String json) { + assertEquals(BinaryData.fromString(json).toObject(Object.class), + OpenAIJsonHelper.toBinaryData(event).toObject(Object.class)); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void handshakeFailuresSurfaceAsAzureExceptions(boolean async) { + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.status(404).send()) + .bindNow(); + if (async) { + StepVerifier + .create(builder().beta().buildBetaVoiceAgentsLiveWebSocketAsyncClient().openWebSocketSession("missing")) + .expectError(ResourceNotFoundException.class) + .verify(TIMEOUT); + } else { + assertThrows(ResourceNotFoundException.class, + () -> builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("missing")); + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void malformedJsonTerminatesReception(boolean async) { + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.sendWebsocket( + (inbound, outbound) -> outbound.sendString(Mono.just("[]")).then(inbound.receive().then()))) + .bindNow(); + if (async) { + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = builder().beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent") + .block(TIMEOUT); + assertNotNull(session); + try { + StepVerifier.create(session.receiveEvents()) + .expectError(IllegalArgumentException.class) + .verify(TIMEOUT); + } finally { + session.closeAsync().block(TIMEOUT); + } + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + assertThrows(IllegalArgumentException.class, () -> session.receiveEvents(TIMEOUT).iterator().hasNext()); + } + } + } + + @Test + public void connectionOptionsValidateBoundsAndCopyAllSettings() { + VoiceAgentLiveWebSocketConnectionOptions options = new VoiceAgentLiveWebSocketConnectionOptions(); + assertEquals(256, options.getReceiveBufferCapacity()); + assertEquals(32 * 1024 * 1024, options.getMaxMessageSize()); + assertThrows(IllegalArgumentException.class, () -> options.setReceiveBufferCapacity(0)); + assertThrows(IllegalArgumentException.class, () -> options.setReceiveBufferCapacity(65537)); + assertThrows(IllegalArgumentException.class, () -> options.setMaxMessageSize(0)); + assertThrows(IllegalArgumentException.class, () -> options.setHandshakeTimeout(Duration.ZERO)); + assertThrows(IllegalArgumentException.class, () -> options.setCloseTimeout(Duration.ofSeconds(-1))); + options.setReceiveBufferCapacity(1) + .setMaxMessageSize(1024) + .setStoreEnabled(true) + .setHandshakeTimeout(TIMEOUT) + .setCloseTimeout(TIMEOUT) + .setAgentVersionOverride("1") + .setStructuredInputs("{\"name\":\"value\"}"); + VoiceAgentLiveWebSocketConnectionOptions copy = new VoiceAgentLiveWebSocketConnectionOptions(options); + options.setReceiveBufferCapacity(65536).setStoreEnabled(false).setStructuredInputs("{}"); + assertEquals(1, copy.getReceiveBufferCapacity()); + assertEquals(1024, copy.getMaxMessageSize()); + assertEquals(true, copy.isStoreEnabled()); + assertEquals(TIMEOUT, copy.getHandshakeTimeout()); + assertEquals(TIMEOUT, copy.getCloseTimeout()); + assertEquals("1", copy.getAgentVersionOverride()); + assertEquals("{\"name\":\"value\"}", copy.getStructuredInputs()); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void throwingMalformedEventCallbackTerminatesTypedReception(boolean async) { + IllegalStateException failure = new IllegalStateException("Malformed event callback failed."); + VoiceAgentLiveWebSocketConnectionOptions options + = new VoiceAgentLiveWebSocketConnectionOptions().setMalformedEventHandler(error -> { + throw failure; + }); + server + = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.sendWebsocket((inbound, + outbound) -> outbound.sendString(Flux.just("not-json", STARTED)).then(inbound.receive().then()))) + .bindNow(); + if (async) { + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = builder().beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent", options) + .block(TIMEOUT); + assertNotNull(session); + try { + StepVerifier.create(session.receiveEvents()) + .expectErrorMatches(error -> error == failure) + .verify(TIMEOUT); + assertFalse(session.isOpen()); + } finally { + session.closeAsync().block(TIMEOUT); + } + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent", options)) { + Iterator events = session.receiveEvents(TIMEOUT).iterator(); + assertEquals(failure, assertThrows(IllegalStateException.class, events::hasNext)); + assertFalse(events.hasNext()); + assertFalse(session.isOpen()); + } + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void peerClosurePreservesEventsAndCloseMetadata(boolean async) { + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, + response) -> response.sendWebsocket((inbound, outbound) -> outbound.sendString(Mono.just(STARTED)) + .then(outbound.sendClose(1000, "finished")))) + .bindNow(); + if (async) { + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = builder().beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent") + .block(TIMEOUT); + assertNotNull(session); + try { + StepVerifier.create(session.receiveRawEvents().map(BinaryData::toString)) + .expectNext(STARTED) + .expectComplete() + .verify(TIMEOUT); + assertEquals(1000, session.getCloseCode()); + assertEquals("finished", session.getCloseReason()); + assertFalse(session.isOpen()); + } finally { + session.closeAsync().block(TIMEOUT); + } + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent")) { + Iterator events = session.receiveRawEvents(TIMEOUT).iterator(); + assertEquals(STARTED, events.next().toString()); + assertFalse(events.hasNext()); + assertEquals(1000, session.getCloseCode()); + assertEquals("finished", session.getCloseReason()); + assertFalse(session.isOpen()); + } + } + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void malformedEventCallbackCanSkipInvalidJson(boolean async) { + AtomicInteger malformed = new AtomicInteger(); + VoiceAgentLiveWebSocketConnectionOptions options = new VoiceAgentLiveWebSocketConnectionOptions() + .setMalformedEventHandler(error -> malformed.incrementAndGet()); + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.sendWebsocket((inbound, + outbound) -> outbound.sendString(Flux.just("[]", "not-json", STARTED)).then(inbound.receive().then()))) + .bindNow(); + if (async) { + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = builder().beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent", options) + .block(TIMEOUT); + assertNotNull(session); + try { + StepVerifier.create(session.receiveEvents()) + .assertNext(VoiceAgentLiveWebSocketTests::assertStartedEvent) + .thenCancel() + .verify(TIMEOUT); + } finally { + session.closeAsync().block(TIMEOUT); + } + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent", options)) { + assertStartedEvent(session.receiveEvents(TIMEOUT).iterator().next()); + } + } + assertEquals(2, malformed.get()); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void oversizedEventsFailEvenWhenMalformedCallbackIsConfigured(boolean async) { + AtomicInteger malformed = new AtomicInteger(); + VoiceAgentLiveWebSocketConnectionOptions options + = new VoiceAgentLiveWebSocketConnectionOptions().setMaxMessageSize(32) + .setMalformedEventHandler(error -> malformed.incrementAndGet()); + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, response) -> response.sendWebsocket( + (inbound, outbound) -> outbound.sendString(Mono.just(STARTED)).then(inbound.receive().then()))) + .bindNow(); + if (async) { + BetaVoiceAgentsLiveWebSocketSessionAsyncClient session = builder().beta() + .buildBetaVoiceAgentsLiveWebSocketAsyncClient() + .openWebSocketSession("agent", options) + .block(TIMEOUT); + assertNotNull(session); + try { + StepVerifier.create(session.receiveRawEvents()).expectError().verify(TIMEOUT); + } finally { + session.closeAsync().block(TIMEOUT); + } + } else { + try (BetaVoiceAgentsLiveWebSocketSessionClient session + = builder().beta().buildBetaVoiceAgentsLiveWebSocketClient().openWebSocketSession("agent", options)) { + assertThrows(IllegalArgumentException.class, + () -> session.receiveRawEvents(TIMEOUT).iterator().hasNext()); + } + } + assertEquals(0, malformed.get()); + } + + @ParameterizedTest + @ValueSource(booleans = { false, true }) + public void unsupportedHttpSettingsAreRejected(boolean async) { + AgentsClientBuilder builder = new AgentsClientBuilder().endpoint("https://example.com") + .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))) + .httpClient(com.azure.core.http.HttpClient.createDefault()); + if (async) { + assertThrows(IllegalStateException.class, + () -> builder.beta().buildBetaVoiceAgentsLiveWebSocketAsyncClient()); + } else { + assertThrows(IllegalStateException.class, () -> builder.beta().buildBetaVoiceAgentsLiveWebSocketClient()); + } + } + + private AgentsClientBuilder builder() { + return new AgentsClientBuilder().endpoint("https://localhost:" + server.port()) + .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))) + .configuration(Configuration.NONE); + } + + private void startEchoServer(List sent) { + server = VoiceAgentWebSocketSessionTests.tlsServer() + .host("localhost") + .port(0) + .handle((request, + response) -> response.sendWebsocket((inbound, + outbound) -> outbound.sendString(Flux.concat(Flux.just(STARTED, RESPONSE, FUTURE, ERROR), + inbound.receive().asString().doOnNext(sent::add))).then())) + .bindNow(); + } +} diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentRealtimeLiveTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentRealtimeLiveTests.java index 9b05562145d25..613b92cc46172 100644 --- a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentRealtimeLiveTests.java +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentRealtimeLiveTests.java @@ -6,8 +6,8 @@ import com.azure.ai.agents.AgentsAsyncClient; import com.azure.ai.agents.AgentsClient; import com.azure.ai.agents.AgentsClientBuilder; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionAsyncClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionClient; import com.azure.ai.agents.models.CreateAgentVersionInput; import com.azure.ai.agents.models.RealtimeClientEvent; import com.azure.ai.agents.models.RealtimeConversationItemCreateEvent; @@ -87,8 +87,8 @@ public void realtimeLive(Scenario scenario) { try { agents.createAgentVersion(agentName, new CreateAgentVersionInput(definition(scenario))); created = true; - try (BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession(agentName)) { + try (BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession(agentName)) { Turn turn = new Turn(scenario); Iterator events = session.receiveEvents(EVENT_TIMEOUT).iterator(); turn.accept(events.next()).forEach(session::sendEvent); @@ -118,7 +118,7 @@ public void realtimeLiveAsync(Scenario scenario) { .block(EVENT_TIMEOUT); created = true; Turn turn = new Turn(scenario); - Mono.usingWhen(builder.beta().buildBetaVoiceAgentWebSocketAsyncClient().openWebSocketSession(agentName), + Mono.usingWhen(builder.beta().buildBetaVoiceAgentsWebSocketAsyncClient().openWebSocketSession(agentName), session -> session.receiveEvents() .timeout(EVENT_TIMEOUT) .concatMap(event -> Flux.fromIterable(turn.accept(event)) @@ -130,8 +130,8 @@ public void realtimeLiveAsync(Scenario scenario) { .timeout(RESPONSE_TIMEOUT) .doOnNext(ignored -> turn.assertComplete()) .then(), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync, (session, error) -> session.closeAsync(), - BetaVoiceAgentWebSocketSessionAsyncClient::closeAsync).block(Duration.ofSeconds(90)); + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync, (session, error) -> session.closeAsync(), + BetaVoiceAgentsWebSocketSessionAsyncClient::closeAsync).block(Duration.ofSeconds(90)); } finally { if (created) { agents.deleteAgent(agentName).block(EVENT_TIMEOUT); diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentWebSocketSessionAsyncTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentWebSocketSessionAsyncTests.java index 225f80c03e38b..0e7fa4bfa9d31 100644 --- a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentWebSocketSessionAsyncTests.java +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentWebSocketSessionAsyncTests.java @@ -4,8 +4,8 @@ package com.azure.ai.agents.voice; import com.azure.ai.agents.AgentsClientBuilder; -import com.azure.ai.agents.BetaVoiceAgentWebSocketAsyncClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketAsyncClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionAsyncClient; import com.azure.ai.agents.implementation.realtime.VoiceAgentWebSocketHttpResponse; import com.azure.ai.agents.models.RawRealtimeServerEvent; import com.azure.ai.agents.models.RealtimeResponseCreateEvent; @@ -51,6 +51,7 @@ import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.parallel.Isolated; import org.junit.jupiter.api.parallel.ResourceLock; import reactor.core.publisher.Flux; import reactor.core.publisher.Mono; @@ -66,6 +67,7 @@ import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively; import static org.junit.jupiter.api.Assertions.assertTrue; +@Isolated("Virtual-time tests replace the global Reactor schedulers") @ResourceLock("voice-agent-websocket-tls") public class VoiceAgentWebSocketSessionAsyncTests { private DisposableServer server; @@ -91,8 +93,8 @@ public void handshakeUsesSdkManagedProtocolValues() { new Header("Foundry-Features", "override-feature"), new Header("Sec-WebSocket-Protocol", "override-protocol")))); - BetaVoiceAgentWebSocketSessionAsyncClient session = builder.beta() - .buildBetaVoiceAgentWebSocketAsyncClient() + BetaVoiceAgentsWebSocketSessionAsyncClient session = builder.beta() + .buildBetaVoiceAgentsWebSocketAsyncClient() .openWebSocketSession("agent name") .block(Duration.ofSeconds(5)); session.closeAsync().block(Duration.ofSeconds(5)); @@ -131,8 +133,8 @@ public void handshakeUsesHttpLogApplicationIdFallback() { .httpLogOptions(new HttpLogOptions().setApplicationId("log-app")) .clientOptions(new ClientOptions()); - BetaVoiceAgentWebSocketSessionAsyncClient session = builder.beta() - .buildBetaVoiceAgentWebSocketAsyncClient() + BetaVoiceAgentsWebSocketSessionAsyncClient session = builder.beta() + .buildBetaVoiceAgentsWebSocketAsyncClient() .openWebSocketSession("agent") .block(Duration.ofSeconds(5)); session.closeAsync().block(Duration.ofSeconds(5)); @@ -166,8 +168,8 @@ public void typedStringAndMappingSendsRejectInvalidJson() { AgentsClientBuilder builder = builder(server.port()); String raw = "{\"type\": \"response.create\"}"; BinaryData mapping = BinaryData.fromObject(Collections.singletonMap("type", "response.cancel")); - BetaVoiceAgentWebSocketSessionAsyncClient session = builder.beta() - .buildBetaVoiceAgentWebSocketAsyncClient() + BetaVoiceAgentsWebSocketSessionAsyncClient session = builder.beta() + .buildBetaVoiceAgentsWebSocketAsyncClient() .openWebSocketSession("agent", tlsOptions()) .block(Duration.ofSeconds(5)); try { @@ -194,7 +196,7 @@ public void pingPongFramesAreNotApplicationEvents() { new TextWebSocketFrame("{\"type\":\"session.created\",\"session\":{}}"), new TextWebSocketFrame("{\"type\":\"future.event\",\"foo\":\"bar\"}"))); server = frameWebSocketServer(frames, false); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", tlsOptions()).block(Duration.ofSeconds(5)); List events; try { @@ -217,7 +219,7 @@ public void malformedEventsCanBeReportedAndSkipped() { new TextWebSocketFrame(warningJson()))), true); VoiceAgentWebSocketConnectionOptions options = tlsOptions().setMalformedEventHandler(error -> failures.incrementAndGet()); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", options).block(Duration.ofSeconds(5)); StepVerifier.create(session.receiveEvents()) .assertNext(this::assertWarningEvent) @@ -234,7 +236,7 @@ public void boundedQueuesHonorOverflowPolicies() { .map(index -> new TextWebSocketFrame("{\"type\":\"future.event\",\"index\":" + index + "}")), true); VoiceAgentWebSocketConnectionOptions options = tlsOptions().setReceiveBufferCapacity(2).setOverflowStrategy(strategy); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", options).block(Duration.ofSeconds(5)); boolean overflowError = strategy == VoiceAgentWebSocketOverflowStrategy.ERROR; List received = new ArrayList<>(); @@ -271,7 +273,7 @@ public void messageSizeLimitCannotBeBypassedByRecoveryHandler() { VoiceAgentWebSocketConnectionOptions options = tlsOptions().setMaxMessageSize(16).setMalformedEventHandler(error -> recovered.set(true)); StepVerifier.create(createAsyncClient(server.port()).openWebSocketSession("agent", options) - .flatMapMany(BetaVoiceAgentWebSocketSessionAsyncClient::receiveEvents)).expectError().verify(); + .flatMapMany(BetaVoiceAgentsWebSocketSessionAsyncClient::receiveEvents)).expectError().verify(); assertFalse(recovered.get()); } @@ -282,7 +284,7 @@ public void rawEventRoundTripsAndOptionsValidateBounds() throws Exception { RawRealtimeServerEvent copy = BinaryData.fromObject(event).toObject(RawRealtimeServerEvent.class); assertEquals(payload.toObject(Map.class), copy.getRawEvent().toObject(Map.class)); server = oneShotWebSocketServer("{\"type\":42,\"value\":1}"); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", tlsOptions()).block(Duration.ofSeconds(5)); try { StepVerifier.create(session.receiveEvents()) @@ -312,7 +314,7 @@ public void insecureEndpointsAreRejectedBeforeAuthentication() { "https://example.com/#fragment" }) { AgentsClientBuilder clientBuilder = new AgentsClientBuilder().endpoint(endpoint).credential(credential); StepVerifier - .create(clientBuilder.beta().buildBetaVoiceAgentWebSocketAsyncClient().openWebSocketSession("agent")) + .create(clientBuilder.beta().buildBetaVoiceAgentsWebSocketAsyncClient().openWebSocketSession("agent")) .expectError(IllegalArgumentException.class) .verify(); } @@ -330,7 +332,7 @@ public void rawEventsUseCustomizedTlsTransport() { WebsocketServerSpec.builder().protocols("realtime").build())) .bindNow(); BinaryData payload = BinaryData.fromString("{\"type\":\"future.event\",\"value\":42}"); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", tlsOptions()).block(Duration.ofSeconds(5)); StepVerifier.create(session.sendEvent(BinaryData.fromString("[]"))) .expectError(IllegalArgumentException.class) @@ -348,7 +350,7 @@ public void rawEventsUseCustomizedTlsTransport() { public void customCloseFrame() { server = startServer(new CopyOnWriteArrayList<>(), new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), false); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", tlsOptions()).block(Duration.ofSeconds(5)); assertNotNull(session); StepVerifier.create(session.closeAsync(1006, "invalid")).expectError(IllegalArgumentException.class).verify(); @@ -374,18 +376,18 @@ public void asyncSessionNegotiatesHandshakeAndExchangesTypedEvents() { }; VoiceAgentWebSocketConnectionOptions options = tlsOptions().setStoreEnabled(true).setAgentVersionOverride("version 2"); - BetaVoiceAgentWebSocketAsyncClient client + BetaVoiceAgentsWebSocketAsyncClient client = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(credential) .configuration(Configuration.NONE) .clientOptions(new ClientOptions().setApplicationId("test-app") .setHeaders(Collections.singletonList(new Header("X-Test-Header", "test-value")))) .beta() - .buildBetaVoiceAgentWebSocketAsyncClient(); - Mono sessionMono + .buildBetaVoiceAgentsWebSocketAsyncClient(); + Mono sessionMono = client.openWebSocketSession("agent name", options); options.setStoreEnabled(false).setAgentVersionOverride("mutated"); - BetaVoiceAgentWebSocketSessionAsyncClient session = sessionMono.block(); + BetaVoiceAgentsWebSocketSessionAsyncClient session = sessionMono.block(); assertTrue(session.isOpen()); StepVerifier.create(session.receiveEvents().take(4)) .then(() -> session.sendText("hello").block()) @@ -426,12 +428,12 @@ public void tokenFailureOccursBeforeNetworkAccess() { return response.send(); }).bindNow(); TokenCredential credential = request -> Mono.error(new IllegalStateException("token unavailable")); - BetaVoiceAgentWebSocketAsyncClient client + BetaVoiceAgentsWebSocketAsyncClient client = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketAsyncClient(); + .buildBetaVoiceAgentsWebSocketAsyncClient(); StepVerifier.create(client.openWebSocketSession("agent", tlsOptions())) .expectErrorMatches( error -> error instanceof IllegalStateException && error.getMessage().contains("token unavailable")) @@ -445,12 +447,12 @@ public void tokenAcquisitionDoesNotUseHandshakeTimeout() { new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), false); TokenCredential credential = request -> Mono.delay(Duration.ofMillis(1500)) .map(ignored -> new AccessToken("test-token", OffsetDateTime.now().plusHours(1))); - BetaVoiceAgentWebSocketAsyncClient client + BetaVoiceAgentsWebSocketAsyncClient client = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketAsyncClient(); + .buildBetaVoiceAgentsWebSocketAsyncClient(); VoiceAgentWebSocketConnectionOptions options = tlsOptions().setHandshakeTimeout(Duration.ofSeconds(1)); StepVerifier.withVirtualTime(() -> client.openWebSocketSession("agent", options).flatMap(session -> { assertTrue(session.isOpen()); @@ -488,11 +490,11 @@ public void cancellingAsyncConnectCancelsTokenRequest() { AtomicBoolean tokenRequestCancelled = new AtomicBoolean(); TokenCredential credential = request -> Mono.never().doOnCancel(() -> tokenRequestCancelled.set(true)); - BetaVoiceAgentWebSocketAsyncClient client = new AgentsClientBuilder().endpoint("https://example.com") + BetaVoiceAgentsWebSocketAsyncClient client = new AgentsClientBuilder().endpoint("https://example.com") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketAsyncClient(); + .buildBetaVoiceAgentsWebSocketAsyncClient(); StepVerifier.create(client.openWebSocketSession("agent", tlsOptions())).thenCancel().verify(); assertTrue(tokenRequestCancelled.get()); } @@ -500,7 +502,7 @@ public void cancellingAsyncConnectCancelsTokenRequest() { @Test public void unknownEventFallsBackToRealtimeServerEvent() { server = oneShotWebSocketServer("{\"type\":\"future.event\",\"value\":42}"); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", tlsOptions()).block(); StepVerifier.create(session.receiveEvents()).assertNext(event -> { assertEquals("future.event", event.getType().toString()); @@ -516,7 +518,7 @@ public void fragmentedTextFrameIsAggregated() { int split = message.length() / 2; server = frameWebSocketServer(Flux.just(new TextWebSocketFrame(false, 0, message.substring(0, split)), new ContinuationWebSocketFrame(true, 0, message.substring(split))), true); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", tlsOptions()).block(); StepVerifier.create(session.receiveEvents()).assertNext(this::assertWarningEvent).verifyComplete(); session.close(); @@ -526,7 +528,7 @@ public void fragmentedTextFrameIsAggregated() { public void binaryJsonFrameIsParsed() { server = frameWebSocketServer( Mono.just(new BinaryWebSocketFrame(Unpooled.copiedBuffer(warningJson(), StandardCharsets.UTF_8))), true); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", tlsOptions()).block(); StepVerifier.create(session.receiveEvents()).assertNext(this::assertWarningEvent).verifyComplete(); session.close(); @@ -535,7 +537,7 @@ public void binaryJsonFrameIsParsed() { @Test public void malformedJsonTerminatesReceiveStream() { server = oneShotWebSocketServer("{not-json"); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", tlsOptions()).block(); StepVerifier.create(session.receiveEvents()).expectError().verify(); assertFalse(session.isOpen()); @@ -545,7 +547,7 @@ public void malformedJsonTerminatesReceiveStream() { public void closeIsIdempotentAndSendAfterCloseFails() { server = startServer(new CopyOnWriteArrayList<>(), new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), false); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("agent", tlsOptions()).block(); StepVerifier.create(session.closeAsync().then(session.closeAsync())).verifyComplete(); StepVerifier.create(session.sendText("after close")) @@ -557,7 +559,7 @@ public void closeIsIdempotentAndSendAfterCloseFails() { @Test public void secureSessionUsesWssAndReceivesTypedEvent() throws Exception { server = oneShotWebSocketServer(warningJson()); - BetaVoiceAgentWebSocketSessionAsyncClient session + BetaVoiceAgentsWebSocketSessionAsyncClient session = createAsyncClient(server.port()).openWebSocketSession("secure-agent", tlsOptions()) .block(Duration.ofSeconds(5)); assertEquals("wss", URI.create(session.getEndpoint()).getScheme()); @@ -570,8 +572,8 @@ private AgentsClientBuilder builder(int port) { .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))); } - private BetaVoiceAgentWebSocketAsyncClient createAsyncClient(int port) { - return builder(port).configuration(Configuration.NONE).beta().buildBetaVoiceAgentWebSocketAsyncClient(); + private BetaVoiceAgentsWebSocketAsyncClient createAsyncClient(int port) { + return builder(port).configuration(Configuration.NONE).beta().buildBetaVoiceAgentsWebSocketAsyncClient(); } private static VoiceAgentWebSocketConnectionOptions tlsOptions() { diff --git a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentWebSocketSessionTests.java b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentWebSocketSessionTests.java index b4acb9796b16a..e7d60fc6d7744 100644 --- a/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentWebSocketSessionTests.java +++ b/sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/voice/VoiceAgentWebSocketSessionTests.java @@ -4,8 +4,8 @@ package com.azure.ai.agents.voice; import com.azure.ai.agents.AgentsClientBuilder; -import com.azure.ai.agents.BetaVoiceAgentWebSocketClient; -import com.azure.ai.agents.BetaVoiceAgentWebSocketSessionClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketClient; +import com.azure.ai.agents.BetaVoiceAgentsWebSocketSessionClient; import com.azure.ai.agents.models.RawRealtimeServerEvent; import com.azure.ai.agents.models.RealtimeResponseCreateEvent; import com.azure.ai.agents.models.RealtimeSessionCreatedEvent; @@ -111,8 +111,8 @@ public void handshakeUsesSdkManagedProtocolValues() { new Header("Foundry-Features", "override-feature"), new Header("Sec-WebSocket-Protocol", "override-protocol")))); - BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent name"); + BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent name"); session.close(); assertFalse(session.isOpen()); @@ -149,8 +149,8 @@ public void handshakeUsesHttpLogApplicationIdFallback() { .httpLogOptions(new HttpLogOptions().setApplicationId("log-app")) .clientOptions(new ClientOptions()); - BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent"); + BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent"); session.close(); assertTrue(userAgent.get().startsWith("log-app azsdk-java-azure-ai-agents/"), userAgent.get()); @@ -167,8 +167,8 @@ public void typedStringAndMappingSendsRejectInvalidJson() { .configuration(Configuration.NONE); String raw = "{\"type\": \"response.create\"}"; BinaryData mapping = BinaryData.fromObject(Collections.singletonMap("type", "response.cancel")); - try (BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent", tlsOptions())) { + try (BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent", tlsOptions())) { assertThrows(IllegalArgumentException.class, () -> session.sendEvent(BinaryData.fromString("not valid json"))); session.sendEvent(new RealtimeResponseCreateEvent()); @@ -200,8 +200,8 @@ public void pingPongFramesAreNotApplicationEvents() { .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))) .configuration(Configuration.NONE); List events = new ArrayList<>(); - try (BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent", tlsOptions())) { + try (BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent", tlsOptions())) { Iterator iterator = session.receiveEvents(Duration.ofSeconds(5)).iterator(); events.add(iterator.next()); events.add(iterator.next()); @@ -224,8 +224,8 @@ public void malformedEventsCanBeReportedAndSkipped() { AgentsClientBuilder builder = new AgentsClientBuilder().endpoint("https://localhost:" + server.port()) .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))) .configuration(Configuration.NONE); - try (BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent", options)) { + try (BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent", options)) { Iterator events = session.receiveEvents(Duration.ofSeconds(5)).iterator(); assertWarningEvent(events.next()); assertWarningEvent(events.next()); @@ -246,8 +246,8 @@ public void boundedQueuesHonorOverflowPolicies() { .configuration(Configuration.NONE); List received = new ArrayList<>(); boolean overflowError = strategy == VoiceAgentWebSocketOverflowStrategy.ERROR; - try (BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent", options)) { + try (BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent", options)) { assertTimeoutPreemptively(Duration.ofSeconds(5), () -> { while (session.isOpen()) { Thread.yield(); @@ -282,8 +282,8 @@ public void messageSizeLimitCannotBeBypassedByRecoveryHandler() { .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))) .configuration(Configuration.NONE); assertThrows(RuntimeException.class, () -> { - try (BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent", options)) { + try (BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent", options)) { session.receiveEvents(Duration.ofSeconds(5)).iterator().next(); } }); @@ -304,7 +304,7 @@ public void insecureEndpointsAreRejectedBeforeAuthentication() { "https://example.com/#fragment" }) { AgentsClientBuilder builder = new AgentsClientBuilder().endpoint(endpoint).credential(credential); assertThrows(IllegalArgumentException.class, - () -> builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent")); + () -> builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent")); } assertFalse(requested.get()); } @@ -322,13 +322,13 @@ private static VoiceAgentWebSocketConnectionOptions tlsOptions() { public void syncReceiveStreamRejectsSecondIterator() { server = startServer(new CopyOnWriteArrayList<>(), new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), new AtomicReference<>(), false); - BetaVoiceAgentWebSocketClient client + BetaVoiceAgentsWebSocketClient client = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))) .beta() - .buildBetaVoiceAgentWebSocketClient(); + .buildBetaVoiceAgentsWebSocketClient(); - try (BetaVoiceAgentWebSocketSessionClient session = client.openWebSocketSession("agent", tlsOptions())) { + try (BetaVoiceAgentsWebSocketSessionClient session = client.openWebSocketSession("agent", tlsOptions())) { Iterable events = session.receiveEvents(Duration.ofSeconds(5)); events.iterator(); IllegalStateException exception = assertThrows(IllegalStateException.class, events::iterator); @@ -349,8 +349,8 @@ public void rawEventsUseCustomizedTlsTransport() { .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))) .configuration(Configuration.NONE); BinaryData payload = BinaryData.fromString("{\"type\":\"future.event\",\"value\":42}"); - try (BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent", tlsOptions())) { + try (BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent", tlsOptions())) { assertThrows(IllegalArgumentException.class, () -> session.sendEvent(BinaryData.fromString("[]"))); assertThrows(IllegalArgumentException.class, () -> session.sendEvent(BinaryData.fromString("{} {}"))); session.sendEvent(payload); @@ -368,8 +368,8 @@ public void customCloseFrameAndReceiveTimeout() { AgentsClientBuilder builder = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1)))); - try (BetaVoiceAgentWebSocketSessionClient session - = builder.beta().buildBetaVoiceAgentWebSocketClient().openWebSocketSession("agent", tlsOptions())) { + try (BetaVoiceAgentsWebSocketSessionClient session + = builder.beta().buildBetaVoiceAgentsWebSocketClient().openWebSocketSession("agent", tlsOptions())) { Iterator iterator = session.receiveEvents(Duration.ofMillis(20)).iterator(); IllegalStateException timeout = assertThrows(IllegalStateException.class, iterator::hasNext); assertInstanceOf(TimeoutException.class, timeout.getCause()); @@ -400,12 +400,12 @@ static void restoreTlsConfiguration() { public void syncConnectRejectsNullArguments() { TokenCredential credential = request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1))); - BetaVoiceAgentWebSocketClient client + BetaVoiceAgentsWebSocketClient client = new AgentsClientBuilder().endpoint("https://localhost/api/projects/project") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketClient(); + .buildBetaVoiceAgentsWebSocketClient(); NullPointerException agentNameException = assertThrows(NullPointerException.class, () -> client.openWebSocketSession(null, tlsOptions())); @@ -424,12 +424,12 @@ public void syncTokenFailureOccursBeforeNetworkAccess() { return response.send(); }).bindNow(); TokenCredential credential = request -> Mono.error(new IllegalStateException("token unavailable")); - BetaVoiceAgentWebSocketClient client + BetaVoiceAgentsWebSocketClient client = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketClient(); + .buildBetaVoiceAgentsWebSocketClient(); IllegalStateException exception = assertThrows(IllegalStateException.class, () -> client.openWebSocketSession("agent", tlsOptions())); @@ -446,12 +446,12 @@ public void syncRejectedHandshakeMapsConflictToAzureException() { .bindNow(); TokenCredential credential = request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1))); - BetaVoiceAgentWebSocketClient client + BetaVoiceAgentsWebSocketClient client = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketClient(); + .buildBetaVoiceAgentsWebSocketClient(); ResourceModifiedException exception = assertThrows(ResourceModifiedException.class, () -> client.openWebSocketSession("disabled-agent", tlsOptions())); @@ -463,11 +463,11 @@ public void syncRejectedHandshakeMapsConflictToAzureException() { public void syncClientRejectsEmptyAgentName() { TokenCredential credential = request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1))); - BetaVoiceAgentWebSocketClient client = new AgentsClientBuilder().endpoint("https://example.com") + BetaVoiceAgentsWebSocketClient client = new AgentsClientBuilder().endpoint("https://example.com") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketClient(); + .buildBetaVoiceAgentsWebSocketClient(); IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, () -> client.openWebSocketSession("", tlsOptions())); @@ -480,14 +480,14 @@ public void syncReceiveBufferOverflowFailsTheEventStream() { server = frameWebSocketServer(frames); TokenCredential credential = request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1))); - BetaVoiceAgentWebSocketClient client + BetaVoiceAgentsWebSocketClient client = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketClient(); + .buildBetaVoiceAgentsWebSocketClient(); - try (BetaVoiceAgentWebSocketSessionClient session = client.openWebSocketSession("agent", tlsOptions())) { + try (BetaVoiceAgentsWebSocketSessionClient session = client.openWebSocketSession("agent", tlsOptions())) { assertTimeoutPreemptively(Duration.ofSeconds(5), () -> { while (session.isOpen()) { Thread.yield(); @@ -505,14 +505,14 @@ public void syncOrderlyClosePreservesFullReceiveBuffer() { server = frameWebSocketServer(frames); TokenCredential credential = request -> Mono.just(new AccessToken("test-token", OffsetDateTime.now().plusHours(1))); - BetaVoiceAgentWebSocketClient client + BetaVoiceAgentsWebSocketClient client = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketClient(); + .buildBetaVoiceAgentsWebSocketClient(); - try (BetaVoiceAgentWebSocketSessionClient session = client.openWebSocketSession("agent", tlsOptions())) { + try (BetaVoiceAgentsWebSocketSessionClient session = client.openWebSocketSession("agent", tlsOptions())) { assertTimeoutPreemptively(Duration.ofSeconds(5), () -> { Iterator events = session.receiveEvents().iterator(); int eventCount = 0; @@ -536,14 +536,14 @@ public void syncSessionReceivesTypedEventAndCloses() { new AtomicReference<>(), true); TokenCredential credential = request -> Mono.just(new AccessToken("sync-token", OffsetDateTime.now().plusHours(1))); - BetaVoiceAgentWebSocketClient client + BetaVoiceAgentsWebSocketClient client = new AgentsClientBuilder().endpoint("https://localhost:" + server.port() + "/api/projects/project") .credential(credential) .configuration(Configuration.NONE) .beta() - .buildBetaVoiceAgentWebSocketClient(); + .buildBetaVoiceAgentsWebSocketClient(); - try (BetaVoiceAgentWebSocketSessionClient session = client.openWebSocketSession("sync-agent", tlsOptions())) { + try (BetaVoiceAgentsWebSocketSessionClient session = client.openWebSocketSession("sync-agent", tlsOptions())) { Iterator events = session.receiveEvents().iterator(); assertWarningEvent(events.next()); session.sendFunctionCallOutput("call-1", "{\"temperature\":72}");