Skip to content

Commit c6befb8

Browse files
committed
Add x.ai Grok voice model support
1 parent d42d6da commit c6befb8

9 files changed

Lines changed: 173 additions & 35 deletions

File tree

‎README.md‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ A multimodal AI system for the Pepper robot powered by advanced speech-to-speech
155155
- **Real-time Voice Chat** - Natural conversations powered by speech-to-speech (S2S) AI models:
156156
- **OpenAI** - gpt-realtime, gpt-realtime-mini, gpt-4o-realtime-preview, gpt-4o-mini-realtime-preview
157157
- **Azure OpenAI** - Same models with network-level isolation and customer-managed encryption for enterprise compliance
158-
- **x.ai** - Grok Voice Agent API with native web/X search
158+
- **x.ai** - `grok-voice-think-fast-1.0` (default) and `grok-voice-fast-1.0` Voice Agent API with native web/X search
159159
- **Google** - Gemini Live API with native audio model and 30 voices
160160
- **Audio Input Modes** - Two optimized modes for voice capture:
161161
- **Direct Audio** - Low-latency streaming with native voice activity detection
@@ -476,12 +476,18 @@ This allows the Android app to automatically start the face recognition server v
476476
#### Option 3: x.ai Grok Voice Agent API (Alternative with Native Search)
477477
1. Go to [x.ai](https://x.ai/)
478478
2. Create an API key
479-
3. Uses the Grok Voice Agent API (speech-to-speech model, OpenAI Realtime API compatible)
479+
3. Uses x.ai Voice Agent models (speech-to-speech, OpenAI Realtime API compatible)
480+
4. Add the key to `local.properties`:
481+
```properties
482+
XAI_API_KEY=your_xai_api_key_here
483+
```
480484

481485
**Unique Features:**
486+
- **Model Selection**: Defaults to `grok-voice-think-fast-1.0`; `grok-voice-fast-1.0` remains selectable in settings
487+
- **Explicit Realtime Model URL**: Uses `wss://api.x.ai/v1/realtime?model=<selected-model>`
482488
- **Native Web Search**: Built-in web search without Tavily API (configurable in settings)
483489
- **Native X Search**: Search posts on X/Twitter in real-time (configurable in settings)
484-
- **5 Distinct Voices**: Ara, Rex, Sal, Eve, Leo
490+
- **5 Distinct Voices**: `eve`, `ara`, `rex`, `sal`, `leo`
485491
- **100+ Languages**: Multilingual support out of the box
486492
- **Note**: Vision analysis requires `GROQ_API_KEY` (Grok Voice Agent API doesn't support images)
487493

@@ -639,7 +645,7 @@ You can switch between different AI providers in the settings:
639645

640646
- **OpenAI Direct** (Realtime API): Supports all four models (`gpt-realtime`, `gpt-realtime-mini`, `gpt-4o-realtime-preview`, `gpt-4o-mini-realtime-preview`) directly from OpenAI with 10 voices
641647
- **Azure OpenAI** (Realtime API): Supports all four models with your Azure deployment, network-level isolation, and customer-managed encryption for enterprise compliance
642-
- **x.ai Grok** (Grok Voice Agent API): Speech-to-speech voice model with native web/X search capabilities and 5 unique voices
648+
- **x.ai Grok** (Grok Voice Agent API): `grok-voice-think-fast-1.0` by default, with `grok-voice-fast-1.0` also selectable; native web/X search capabilities, configurable server VAD, and 5 voices (`eve`, `ara`, `rex`, `sal`, `leo`)
643649
- **Google Gemini** (Live API): Native audio model with end-to-end speech, configurable VAD, thinking budget, and 30 voices
644650

645651
**Note**: Changing the API provider automatically updates available models and voices. Model/voice changes restart the session automatically.
@@ -1440,7 +1446,7 @@ Robot: "Happy Birthday! 🎂"
14401446
```
14411447

14421448
**Key Flow Characteristics:**
1443-
- **Model Flexibility**: Supports OpenAI (gpt-4o-realtime, gpt-4o-mini-realtime) and x.ai Grok Voice Agent
1449+
- **Model Flexibility**: Supports OpenAI Realtime models, Google Gemini Live models, and x.ai `grok-voice-think-fast-1.0` / `grok-voice-fast-1.0`
14441450
- **Dual Audio Input**: Realtime API (simple) or Azure Speech (dialect quality)
14451451
- **Server-side VAD**: Realtime API handles turn detection automatically
14461452
- **Conditional Tool Calls**: AI decides when tools are needed (not every response uses tools)
@@ -1664,6 +1670,7 @@ app/src/
16641670
- Check your OpenAI, Azure OpenAI, x.ai, or Google API key and endpoint
16651671
- Verify internet connectivity
16661672
- For Azure: Ensure the deployment name matches your Azure setup
1673+
- For x.ai: Ensure `XAI_API_KEY` is set and the selected model is `grok-voice-think-fast-1.0` or `grok-voice-fast-1.0`
16671674

16681675
#### "Vision analysis not available"
16691676
- Vision works automatically with `gpt-realtime` and `gpt-realtime-mini` models (built-in)
@@ -1762,7 +1769,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
17621769
## 🙏 Acknowledgments
17631770

17641771
- **OpenAI** - Realtime API
1765-
- **x.ai** - Grok Voice Agent API
1772+
- **x.ai** - Grok Voice Agent API (`grok-voice-think-fast-1.0`, `grok-voice-fast-1.0`)
17661773
- **SoftBank Robotics** - Pepper robot platform
17671774
- **Microsoft** - Azure Speech and OpenAI services
17681775
- **Groq** - Alternative vision analysis provider

‎app/src/main/java/ch/fhnw/pepper_realtime/controller/RobotFocusManager.kt‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ class RobotFocusManager(private val activity: ChatActivity) {
2626
var qiContext: Any? = null
2727
private set
2828
private var listener: Listener? = null
29+
private var lastFocusRefusedReason: String? = null
2930

3031
interface Listener {
3132
fun onRobotReady(robotContext: Any?)
@@ -51,6 +52,10 @@ class RobotFocusManager(private val activity: ChatActivity) {
5152
override fun onRobotFocusLost() {
5253
handleRobotFocusLost()
5354
}
55+
56+
override fun onRobotFocusRefused(reason: String) {
57+
handleRobotFocusRefused(reason)
58+
}
5459
})
5560

5661
startFocusTimeout()
@@ -72,10 +77,16 @@ class RobotFocusManager(private val activity: ChatActivity) {
7277
if (!isFocusAvailable) {
7378
Log.e(TAG, "🤖 DIAGNOSTIC: TIMEOUT - No robot focus response after 30 seconds!")
7479
Log.e(TAG, "🤖 DIAGNOSTIC: Mode: ${robotController.getModeName()}")
80+
lastFocusRefusedReason?.let {
81+
Log.e(TAG, "🤖 DIAGNOSTIC: Last focus-refused reason reported by QiSDK: $it")
82+
}
7583
if (robotController.isRobotHardwareAvailable()) {
7684
Log.e(TAG, "🤖 DIAGNOSTIC: Check: 1) Robot is awake, 2) Robot is enabled, 3) QiSDK service is running")
7785
}
78-
listener?.onRobotInitializationFailed(activity.getString(R.string.robot_initialization_timeout))
86+
val message = lastFocusRefusedReason?.let { reason ->
87+
activity.getString(R.string.robot_initialization_timeout) + " (reason: $reason)"
88+
} ?: activity.getString(R.string.robot_initialization_timeout)
89+
listener?.onRobotInitializationFailed(message)
7990
}
8091
}
8192
focusTimeoutHandler.postDelayed(focusTimeoutRunnable!!, FOCUS_TIMEOUT_MS)
@@ -108,5 +119,13 @@ class RobotFocusManager(private val activity: ChatActivity) {
108119

109120
listener?.onRobotFocusLost()
110121
}
122+
123+
private fun handleRobotFocusRefused(reason: String) {
124+
// Cache the latest refusal reason so the timeout path can surface it in the UI.
125+
// We do NOT fail fast here because QiSDK sometimes recovers after a refusal
126+
// and grants focus on a subsequent attempt within the timeout window.
127+
Log.e(TAG, "🤖 Robot focus refused by QiSDK: $reason")
128+
lastFocusRefusedReason = reason
129+
}
111130
}
112131

‎app/src/main/java/ch/fhnw/pepper_realtime/manager/SettingsRepository.kt‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ import android.util.Log
66
import dagger.hilt.android.qualifiers.ApplicationContext
77
import ch.fhnw.pepper_realtime.R
88
import ch.fhnw.pepper_realtime.network.RealtimeApiProvider
9+
import ch.fhnw.pepper_realtime.network.XAI_LEGACY_MODEL_LABEL
10+
import ch.fhnw.pepper_realtime.network.XAI_THINK_FAST_MODEL
911
import ch.fhnw.pepper_realtime.tools.ToolRegistry
1012
import ch.fhnw.pepper_realtime.ui.settings.LanguageOption
1113
import javax.inject.Inject
@@ -35,28 +37,38 @@ class SettingsRepository @Inject constructor(
3537
var model: String
3638
get() {
3739
val defaultModel = when (apiProviderEnum) {
38-
RealtimeApiProvider.XAI -> "Grok Voice Agent"
40+
RealtimeApiProvider.XAI -> XAI_THINK_FAST_MODEL
3941
RealtimeApiProvider.GOOGLE_GEMINI -> "models/gemini-3.1-flash-live-preview"
4042
else -> context.getString(R.string.openai_default_model)
4143
}
4244
val key = "${KEY_MODEL}_${apiProvider}"
43-
return settings.getString(key, null)
45+
val savedModel = settings.getString(key, null)
4446
?: settings.getString(KEY_MODEL, defaultModel)
4547
?: defaultModel
48+
return if (apiProviderEnum == RealtimeApiProvider.XAI && savedModel == XAI_LEGACY_MODEL_LABEL) {
49+
XAI_THINK_FAST_MODEL
50+
} else {
51+
savedModel
52+
}
4653
}
4754
set(value) = settings.edit().putString("${KEY_MODEL}_${apiProvider}", value).apply()
4855

4956
var voice: String
5057
get() {
5158
val defaultVoice = when (apiProviderEnum) {
52-
RealtimeApiProvider.XAI -> "Ara"
59+
RealtimeApiProvider.XAI -> "ara"
5360
RealtimeApiProvider.GOOGLE_GEMINI -> "Puck"
5461
else -> "ash"
5562
}
5663
val key = "${KEY_VOICE}_${apiProvider}"
57-
return settings.getString(key, null)
64+
val savedVoice = settings.getString(key, null)
5865
?: settings.getString(KEY_VOICE, defaultVoice)
5966
?: defaultVoice
67+
return if (apiProviderEnum == RealtimeApiProvider.XAI) {
68+
savedVoice.lowercase()
69+
} else {
70+
savedVoice
71+
}
6072
}
6173
set(value) = settings.edit().putString("${KEY_VOICE}_${apiProvider}", value).apply()
6274

‎app/src/main/java/ch/fhnw/pepper_realtime/network/RealtimeApiProvider.kt‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@ package ch.fhnw.pepper_realtime.network
22

33
import android.util.Log
44

5+
const val XAI_THINK_FAST_MODEL = "grok-voice-think-fast-1.0"
6+
const val XAI_FAST_MODEL = "grok-voice-fast-1.0"
7+
const val XAI_LEGACY_MODEL_LABEL = "Grok Voice Agent"
8+
59
/**
610
* Enumeration of available Realtime API providers
711
* Supports Azure OpenAI, direct OpenAI, and x.ai connections
@@ -12,7 +16,7 @@ enum class RealtimeApiProvider(
1216
) {
1317
AZURE_OPENAI("Azure OpenAI", "gpt-4o-realtime-preview"),
1418
OPENAI_DIRECT("OpenAI Direct", "gpt-realtime-1.5"),
15-
XAI("x.ai Grok", "Grok Voice Agent"),
19+
XAI("x.ai Grok", XAI_THINK_FAST_MODEL),
1620
// Google Live API requires models/ prefix for BidiGenerateContent
1721
GOOGLE_GEMINI("Google Gemini", "models/gemini-3.1-flash-live-preview");
1822

@@ -27,7 +31,12 @@ enum class RealtimeApiProvider(
2731
* @return WebSocket URL
2832
*/
2933
fun getWebSocketUrl(azureEndpoint: String?, model: String?, googleApiKey: String? = null): String {
30-
val actualModel = if (!model.isNullOrEmpty()) model else modelName
34+
val requestedModel = if (!model.isNullOrBlank()) model.trim() else modelName
35+
val actualModel = if (this == XAI && requestedModel == XAI_LEGACY_MODEL_LABEL) {
36+
XAI_THINK_FAST_MODEL
37+
} else {
38+
requestedModel
39+
}
3140
return when (this) {
3241
AZURE_OPENAI -> if (isOpenAiGaRealtimeModel(actualModel)) {
3342
// Azure GA Realtime endpoint
@@ -37,7 +46,7 @@ enum class RealtimeApiProvider(
3746
"wss://$azureEndpoint/openai/realtime?api-version=2025-04-01-preview&deployment=$actualModel"
3847
}
3948
OPENAI_DIRECT -> "wss://api.openai.com/v1/realtime?model=$actualModel"
40-
XAI -> "wss://api.x.ai/v1/realtime"
49+
XAI -> "wss://api.x.ai/v1/realtime?model=$actualModel"
4150
// v1alpha is required for newer native audio models
4251
GOOGLE_GEMINI -> "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key=${googleApiKey ?: ""}"
4352
}

‎app/src/main/java/ch/fhnw/pepper_realtime/network/RealtimeSessionManager.kt‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -920,19 +920,20 @@ class RealtimeSessionManager @Inject constructor() {
920920

921921
sessionConfig.put("audio", audio)
922922

923-
// x.ai voice: Use directly if it's an x.ai voice, otherwise map from OpenAI names
924-
val xaiVoices = setOf("Ara", "Rex", "Sal", "Eve", "Leo")
925-
val xaiVoice = if (voice in xaiVoices) {
926-
voice // Already an x.ai voice
923+
// x.ai voice IDs are lowercase: eve, ara, rex, sal, leo.
924+
val normalizedVoice = voice.lowercase()
925+
val xaiVoices = setOf("ara", "rex", "sal", "eve", "leo")
926+
val xaiVoice = if (normalizedVoice in xaiVoices) {
927+
normalizedVoice
927928
} else {
928929
// Fallback mapping from OpenAI voice names (for backwards compatibility)
929-
when (voice.lowercase()) {
930-
"alloy", "ash" -> "Ara"
931-
"echo", "ballad" -> "Rex"
932-
"shimmer", "coral" -> "Eve"
933-
"verse" -> "Sal"
934-
"sage" -> "Leo"
935-
else -> "Ara"
930+
when (normalizedVoice) {
931+
"alloy", "ash" -> "ara"
932+
"echo", "ballad" -> "rex"
933+
"shimmer", "coral" -> "eve"
934+
"verse" -> "sal"
935+
"sage" -> "leo"
936+
else -> "ara"
936937
}
937938
}
938939
sessionConfig.put("voice", xaiVoice)
@@ -941,6 +942,9 @@ class RealtimeSessionManager @Inject constructor() {
941942
if (settings.isUsingRealtimeAudioInput) {
942943
sessionConfig.put("turn_detection", JSONObject().apply {
943944
put("type", "server_vad")
945+
put("threshold", settings.vadThreshold.coerceIn(0.1f, 0.9f))
946+
put("prefix_padding_ms", settings.prefixPadding)
947+
put("silence_duration_ms", settings.silenceDuration)
944948
})
945949
}
946950

@@ -1147,4 +1151,3 @@ class RealtimeSessionManager @Inject constructor() {
11471151
}
11481152
}
11491153

1150-

‎app/src/main/java/ch/fhnw/pepper_realtime/robot/RobotLifecycleBridge.kt‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,15 @@ interface RobotLifecycleBridge {
3939
* Called when robot focus is lost (Pepper) or simulation paused (Standalone)
4040
*/
4141
fun onRobotFocusLost()
42+
43+
/**
44+
* Called when QiSDK refuses to grant robot focus (Pepper only).
45+
* Surfaces the exact reason reported by the robot service so callers can
46+
* display it instead of a generic timeout.
47+
*
48+
* Default implementation is empty to avoid breaking existing listeners.
49+
*/
50+
fun onRobotFocusRefused(reason: String) {}
4251
}
4352
}
4453

‎app/src/main/java/ch/fhnw/pepper_realtime/ui/compose/settings/SettingsScreen.kt‎

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ import androidx.lifecycle.compose.collectAsStateWithLifecycle
1414
import ch.fhnw.pepper_realtime.manager.ApiKeyManager
1515
import ch.fhnw.pepper_realtime.manager.SettingsRepository
1616
import ch.fhnw.pepper_realtime.network.RealtimeApiProvider
17+
import ch.fhnw.pepper_realtime.network.XAI_FAST_MODEL
18+
import ch.fhnw.pepper_realtime.network.XAI_THINK_FAST_MODEL
1719
import ch.fhnw.pepper_realtime.tools.ToolRegistry
1820
import ch.fhnw.pepper_realtime.ui.compose.ChatTheme
1921
import ch.fhnw.pepper_realtime.ui.settings.SettingsViewModel
@@ -61,7 +63,10 @@ fun SettingsScreen(
6163
"gpt-4o-realtime-preview",
6264
"gpt-4o-mini-realtime-preview"
6365
)
64-
val xaiModels = listOf("Grok Voice Agent")
66+
val xaiModels = listOf(
67+
XAI_THINK_FAST_MODEL,
68+
XAI_FAST_MODEL
69+
)
6570
// Google Live API requires models/ prefix for BidiGenerateContent
6671
// Include fallback model for debugging
6772
val googleModels = listOf(
@@ -88,11 +93,11 @@ fun SettingsScreen(
8893
"verse" to "Versatile, expressive"
8994
)
9095
val xaiVoices = linkedMapOf(
91-
"Ara" to "Female, warm, friendly",
92-
"Rex" to "Male, confident, clear",
93-
"Sal" to "Neutral, smooth, balanced",
94-
"Eve" to "Female, energetic, upbeat",
95-
"Leo" to "Male, authoritative, strong"
96+
"eve" to "Female",
97+
"ara" to "Female",
98+
"rex" to "Male",
99+
"sal" to "Male",
100+
"leo" to "Male"
96101
)
97102
val googleVoices = linkedMapOf(
98103
"Zephyr" to "Bright",
@@ -263,12 +268,12 @@ fun SettingsScreen(
263268
SettingsVoiceDropdown(
264269
label = "Voice",
265270
voices = voices,
266-
selectedVoice = settings.voice,
271+
selectedVoice = if (isXaiProvider) settings.voice.lowercase() else settings.voice,
267272
onVoiceSelected = { viewModel.setVoice(it) }
268273
)
269274

270-
// Speed and Temperature - not supported by Google Live API
271-
if (!isGoogleProvider) {
275+
// Speed and Temperature - not supported by Google Live API or x.ai Voice Agent
276+
if (!isGoogleProvider && !isXaiProvider) {
272277
SettingsSlider(
273278
label = "Speech Speed",
274279
value = settings.speedProgress.toFloat(),
@@ -415,6 +420,33 @@ fun SettingsScreen(
415420
checked = settings.googleContextCompression,
416421
onCheckedChange = { viewModel.setGoogleContextCompression(it) }
417422
)
423+
} else if (isXaiProvider) {
424+
// x.ai Voice Agent Settings
425+
SettingsSectionHeader(title = "x.ai Voice Agent Settings")
426+
427+
SettingsSlider(
428+
label = "VAD Activation Threshold",
429+
value = settings.vadThreshold.coerceIn(0.1f, 0.9f),
430+
onValueChange = { viewModel.setVadThreshold(it) },
431+
valueRange = 0.1f..0.9f,
432+
valueDisplay = "%.2f".format(settings.vadThreshold.coerceIn(0.1f, 0.9f))
433+
)
434+
435+
SettingsSlider(
436+
label = "Prefix Padding",
437+
value = settings.prefixPadding.toFloat(),
438+
onValueChange = { viewModel.setPrefixPadding(it.toInt()) },
439+
valueRange = 0f..1000f,
440+
valueDisplay = "${settings.prefixPadding} ms"
441+
)
442+
443+
SettingsSlider(
444+
label = "Silence Duration",
445+
value = settings.silenceDuration.toFloat(),
446+
onValueChange = { viewModel.setSilenceDuration(it.toInt()) },
447+
valueRange = 200f..2000f,
448+
valueDisplay = "${settings.silenceDuration} ms"
449+
)
418450
} else {
419451
// OpenAI Realtime API Settings
420452
SettingsSectionHeader(title = "Voice API Settings")

‎app/src/pepper/java/ch/fhnw/pepper_realtime/robot/RobotLifecycleBridgeImpl.kt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ class RobotLifecycleBridgeImpl : RobotLifecycleBridge, RobotLifecycleCallbacks {
5252

5353
override fun onRobotFocusRefused(reason: String) {
5454
Log.e(TAG, "🤖 Robot focus refused: $reason")
55+
listener?.onRobotFocusRefused(reason)
5556
}
5657
}
5758

0 commit comments

Comments
 (0)