Skip to content

voice-conversation.md

Latest commit

 

History

History
200 lines (166 loc) · 8.84 KB

voice-conversation.md

File metadata and controls

200 lines (166 loc) · 8.84 KB
Raw

Bounded voice conversation

Voice is a bounded interaction capability, not Mira's execution backbone. The robot continues to exist, observe, pursue goals, and exercise bounded initiative without an active microphone or model stream.

The implementation is deliberately half-duplex and enters the same bounded conversation either from the console voice command or an optional local, exact recognition of a configured local wake phrase. A wake-triggered session first plays a fixed, local acknowledgement chirp (100 ms at 880 Hz, a 20 ms pause, then 100 ms at 1,175 Hz) and only then begins turn-1 listening. The cue is deterministic speaker output: it does not involve cognition, TTS, working memory, RuntimeState, or the EventBus. Manual voice sessions do not play it.

  1. Enter voice in the runtime console.
  2. The Fusion HAT voice provider uses local SunFounder Vosk STT to listen for one utterance (18 seconds by default).
  3. VoiceInteraction sends recognized text through the same application cognition path as console ask.
  4. The final text response is passed unchanged to a separate TextToSpeechProvider; the selected eSpeak, Piper, or OpenAI provider speaks it through the Fusion HAT speaker.
  5. Vosk offers one short follow-up opportunity (10 seconds by default).
  6. The session closes after the second utterance or a timeout, stops capture, and disables the Fusion HAT speaker.

Physical voice is opt-in and is only constructed for the Fusion HAT backend. A minimal configuration is:

[voice]
enabled = true
wake_word_enabled = true
wake_words = ["mira", "mirror"]
tts = "espeak"
initial_timeout_seconds = 18
followup_timeout_seconds = 10

The application constructs FusionHatVoiceProvider for local STT and wake interaction and the selected physical speech provider as distinct dependencies. eSpeak is the historical default. To use offline neural speech, install the optional maintained OHF Piper package and an operator-managed voice:

python -m pip install -e '.[openai,piper]'
mkdir -p ~/.local/share/embodied-runtime/piper
python -m piper.download_voices \
  --data-dir ~/.local/share/embodied-runtime/piper \
  en_US-lessac-medium

python -m pip install -e '.[piper]' is sufficient when OpenAI support is not needed. Confirm that both en_US-lessac-medium.onnx and en_US-lessac-medium.onnx.json were downloaded, then select it:

[voice]
enabled = true
wake_word_enabled = true
wake_words = ["mira", "mirror"]
tts = "piper"
piper_model = "/home/pi/.local/share/embodied-runtime/piper/en_US-lessac-medium.onnx"
initial_timeout_seconds = 18
followup_timeout_seconds = 10

~ is expanded in model paths. Piper loads the local model on first speech, keeps that one voice resident across turns and sessions, synthesizes a complete WAV in memory, and plays it through the Fusion HAT speaker. Once the package and model files are present, synthesis requires no network. Startup does not download models, and there is no automatic fallback to eSpeak. Voice models can carry their own dataset/model licensing terms; review the voice's model card before selecting or distributing it. en_US-lessac-medium is only the first benchmark voice, not a hard-coded runtime choice.

VoiceInteraction coordinates both providers and keeps microphone capture and TTS playback half-duplex. It owns session-level coordination while input cleanup remains with the voice provider and speaker cleanup remains with the TTS provider; failure to close either one does not skip the other cleanup attempt.

The narrow TextToSpeechProvider seam keeps selection from changing the bounded conversation architecture. There is no provider probing, fallback, runtime switching, streaming synthesis, or TTS text rewriting.

Hosted speech is an explicit third option:

[voice]
enabled = true
tts = "openai"
openai_tts_model = "gpt-4o-mini-tts"
openai_tts_voice = "cedar"

Install it with python -m pip install -e '.[openai]' and provide OPENAI_API_KEY in the environment, never in TOML. OpenAI TTS requires network access. This initial implementation requests the complete WAV response in memory, then passes it to aplay; it does not stream or create a temporary file. eSpeak and Piper remain explicit alternatives, and failures never trigger an automatic fallback. Applications using OpenAI-generated voices must clearly disclose to end users that the voice they hear is AI-generated.

ElevenLabs-hosted speech is a fourth explicit option:

[voice]
enabled = true
tts = "elevenlabs"
elevenlabs_tts_model = "eleven_flash_v2_5"
elevenlabs_tts_voice_id = "YOUR_VOICE_ID"
elevenlabs_tts_speed = 1.0

Install the official SDK with python -m pip install -e '.[elevenlabs]', select and copy a voice ID from ElevenLabs, and provide the credential only through the environment:

export ELEVENLABS_API_KEY='your-key-from-a-secure-source'

ElevenLabs requires network access and a voice ID; the initial low-latency model default is eleven_flash_v2_5. The provider requests wav_24000 and collects all asynchronously delivered SDK chunks into one complete in-memory WAV before enabling the speaker and invoking aplay. This is not streaming playback and does not create temporary files. eSpeak, Piper, and OpenAI remain explicit alternatives, and there is no automatic fallback.

elevenlabs_tts_speed accepts numeric values from 0.7 through 1.2; values above 1.0 speak faster. It is sent as request-level VoiceSettings(speed=...) and neither changes the saved ElevenLabs voice nor overrides stability, similarity boost, style, or speaker boost. Mira's checked-in physical profile selects ElevenLabs model eleven_flash_v2_5, voice ID pFZP5JQG7iQjIQuC4Bku, and speed 1.1 (110%). With the optional ElevenLabs dependency installed and ELEVENLABS_API_KEY available in the environment, normal Mira startup needs no TTS override:

python main.py --config config/mira-agentic.toml

OpenAI model gpt-4o-mini-tts with voice marin remains configured as an alternative. An operator can select it temporarily for comparison or fallback:

python main.py \
  --config config/mira-agentic.toml \
  --tts openai

This is explicit operator selection; a missing ElevenLabs credential or other ElevenLabs failure does not trigger automatic fallback.

Hosted synthesis logs derive audio_ms by reading the PCM frames actually present in the returned in-memory WAV in bounded chunks. They do not blindly trust declared RIFF/data lengths, which may use an unknown-length sentinel.

Wake listening is local trigger detection, not an always-running cloud conversation. Matching is case-insensitive and exact after trimming against the configured phrases. "mirror" is intentionally accepted because live Vosk testing commonly returned it for the spoken name “Mira”. Ambient non-matches remain local and never enter cognition or working memory. There is one microphone owner: a manual or wake-triggered bounded session cooperatively stops the wake capture, owns recognition through the wake cue (when applicable), all STT, TTS, speaker disable, and cleanup, and only then allows wake listening to resume. Cue playback finishes and disables the speaker before turn-1 microphone capture starts. Failure to play the acknowledgement is logged but does not prevent the bounded session. The manual console command remains available while wake mode is on.

The Vosk model may be downloaded by the vendor library on the first listen command. Initialization and download errors are reported for that session and do not stop the runtime. Raw audio and partial recognition results remain transient; they are never placed in RuntimeState, the EventBus, or persistent memory.

The onboard MEMS microphone has shown low native sensitivity in bench testing. Approximately +30 dB post-capture gain helped a sample recording, but this path intentionally preserves the vendor capture implementation. Recognition quality and input-level tuning require live validation before adding preprocessing.

Intentionally out of scope

This version does not implement wake-word-plus-command parsing, fuzzy wake aliases, continuous cloud transcription, a realtime LLM audio stream, full duplex or interruption, barge-in, background recording, persistent audio storage, OpenAI realtime audio, or SunFounder's VoiceAssistant orchestration.

Successful third-party HTTP request INFO lines are intentionally hidden in normal logging; first-party lifecycle INFO records and third-party warnings and errors remain visible.

Attention boundary

A voice session is transport, not an attention episode. Each recognized utterance is passed to RobotApplication with source voice and creates a separate operator episode. That episode closes before TTS playback and before follow-up listening; a second voice turn creates a new episode and obtains continuity from bounded WorkingMemory. Voice owns no episode identity or provider conversation session.