agent-voice

A fully local voice stack for agents: microphone capture, voice-activity detection, speech-to-text, and text-to-speech — no cloud audio services. Built to be embedded in an agent host (gilamonster-agent / newt-agent) either as an in-process library or as a stdio MCP server.

Supported platforms: Linux, macOS, Windows x86_64 (audio via cpal; ONNX Runtime via the ort crate's prebuilt binaries; whisper.cpp built from source by whisper-rs). Windows-on-ARM is currently not supported: the piper-phonemize-sys crate ships no arm64 Windows prebuilt and has no from-source fallback there.

Crate map

Crate	Role
agent-voice	Facade. VoiceStack bundles the four services below behind listen() / say() / transcribe(). This is the crate a host embeds. Feature mcp re-exports the MCP server.
agent-voice-core	Shared types and traits (AudioIO, Vad, Stt, Tts, configs, results). No heavy deps.
agent-voice-audio	Capture/playback via cpal (CoreAudio / ALSA / WASAPI).
agent-voice-vad	Silero VAD via voice_activity_detector (model bundled — nothing to download).
agent-voice-stt	Whisper via whisper-rs (whisper.cpp).
agent-voice-tts	Piper TTS via ort + piper-phonemize (espeak-ng data embedded in the binary).
agent-voice-conversation	Optional LLM turn-taking (Ollama / Claude).
agent-voice-mcp	MCP server exposing listen, speak, transcribe, converse, list_audio_devices over stdio. Also a binary.
agent-voice-daemon	Long-running wrapper around the MCP server with signal handling.
agent-voice-tui	Interactive terminal UI for exercising the stack.

Build

cargo build --workspace
cargo test --workspace

Requirements: a C/C++ toolchain and CMake (for whisper.cpp), plus on Linux the ALSA headers (libasound2-dev on Debian/Ubuntu, alsa-lib-devel on Fedora). ONNX Runtime is downloaded automatically by ort at build time; the Silero VAD model and espeak-ng phonemizer data are embedded — neither needs installing.

Note: this workspace pins target-dir = "target" in .cargo/config.toml. The piper-phonemize build script locates its embedded espeak data by finding a directory literally named target, so a global CARGO_TARGET_DIR override would break the build.

Models

Two model files are loaded at runtime from <data dir>/agent-voice/models (~/.local/share on Linux, ~/Library/Application Support on macOS, %APPDATA% on Windows):

• Whisper: ggml-<size>.bin (default size base) from ggerganov/whisper.cpp.
• Piper voice: piper/<voice>.onnx and piper/<voice>.onnx.json from rhasspy/piper-voices.

scripts/fetch-models.sh downloads a working default set (on Windows run it from Git Bash; it needs curl). A repo-local models/piper/voice.onnx is also honored for development.

Smoke tests

# TTS: synthesize to a WAV
cargo run -p agent-voice-tts --example say_to_wav -- "Hello there" /tmp/hello.wav

# STT: transcribe it back
cargo run -p agent-voice-stt --example transcribe_wav -- /tmp/hello.wav

# Interactive TUI (l = listen, s = speak, c = converse, d = devices, q = quit)
cargo run -p agent-voice-tui

# MCP server over stdio
cargo run -p agent-voice-mcp

Embedding in an agent host

See docs/GILA_PLUGIN.md for the two integration paths (stdio MCP server with no host rebuild, or the agent-voice facade crate compiled into the host) and the gilamonster-agent wiring specifically.