Skip to content

GILA_PLUGIN.md

Latest commit

 

History

History
119 lines (91 loc) · 4.95 KB

GILA_PLUGIN.md

File metadata and controls

119 lines (91 loc) · 4.95 KB
Raw

agent-voice as a gilamonster-agent plugin

How gila grows a voice interface. Two paths, both supported by this repo today; they share the same stack and can coexist. Both couple only to stable seams — newt's MCP client config and a plain Rust library API — so they survive the planned gilamonster-agent rewrite on the new newt-agent airframe.

Path A — stdio MCP server (no gila rebuild)

gila already consumes any stdio MCP server through newt's MCP client ([[mcp_servers]] in newt config). Build the binary once and wire it in:

cargo build --release -p agent-voice-mcp

Global ~/.newt/config.toml, or a project .newt/config.toml with [merge] arrays = "append":

[[mcp_servers]]
name = "voice"
command = "/path/to/agent-voice/target/release/agent-voice-mcp"

The session then has listen, speak, transcribe, converse, and list_audio_devices as tools. The model can speak its answers and listen for the user's reply with no gila changes at all. agent-voice-daemon is the same server with signal handling for long-running/background use.

This is also the shape of a gilamonster-capabilities capability: those are pip-installed Python packages exposing gila-cap-<name>-mcp servers, and this is the identical contract with a Rust binary. If a gilacap-style registry entry is wanted later, a thin capability package can point its MCP command at agent-voice-mcp.

Trade-offs: subprocess + JSON hop per call; voice acts as tools the model calls, not as an ambient input surface. Good for "speak this summary", not for push-to-talk-driven prompting.

Path B — built in (the agent-voice facade crate)

gila is a private binary that consumes sibling repos as git dependencies (newt-tui, agent-bridle, …). Voice joins the same way, behind a feature so the lean airframe build stays lean:

# gilamonster-agent/Cargo.toml
[features]
voice = ["dep:agent-voice"]

[dependencies]
agent-voice = { git = "https://github.com/Gilamonster-Foundation/agent-voice", optional = true }

(For local dev, overlay the checkout via the git-ignored .cargo/config.toml patch, the same pattern used for the newt checkout.)

Inside gila, one VoiceStack lives for the life of the process:

#[cfg(feature = "voice")]
use agent_voice::{VoiceStack, VoiceSettings};

// At startup (settings deserialize from any TOML table, all fields optional):
let voice = VoiceStack::new(settings)?;   // loads Whisper/Piper/Silero once

// Anywhere a surface wants voice:
let heard: ListenResult = voice.listen().await;      // mic -> VAD endpoint -> transcript
let said: SayResult = voice.say("On it.").await;     // Piper -> speakers

VoiceSettings is serde-friendly, so it can hang off gila's existing config as a [voice] table with zero glue.

Where the voice interface mounts ("some locations")

The facade is surface-agnostic; these are the natural mount points, each a small, independent change:

  1. TUI push-to-talk — a keybinding in the chat prompt calls voice.listen() and inserts the transcript into the input buffer. This is the highest-value surface and needs nothing but the transcript.
  2. Spoken replies — an on/off toggle that pipes final assistant messages (stripped of code blocks) through voice.say().
  3. Hotseat / cockpit announcements — matrix events ("agent finished", "review ready") spoken via voice.say(); a herdr-style pane supervisor can announce which pane wants attention.
  4. A gila voice subcommand — one-shot gila voice say "..." / gila voice listen for scripts and testing.

Rewrite guidance

For the newt-agent-based rewrite, the clean shape is: newt-tui grows an optional VoiceInput/VoiceOutput hook trait pair (transcript-in, text-out), and gila's voice feature provides the VoiceStack-backed implementation. That keeps newt free of audio deps while letting every surface inherited from the airframe be voice-enabled. Until then, Path A gives voice today and Path B's facade API is the contract to build against.

Runtime requirements (both paths)

  • Model files in <data dir>/agent-voice/models — see the README's Models section (scripts/fetch-models.sh provisions defaults).
  • Microphone/speaker permissions: on macOS the host process (gila or the terminal running it) needs mic permission; the embedded path inherits gila's grant, the MCP path prompts for the spawned binary's parent.
  • Everything is local; no audio leaves the machine. The optional converse tool talks to Ollama/Claude only when configured.

Platform notes

Linux macOS Windows (x86_64 only)
Audio (cpal) ALSA (libasound2-dev to build) CoreAudio WASAPI
ONNX Runtime (ort) prebuilt download prebuilt download (incl. arm64) prebuilt download
Whisper (whisper.cpp) CMake + gcc/clang CMake + Xcode CLT (Accelerate) CMake + MSVC
Piper phonemizer espeak-ng data embedded at build same same (no arm64 prebuilt — Windows-on-ARM unsupported)