⚡ Nova — AI Voice Agent

A full-stack voice assistant with a real-time Speech → Intelligence → Speech pipeline.

🎤 Browser Mic → /transcribe (Whisper) → /chat (GPT-4o) → /speak (ElevenLabs) → 🔊 Browser Speaker

Architecture

Layer	Technology	Purpose
STT	OpenAI Whisper (whisper-1)	Convert speech to text
LLM	OpenAI GPT-4o	Conversational reasoning
TTS	ElevenLabs (eleven_multilingual_v2)	Natural voice synthesis
Backend	Python FastAPI	API orchestration & streaming
Frontend	Single HTML file (vanilla JS)	Push-to-talk UI

---

Prerequisites

• Python 3.9+
• OpenAI API key — Get one here
• ElevenLabs API key — Get one here
• A modern browser with microphone access (Chrome, Edge, Firefox)

---

Setup

1. Install dependencies

pip install -r requirements.txt

2. Configure API keys

Copy the example env file and fill in your keys:

cp .env.example .env

Then edit .env:

OPENAI_API_KEY=sk-your-key-here
ELEVENLABS_API_KEY=xi_your-key-here
ELEVENLABS_VOICE_ID=21m00Tcm4TlvDq8ikWAM    # Optional: defaults to "Rachel"

3. Start the server

uvicorn main:app --reload

4. Open in your browser

Navigate to http://localhost:8000

---

Usage

1. Hold the microphone button and speak your message
2. Release the button — the pipeline runs automatically:
   • 🟡 Transcribing — your speech is converted to text via Whisper
   • 🔵 Thinking — GPT-4o generates a response
   • 🟢 Speaking — ElevenLabs reads the response aloud
3. The conversation history is maintained (last 10 turns)
4. Use the voice dropdown to switch ElevenLabs voices
5. Toggle 🔇 mute to skip audio playback
6. Press 🗑️ clear to reset the conversation

---

API Endpoints

Method	Path	Description
POST	/transcribe	Audio blob → transcript text (Whisper)
POST	/chat	Transcript + history → LLM reply
POST	/speak	Text → streamed audio/mpeg (ElevenLabs)
GET	/voices	List available ElevenLabs voices
GET	/	Serve the frontend

---

Customization

• Change Nova's personality: Edit the SYSTEM_PROMPT in main.py
• Switch LLM model: Change "gpt-4o" in the /chat endpoint
• Adjust TTS quality: Modify stability and similarity_boost in /speak
• Token limit: Adjust max_tokens in the /chat endpoint (default: 512)

---

Project Structure

voice-agent/
├── main.py             # FastAPI backend (< 200 lines)
├── index.html          # Single-file frontend (HTML + CSS + JS)
├── requirements.txt    # Python dependencies
├── .env.example        # API key template
├── .env                # Your actual API keys (git-ignored)
└── README.md           # This file