Skip to content

Commit ed2fb69

Browse files
ralyodioclaude
andauthored
feat: voice mode on iOS, the home-screen app and Firefox (0.12.0) (#323)
iOS leaves SpeechRecognition out of home-screen web apps (service-not-allowed, WebKit 225298), Safari's needs Siri and goes deaf after audio plays, and Firefox has none. Where the browser cannot turn speech into text, voice mode now records the turn and transcribes it on qrypt.chat. - voice-recorder.js: RecorderRecognition, a SpeechRecognition look-alike on getUserMedia + MediaRecorder (webm/opus or iOS mp4) with its own pause detection from the mic level. The noise floor is calibrated but capped, so talking the instant the mic opens still counts as speech. The mic is released after each turn and navigator.audioSession switched to playback so replies are loud on iPhone; Web Audio is unlocked in the first tap. - POST /api/voice/transcribe (signed in, 30/min and 1000/day per user, 10000/day site-wide): passes the clip to Workers AI whisper-large-v3-turbo with the chat's names as the prompt, drops Whisper's silence phantoms, stores nothing. Env CF_AI_ACCOUNT_ID / CF_AI_TOKEN. - Engine choice: iOS always records; elsewhere the browser recognizer, falling back to recording on service-not-allowed / network / language-not-supported. - Watchdogs for recognition and speech that never end (iOS), "Hearing you…" / "Transcribing…" in the bar, and a privacy line per engine. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 77a81da commit ed2fb69

14 files changed

Lines changed: 815 additions & 65 deletions

File tree

‎docs/VOICE.md‎

Lines changed: 35 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -24,19 +24,36 @@ The bar above the composer shows what it heard and what it is doing.
2424
After three silent turns it pauses; tap the mic to carry on. Changing chats
2525
ends voice mode.
2626

27+
## Where it works
28+
29+
| | Speech to text | Replies read by |
30+
| --- | --- | --- |
31+
| iPhone / iPad: Safari and the home-screen app | Recorded, transcribed by qrypt.chat | The device |
32+
| Chrome, Edge, Android | The browser (on the device when Chrome has the language pack) | The device |
33+
| Firefox, Safari with Siri off, Chromium without Google's speech service | Recorded, transcribed by qrypt.chat | The device |
34+
35+
iOS leaves speech recognition out of home-screen web apps, and in Safari it
36+
needs Siri and stops hearing after audio plays, so iOS always records the
37+
clip instead. Allow the microphone when asked; if you said no, turn it back on
38+
in Settings > Safari > Microphone (or the site's settings). The first tap
39+
unlocks audio, after which the conversation is hands free. Other browsers
40+
switch to recording by themselves if their recognizer refuses to run.
41+
2742
## Privacy
2843

29-
Both halves use the browser's own speech engines (Web Speech API); nothing new
30-
goes to the qrypt.chat server, and what you say is sent as an ordinary
31-
ML-KEM encrypted message.
44+
What you say is sent as an ordinary ML-KEM encrypted message. How the words
45+
become text depends on the browser, and the voice bar always says which:
3246

33-
- **Recognition** runs on the device when the browser supports it (Chrome's
34-
on-device language packs, which it downloads in the background the first
35-
time, and Safari). Otherwise the audio goes to the browser vendor's speech
36-
service. The bar says which one is in use.
37-
- **Reading aloud** uses the device's voices; on-device voices are preferred.
47+
- **On this device:** Chrome with its on-device language pack (downloaded in
48+
the background the first time).
49+
- **Browser speech service:** Chrome/Edge without the pack send the audio to
50+
the browser vendor.
51+
- **qrypt.chat transcription:** the clip from one turn is posted to
52+
`/api/voice/transcribe` (signed-in users only, rate limited), passed to
53+
Cloudflare Workers AI Whisper, and the text comes back. It is never stored
54+
or logged.
3855

39-
Firefox has no speech recognition, so the mic does not appear there.
56+
Reading aloud always uses the device's own voices.
4057

4158
## For agents
4259

@@ -47,5 +64,12 @@ Code blocks, markdown and links are cleaned up before speaking (links become
4764
"link to github.com"), and long replies stop after about 1,200 characters with
4865
"The rest is in the chat."
4966

50-
Code: `src/lib/chat/voice-mode.js` (pure helpers), `src/lib/hooks/useVoiceMode.js`
51-
(the listen, send, speak loop), `src/lib/components/chat/VoiceBar.jsx`.
67+
Code: `src/lib/chat/voice-mode.js` (pure helpers, engine choice),
68+
`src/lib/chat/voice-recorder.js` (record one turn, find the pause, upload),
69+
`src/lib/hooks/useVoiceMode.js` (the listen, send, speak loop),
70+
`src/lib/components/chat/VoiceBar.jsx`, `src/app/api/voice/transcribe/route.js`
71+
and `src/lib/server/transcribe.js`.
72+
73+
Server env: `CF_AI_ACCOUNT_ID`, `CF_AI_TOKEN` (a Workers AI token), optional
74+
`VOICE_STT_MODEL` (default `@cf/openai/whisper-large-v3-turbo`). Without them
75+
the route answers 503 and only browsers with their own recognizer can talk.

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "qryptchat-web",
33
"private": true,
4-
"version": "0.11.0",
4+
"version": "0.12.0",
55
"type": "module",
66
"bin": {
77
"qc": "./bin/qc.js",
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
import { NextResponse } from 'next/server';
2+
import { withAuth } from '@/lib/api/middleware/auth.js';
3+
import { RateLimiter, applyRateLimitForKey } from '@/lib/server/rate-limiter.js';
4+
import { MAX_AUDIO_BYTES, TranscribeError, cleanHint, isAudioType, transcribe, transcribeConfigured } from '@/lib/server/transcribe.js';
5+
6+
// Each call bills Workers AI, so cap it per person and across the site.
7+
const perMinute = new RateLimiter({ maxRequests: 30, windowMs: 60 * 1000 });
8+
const perDay = new RateLimiter({ maxRequests: 1000, windowMs: 24 * 60 * 60 * 1000 });
9+
const siteDaily = new RateLimiter({ maxRequests: 10000, windowMs: 24 * 60 * 60 * 1000 });
10+
11+
/** GET /api/voice/transcribe: whether server transcription is available. */
12+
export const GET = withAuth(async () => NextResponse.json({ available: transcribeConfigured() }));
13+
14+
/**
15+
* POST /api/voice/transcribe body: the recorded clip (audio/webm, audio/mp4, …)
16+
* Headers: X-Voice-Lang (BCP 47), X-Voice-Hint (URI-encoded, comma-separated names).
17+
* Returns { text } ('' when nothing was said). The audio is never stored.
18+
*/
19+
export const POST = withAuth(async ({ request, locals }) => {
20+
const key = locals.user.id;
21+
const limited =
22+
applyRateLimitForKey(perMinute, key) || applyRateLimitForKey(perDay, key) || applyRateLimitForKey(siteDaily, 'site');
23+
if (limited) return limited;
24+
25+
const type = request.headers.get('content-type');
26+
if (!isAudioType(type)) return NextResponse.json({ error: 'Send the recording as audio/*' }, { status: 415 });
27+
if (Number(request.headers.get('content-length') || 0) > MAX_AUDIO_BYTES) {
28+
return NextResponse.json({ error: 'Recording too long' }, { status: 413 });
29+
}
30+
31+
try {
32+
const audio = new Uint8Array(await request.arrayBuffer());
33+
const text = await transcribe(audio, {
34+
lang: request.headers.get('x-voice-lang') || '',
35+
hint: cleanHint(request.headers.get('x-voice-hint')),
36+
});
37+
return NextResponse.json({ text });
38+
} catch (error) {
39+
if (error instanceof TranscribeError) return NextResponse.json({ error: error.message }, { status: error.status });
40+
console.error('[voice/transcribe]', error?.message);
41+
return NextResponse.json({ error: 'Transcription failed' }, { status: 500 });
42+
}
43+
});

‎src/lib/chat/voice-mode.js‎

Lines changed: 41 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,20 @@
22
* Voice mode: talk to a chat (usually your AI agents) instead of typing.
33
*
44
* Turn by turn, hands free: listen → send what you said → read the replies
5-
* aloud → listen again. Both halves use the browser's own speech engines
6-
* (Web Speech API), so nothing new is sent to qrypt. Recognition runs on the
7-
* device when the browser can (Chrome's on-device packs, Safari), otherwise in
8-
* the browser vendor's speech service; the voice bar says which. Messages are
9-
* still ML-KEM encrypted like anything typed; speech never touches our server.
5+
* aloud → listen again. Replies are read by the device (speechSynthesis).
6+
* Speech becomes text in the browser where it can (Web Speech API, on the
7+
* device when Chrome has the language pack, otherwise the vendor's service).
8+
* Where it cannot (iOS, Firefox) the clip is transcribed by qrypt.chat through
9+
* Workers AI and not stored. The voice bar says which. Messages are ML-KEM
10+
* encrypted like anything typed.
1011
*
1112
* Everything here is pure or takes the window/engine as an argument so it can
1213
* be tested without a browser.
1314
*/
1415

1516
import { CALL_TYPE } from './calls.js';
1617
import { REACTION_TYPE } from './reactions.js';
18+
import { canRecord } from './voice-recorder.js';
1719

1820
/** Marks a message as dictated, so agents know to answer in speakable prose. */
1921
export const DICTATED = '🗣️';
@@ -25,9 +27,42 @@ export function recognitionCtor(win = globalThis) {
2527
return win?.SpeechRecognition || win?.webkitSpeechRecognition || null;
2628
}
2729

30+
/** iPhone, iPad (which reports itself as a Mac) or iPod. */
31+
export function isIOS(win = globalThis) {
32+
const nav = win?.navigator || {};
33+
return /iPad|iPhone|iPod/.test(nav.userAgent || '') || (nav.platform === 'MacIntel' && nav.maxTouchPoints > 1);
34+
}
35+
36+
/**
37+
* How to turn speech into text here:
38+
* 'native' the browser's SpeechRecognition
39+
* 'server' record the clip and transcribe it on qrypt.chat (voice-recorder.js)
40+
* iOS always records: home-screen apps have no SpeechRecognition, Safari's
41+
* needs Siri on, and it goes deaf after audio plays (WebKit 321436).
42+
*/
43+
export function pickEngine(win = globalThis) {
44+
if (recognitionCtor(win) && !isIOS(win)) return 'native';
45+
if (canRecord(win)) return 'server';
46+
return null;
47+
}
48+
49+
/** Native recognition errors that mean "this engine cannot work here", not "try again". */
50+
export const NATIVE_UNUSABLE = ['service-not-allowed', 'network', 'language-not-supported'];
51+
2852
/** What this browser can do: listen (speech to text) and speak (text to speech). */
2953
export function voiceSupport(win = globalThis) {
30-
return { listen: !!recognitionCtor(win), speak: !!win?.speechSynthesis && typeof win?.SpeechSynthesisUtterance === 'function' };
54+
return { listen: !!pickEngine(win), speak: !!win?.speechSynthesis && typeof win?.SpeechSynthesisUtterance === 'function' };
55+
}
56+
57+
/** Names worth spelling right when transcribing: the people and agents in the chat. */
58+
export function nameHints(conversation, selfId) {
59+
const names = new Set(['qrypt.chat']);
60+
for (const p of conversation?.participants || []) {
61+
if ((p.user_id ?? p.id) === selfId) continue;
62+
if (p.user?.username) names.add(p.user.username);
63+
if (p.user?.display_name) names.add(p.user.display_name);
64+
}
65+
return [...names];
3166
}
3267

3368
/**

‎src/lib/chat/voice-mode.test.js‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,11 @@ import {
33
DICTATED,
44
agentsIn,
55
dictatedMessage,
6+
isIOS,
67
isStopPhrase,
8+
nameHints,
79
onDeviceRecognition,
10+
pickEngine,
811
pickVoice,
912
recognitionCtor,
1013
shouldSpeak,
@@ -115,3 +118,24 @@ describe('pickVoice', () => {
115118
expect(pickVoice(voices, 'ja-JP', 'x')).toBeNull();
116119
});
117120
});
121+
122+
describe('engine choice', () => {
123+
const fn = function () {};
124+
const desktop = { navigator: { userAgent: 'Chrome', mediaDevices: { getUserMedia: fn } }, MediaRecorder: fn, AudioContext: fn };
125+
const iphone = { ...desktop, navigator: { userAgent: 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X)', mediaDevices: { getUserMedia: fn } } };
126+
const ipad = { ...desktop, navigator: { userAgent: 'Macintosh', platform: 'MacIntel', maxTouchPoints: 5, mediaDevices: { getUserMedia: fn } } };
127+
128+
it('uses the browser recognizer except on iOS, and records where there is none', () => {
129+
expect(pickEngine({ ...desktop, webkitSpeechRecognition: fn })).toBe('native');
130+
expect(pickEngine({ ...iphone, webkitSpeechRecognition: fn })).toBe('server');
131+
expect(pickEngine({ ...ipad, webkitSpeechRecognition: fn })).toBe('server');
132+
expect(pickEngine(desktop)).toBe('server');
133+
expect(pickEngine({ navigator: {} })).toBeNull();
134+
expect(isIOS(iphone)).toBe(true);
135+
expect(isIOS({ navigator: { userAgent: 'Macintosh', platform: 'MacIntel', maxTouchPoints: 0 } })).toBe(false);
136+
});
137+
138+
it('hints the names in the chat, not your own', () => {
139+
expect(nameHints(agents, 'a')).toEqual(['qrypt.chat', 'riotcoder', 'cha0spunk']);
140+
});
141+
});

0 commit comments

Comments
 (0)