You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(realtime-transcription): add the WebSocket example (#57)
Adds a WebSocket example: realtime speech-to-text where audio streams up
and transcripts stream back on one socket, which HTTP and SSE cannot do.
Fills the last transport the README declared but never showed. A small
aiohttp server wraps faster-whisper; the orchestrator proxies the upgrade
through unchanged.
Dynamic, persistent, USD per hour. The model is pinned at large-v3-turbo,
so the example is GPU-only. Input is a 16 kHz mono WAV, or `-` to pipe a
microphone through ffmpeg.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|[`hello-world`](./hello-world)| The simplest app: one request, one response | dynamic | single-shot | HTTP (JSON) | fixed |
43
-
|[`tiles`](./tiles)| Capacity fan-out — one call per tile | dynamic | single-shot | HTTP (base64 PNG) | fixed |
44
-
|[`api-proxy`](./api-proxy)| Pass calls through to a hosted API — the operator holds the key, callers pay per call | static | single-shot | HTTP (JPEG bytes) | fixed |
45
-
|[`echo`](./echo)| Realtime video, transformed and echoed back | dynamic | persistent | trickle | hour |
|[`hello-world`](./hello-world)| The simplest app: one request, one response | dynamic | single-shot | HTTP (JSON) | fixed |
43
+
|[`tiles`](./tiles)| Capacity fan-out — one call per tile | dynamic | single-shot | HTTP (base64 PNG) | fixed |
44
+
|[`api-proxy`](./api-proxy)| Pass calls through to a hosted API — the operator holds the key, callers pay per call | static | single-shot | HTTP (JPEG bytes) | fixed |
45
+
|[`echo`](./echo)| Realtime video, transformed and echoed back | dynamic | persistent | trickle | hour |
|[`realtime-transcription`](./realtime-transcription)| Audio up, transcripts back, on one socket | dynamic | persistent | WebSocket | hour |
47
48
48
49
Start with `hello-world` (the smallest end-to-end path); the others each layer on one new idea. More will follow, including a full example that exercises every feature. Each is self-contained and runs **offchain** (free, no wallet); most also run **on-chain** (paid) — see each README.
49
50
@@ -53,7 +54,7 @@ This set stays **minimal and curated**: it covers each value of the axes above (
53
54
54
55
How the app attaches to the orchestrator:
55
56
56
-
-**Dynamic** — the app self-registers via the SDK (`register_runner`) and heartbeats; the orchestrator drops it when heartbeats stop. Best for apps that come and go. (`hello-world`, `echo`)
57
+
-**Dynamic** — the app self-registers via the SDK (`register_runner`) and heartbeats; the orchestrator drops it when heartbeats stop. Best for apps that come and go. (`hello-world`, `echo`, `realtime-transcription`)
57
58
-**Static** — the orchestrator is configured with the app's URL in a `runners.json` and health-polls it; the app needs no SDK. Best for fixed, long-running deployments. (`vllm`, `api-proxy`)
58
59
59
60
Both forms also take an optional **`metadata`** string: up to 1 KB of app-controlled UTF-8 for detail the protocol doesn't model, echoed in `/discovery` and never read by the orchestrator. Clients read it off the discovered runner, whose `raw` holds that runner's discovery entry: `cursor.candidates[0].raw["metadata"]` after `runner_selector`, `session.runner.raw["metadata"]` after `reserve_session`. Anything a caller **selects or pays differently for** belongs in the app id instead: discovery filters on `app` and `gpu`, never on metadata, which is why no example here uses it.
@@ -76,15 +77,15 @@ flowchart LR
76
77
77
78
Chosen _at_ registration (above); **defaults to `persistent`** — set on both `register_runner(...)` and in `runners.json`. The examples set it explicitly.
78
79
79
-
-**Persistent** — a held-open session the client reserves and releases, billed per second of wall-clock (or once, with fixed pricing). Best for realtime / streaming. (`echo`)
80
+
-**Persistent** — a held-open session the client reserves and releases, billed per second of wall-clock (or once, with fixed pricing). Best for realtime / streaming. (`echo`, `realtime-transcription`)
80
81
-**Single-shot** — one request in, one response out; the orchestrator reserves a session per call and releases it when the response returns, so the client manages no session at all. Best for batch / request-response. With metered pricing the call pays for as long as it runs, so the work need not be short. (`hello-world`, `tiles`, `api-proxy`, `vllm`)
81
82
82
83
## Calling your app
83
84
84
85
The client side depends on the runner's mode:
85
86
86
87
-**Single-shot** — **discover → call**: find the app via `runner_selector`, then one `call_runner`. The orchestrator reserves a session for the call and releases it when the response returns; on the paid path `call_runner` answers the 402 payment challenge inline. (`hello-world`, `tiles`, `api-proxy`, `vllm`)
87
-
-**Persistent** — **discover → reserve → call → release**: reserve a session (`reserve_session`), call it — `call_runner`, streamed frames, or a WebSocket, depending on transport — then release it (`stop_runner_session`), which settles payment on-chain. (`echo`)
88
+
-**Persistent** — **discover → reserve → call → release**: reserve a session (`reserve_session`), call it — `call_runner`, streamed frames, or a WebSocket, depending on transport — then release it (`stop_runner_session`), which settles payment on-chain. (`echo`, `realtime-transcription`)
88
89
89
90
Each example's `client.py` shows its exact calls — grep `# Livepeer:` to find them.
Realtime speech-to-text on the Livepeer network over a **WebSocket** — the client streams audio _up_ and gets transcripts streamed _back_ on one socket. This is the example where WebSockets are genuinely required: HTTP can't stream audio upstream, and SSE is one-directional (server→client). The app is a small aiohttp server wrapping `faster-whisper` that **self-registers** (dynamic) — the dynamic, WebSocket counterpart to the static HTTP vLLM example.
| Registration | dynamic (self-registers via the SDK) |
10
+
| Model |`large-v3-turbo` (faster-whisper, fixed) |
11
+
| Transport | WebSocket (`/transcribe`) |
12
+
| Port | 8989 |
13
+
14
+
**Requires an NVIDIA GPU.** The model is fixed at `large-v3-turbo`: it swaps large-v3's 32-layer decoder for 4, so it runs far below realtime on a 3090 while staying near large-v3 quality. The device is pinned with it (`cuda`/`float16`) rather than exposed as a flag: on CPU the model loads but falls behind a live stream, which is the one thing this example is about. Prerequisites (Docker, `uv`, the not-yet-released SDK) and the shared on-chain/payment setup are in the [repo README](../README.md).
15
+
16
+
## How it's wired
17
+
18
+
The app is **dynamically registered**: it self-registers with the orchestrator via `register_runner` ([runner.py](runner.py)) and exposes a `GET /transcribe` WebSocket, whose upgrade the orchestrator proxies straight through — your app speaks standard WS, nothing Livepeer-specific in the socket. The client calls it with `reserve_session` → `ws_connect` → `stop_runner_session` ([client.py](client.py)) — reserve a session, open a `wss://` socket to the session URL, stream audio up / transcripts back, release. Grep `# Livepeer:` in either file to see the exact calls.
19
+
20
+
On-chain, the reserved session is the billing unit: `reserve_session` pays at reserve and the meter runs while the socket is open — continuous connection = continuous billing, the right model for live audio.
21
+
22
+
**Staying realtime is the constraint,** and it is why the model is a constant rather than a setting. The app degrades by stretching out partials rather than dropping audio, so a model that cannot keep pace buys accuracy with unbounded lag instead of failing loudly. Serving a different model is a different app, with its own price and its own app id — which is why the id names the capability (`realtime-transcription`) and not the technique.
23
+
24
+
**Realtime design:** the receive loop only appends audio (never blocking on the model); a background worker transcribes the _current utterance_ (bounded to 15s) every ~0.5s, emits partials, and finalizes on trailing silence or max length — so cost stays bounded no matter how long the stream runs, instead of re-transcribing an ever-growing buffer. It uses the low-latency Whisper preset (`beam_size=1`, no cross-segment conditioning). For production-grade streaming you'd reach for a LocalAgreement approach (whisper_streaming / WhisperLive).
Cumulative, not incremental: each partial carries the whole utterance so far and may revise earlier words, so a client replaces rather than appends. That matches Deepgram and Vosk, and it is the honest shape for a decoder that re-runs over the buffer. Delta protocols (OpenAI's `transcript.text.delta`) only become correct once decoding is append-only, which is what the LocalAgreement approach below buys you.
33
+
34
+
## Audio
35
+
36
+
Input must be **16 kHz mono WAV**. Fetch 21s of NASA podcast speech, public domain under 17 U.S.C. 105:
Use a clip with a couple of sentences and a pause between them: the app finalizes on trailing silence, so that is what shows partials turning into finals more than once. The NASA clip is trimmed to three such sentences.
51
+
52
+
Or skip the file and talk into a microphone: pass `-` and pipe raw PCM in, which streams until you Ctrl-C.
uv run client.py --discovery https://localhost:8935/discovery sample.wav
67
+
docker compose down
68
+
```
69
+
70
+
The client reserves a session, opens a WebSocket through the orchestrator, streams the WAV in real-time-paced chunks, and prints partial transcripts as they arrive plus a final one per utterance.
71
+
72
+
## Run on-chain (paid)
73
+
74
+
Layer `compose.onchain.yml` to add a remote signer and run the orchestrator on-chain. Needs an Ethereum RPC, a funded signer wallet (deposit + reserve), and an orchestrator wallet — see [On-chain (paid) setup](../README.md#on-chain-paid-setup) in the repo README.
75
+
76
+
```sh
77
+
cp .env.example .env # fill in RPC, network, keystore paths, accounts, pricing
78
+
docker compose -f compose.yml -f compose.onchain.yml up -d --build
79
+
uv run client.py --discovery https://localhost:8935/discovery \
80
+
--signer http://localhost:7936 sample.wav
81
+
docker compose -f compose.yml -f compose.onchain.yml down
82
+
```
83
+
84
+
`reserve_session` pays for the session through the remote signer; the WebSocket then streams over it. Because the session is metered, keep clips short for the demo — a long-lived socket keeps billing for its duration.
85
+
86
+
## Run without Docker
87
+
88
+
Start an orchestrator built from go-livepeer `v0.9.0` or newer (see [Build from source](https://docs.livepeer.org/v1/orchestrators/guides/install-go-livepeer#build-from-source)), then the app and client directly (the app needs `faster-whisper` installed):
0 commit comments