feat(realtime-transcription): add the WebSocket example

[rickstaa]

README.md declared four transports but shipped examples for three. The WebSocket row pointed at an external repo (external: scope), leaving the one gap the curation rule at README.md:50 recognises: registration, mode, and pricing were already covered on every value.

This fills it with realtime speech-to-text, the case where the socket is required rather than convenient. HTTP cannot stream audio upstream and SSE only runs server to client, so the transcript loop needs both directions at once. The orchestrator proxies the upgrade straight through: nothing inside the socket is Livepeer specific.

Axis coordinates: dynamic / persistent / WebSocket / hour. A held-open socket and a metered session are the same idea, so the session is the billing unit and the meter runs for as long as the client holds it.

Naming and the fixed model

The app id is a capability contract, so it names what a caller is shopping for: livepeer-example/realtime-transcription, with the directory matching. streaming-asr named the technique instead, and leaned on jargon.

The model is pinned at large-v3-turbo, not configurable. That is deliberate. An env knob would let one operator serve base.en and another large-v3 under the same app id at different prices, and discovery could not express why — the caller cannot tell quality from gouging. Since app is the only key runner_selector can filter on and the only thing carrying a price, a variant callers select on belongs in the id, not in a setting. Serving a different model is a different app.

large-v3-turbo swaps large-v3's 32-layer decoder for 4, so it runs far below realtime on a 3090 while staying near large-v3 quality. This makes the example GPU-only, joining vllm and streamdiffusion. That is a real cost — it was the only media example runnable on a laptop — but the alternative was a model that cannot keep pace, and this app degrades by stretching partials rather than dropping audio, so a slow model buys accuracy with unbounded lag instead of failing loudly.

CTranslate2 picks up cuBLAS and cuDNN from the nvidia pip wheels, so no CUDA base image is needed; the driver arrives through the compose device reservation.

Verified end to end

Live stack on an RTX 3090, offchain.

Pacing: 46 s of wall clock for a 45 s clip, so it holds realtime with headroom (the extra second is session setup).

Quality, against the same clip, versus the base.en build this replaces:

base.en	large-v3-turbo
"You have to go to perceive a terminal count"	"You have a go to proceed with terminal count"
"to make the stay possible"	"to make this day possible"

Multi-utterance output: partials stream in and each utterance resolves to its own [FINAL], covering the whole file. _recv previously broke on the first final, so a clip with several sentences printed one line and went quiet while the rest streamed.

Also updates the root README (transport bullet, table row, registration/mode/calling lists) and adds the folder to the image build matrix and its paths filter.

Since review

Consistency with the merged examples. Port is 8989 like every other self-hosted app (was 5005); ENTRYPOINT instead of CMD, which is why compose had to repeat python runner.py; compose and Dockerfile headers cut to the echo template; pyproject flattened, since the runner extra was dead config that also broke the README's uv run runner.py; the client takes its input positionally and holds the session with async with, like echo. --device and --compute-type are gone: their cpu/int8 defaults contradicted the GPU-only docs, and per the section above they were never really settings.

Three fixes. eos now cancels the background worker before its closing transcribe, so a partial can no longer land after the [FINAL]. _rms no longer raises when a chunk ends mid-sample. The client only builds an SSL context when the URL is wss://.

Timestamps on the wire. Every message carries start/end seconds into the stream, matching what Deepgram, AssemblyAI and OpenAI segments all carry; without them a client cannot build subtitles or seek. The format stays cumulative rather than delta on purpose: this decoder re-runs over the buffer and revises earlier words, so append-only deltas (OpenAI's transcript.text.delta) would be incorrect until a LocalAgreement pass makes the output append-only. The README now states that, so the choice reads as forced rather than stylistic.

Microphone input. - reads raw PCM from stdin, reusing echo's convention, so a live mic works through ffmpeg with no new dependency.

sample.wav is fetched, not committed — a public domain NASA clip (17 U.S.C. 105) trimmed to three sentences with pauses, which is what makes partials resolve to finals more than once. Follows echo's generate-don't-commit pattern.

Verified offchain and on-chain against a live orchestrator on the 3090, including a real microphone capture. On-chain the session prepays 10 s and tops up while the socket is held, and funding stops on release.

[Copilot]

Pull request overview

Adds a new WebSocket-based streaming ASR example (streaming-asr) to complete the transport matrix in the repo and updates the root docs/CI image workflow to include it. The example demonstrates a bidirectional, persistent session where audio streams to the runner and transcripts stream back over the same socket.

Changes:

• Introduces streaming-asr: aiohttp WebSocket runner using faster-whisper + a client that reserves a session and connects to /transcribe.
• Adds Docker + compose (offchain + on-chain overlay) and project metadata for the new example.
• Updates root README.md and .github/workflows/images.yml to reference/build the new example.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
streaming-asr/runner.py	New aiohttp WebSocket runner that self-registers and streams transcripts back.
streaming-asr/client.py	New client that reserves a session, connects via WS, streams PCM, prints partial/final.
streaming-asr/README.md	Documentation for protocol, offchain/on-chain runs, and non-Docker flow.
streaming-asr/pyproject.toml	Defines Python package deps with a runner extra for ASR stack.
streaming-asr/Dockerfile	Builds CPU-default runner image and installs required deps from Git.
streaming-asr/compose.yml	Offchain compose setup extending the shared orchestrator service.
streaming-asr/compose.onchain.yml	On-chain overlay adding signer + priced runner registration.
streaming-asr/.env.example	Example env vars for model selection and on-chain config.
streaming-asr/.gitignore	Ignores local media/demo assets (wav/mp4/srt/media/).
README.md	Adds the streaming-asr row and updates WebSocket transport references.
.github/workflows/images.yml	Adds streaming-asr to the image build matrix and path filters.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (4)

streaming-asr/runner.py:137

• After receiving "eos", the handler sends a final message but never explicitly closes the WebSocket. The client’s receive loop waits for the server to close, so this can hang and it also contradicts the README claim that the server closes after eos. Consider also skipping the final message when transcription is empty.

            elif msg.type == web.WSMsgType.TEXT and msg.data.strip() == "eos":
                text = await asyncio.to_thread(_transcribe, bytes(seg))
                await ws.send_json({"text": text, "final": True})
                break

streaming-asr/client.py:58

• TLS verification is always disabled when connecting to the orchestrator WebSocket. Even for examples, it’s useful to have a safe path for non-local orchestrators; add a flag to opt into certificate verification so users don’t have to edit code.

    parser.add_argument(
        "--signer", default="", help="Remote signer base URL (on-chain/paid path)."
    )
    return parser.parse_args()

streaming-asr/runner.py:126

• The background transcription worker can raise (e.g., ws.send_json failing after the socket closes/reset), which will terminate the task and may produce an unhandled task exception while leaving the connection open. Wrap the worker loop with exception handling so it exits cleanly and logs the failure (while still propagating cancellation).

    async def _worker() -> None:
        nonlocal seg, spoke
        while True:
            await asyncio.sleep(STEP_SEC)
            n = len(seg)

streaming-asr/client.py:122

• The client always constructs an SSL context and passes it to ws_connect(), even when the resolved URL could be ws:// (not wss://). aiohttp may reject SSL configuration for non-TLS WebSockets; also, TLS verification should depend on the new --verify-tls flag.

        ctx = ssl.create_default_context()  # orchestrator serves a self-signed cert
        ctx.check_hostname = False
        ctx.verify_mode = ssl.CERT_NONE

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (6)

realtime-transcription/runner.py:137

• On eos, the handler runs a final transcription while the background worker may still be mid-iteration. That can result in partials being sent after the final message, and can also duplicate work. Cancel/await the worker before doing the final eos transcription/send.

            elif msg.type == web.WSMsgType.TEXT and msg.data.strip() == "eos":
                text = await asyncio.to_thread(_transcribe, bytes(seg))
                await ws.send_json({"text": text, "final": True})
                break

realtime-transcription/runner.py:133

• WebSocket binary frames are assumed to be int16 PCM, but _rms()/np.frombuffer(..., int16) will raise ValueError if the frame length is not a multiple of 2 bytes. A malformed client frame would crash the handler instead of cleanly closing the socket.

        async for msg in ws:
            if msg.type == web.WSMsgType.BINARY:
                seg.extend(msg.data)
                if _rms(msg.data) >= SILENCE_RMS:
                    spoke = True

realtime-transcription/runner.py:123

• spoke can be set to True by the receive loop while the worker is awaiting to_thread(_transcribe, ...). If the worker then finalizes and sets spoke = False, it can erase the fact that speech bytes were already appended beyond n, which can prevent the next utterance from finalizing on trailing silence.

                del seg[
                    :n
                ]  # drop finalized audio; keep anything appended during inference
                spoke = False

realtime-transcription/client.py:112

• TLS verification is disabled unconditionally (CERT_NONE) and the SSL context is always passed to ws_connect, even if the URL is ws:// (non-TLS). This can both break plain-WS connections and makes it easy to accidentally disable cert validation against non-local endpoints.

        ctx = ssl.create_default_context()  # orchestrator serves a self-signed cert
        ctx.check_hostname = False
        ctx.verify_mode = ssl.CERT_NONE
        async with aiohttp.ClientSession() as cs:
            async with cs.ws_connect(

realtime-transcription/README.md:75

• The README states this example requires an NVIDIA GPU, but the "Run without Docker" command starts the runner without --device cuda / --compute-type float16, so it will default to CPU (--device defaults to cpu in runner.py) and contradict the stated requirement.

```sh
./livepeer -orchestrator -useLiveRunners -serviceAddr localhost:8935 -orchSecret abcdef -v 6
uv run runner.py --orchestrator https://localhost:8935 --orchSecret abcdef
uv run client.py --file sample.wav

realtime-transcription/README.md:14

• This paragraph says "Requires an NVIDIA GPU" but immediately notes that the model loads on CPU (just falls behind). To avoid a direct contradiction, clarify that the GPU requirement is specifically for staying realtime.

**Requires an NVIDIA GPU.** The model is fixed at `large-v3-turbo`: it swaps large-v3's 32-layer decoder for 4, so it runs far below realtime on a 3090 while staying near large-v3 quality. On CPU it loads but falls behind a live stream, which is the one thing this example is about. Prerequisites (Docker, `uv`, the not-yet-released SDK) and the shared on-chain/payment setup are in the [repo README](../README.md).

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (3)

realtime-transcription/pyproject.toml:19

• As written, uv run runner.py (documented in this README and consistent with other examples) will fail because the runner imports numpy at import-time and requires faster-whisper at startup, but both are only declared in an optional extra. Other examples in this repo put runner+client deps in the base dependencies, so the documented uv run commands work without extra flags.

# Base deps are what the *client* needs (host `uv run client.py`). The runner's
# ASR stack (faster-whisper) is the `runner` extra, installed in the Docker image.
dependencies = [
    "aiohttp",          # client WebSocket
    "livepeer-gateway",

realtime-transcription/runner.py:125

• The background worker can raise (e.g., RuntimeError: WebSocket is closed / connection reset) when sending JSON after the client disconnects. That will silently stop transcription mid-stream and leave a task exception until cleanup. Handle send failures and exit the worker cleanly.

            if finalize:
                if text:
                    await ws.send_json({"text": text, "final": True})
                del seg[
                    :n

realtime-transcription/runner.py:164

• CLI defaults currently select --device=cpu / --compute-type=int8, but this example is documented as GPU-required and the pinned model is unlikely to keep up on CPU. Defaulting to CUDA/float16 makes the "just run it" path match the README, while still allowing CPU via flags.

    parser.add_argument(
        "--device",
        default="cpu",
        help="cpu (default, runs anywhere) or cuda (low latency).",
    )
    parser.add_argument(
        "--compute-type", default="int8", help="int8 (cpu) or float16 (cuda)."
    )