Skip to content

remote-stt.md

Latest commit

 

History

History
131 lines (100 loc) · 5.32 KB

remote-stt.md

File metadata and controls

131 lines (100 loc) · 5.32 KB
Raw

Remote Transcription: Self-Hosted Server Setup

This fork adds a Remote transcription mode: instead of running Whisper on the machine you dictate on, Handy sends the recorded audio to a transcription server you run yourself, on your own network. The rest of Handy is unchanged: same shortcut, same VAD trimming, same paste.

Why you might want this:

  • Dictate on a laptop or thin client while a GPU box elsewhere does the inference
  • Share one GPU (and one downloaded model) across every machine in the house
  • Run a bigger, more accurate model than your desktop can hold
  • Keep VRAM free on the machine you are actually working on

Privacy note: in remote mode your audio does leave the computer, but only to the server you configure. Nothing goes to any third party. Local mode remains the default and is untouched.

Handy talks to any OpenAI-compatible endpoint (POST /v1/audio/transcriptions). The guide below uses speaches (GitHub), a self-hosted server built on faster-whisper, because it is easy to run and fast. Other compatible servers work too (see Other servers).

1. Run the server

You need Docker on the server machine. For GPU inference (recommended) you also need an NVIDIA GPU with the NVIDIA Container Toolkit installed.

Create a compose.yaml on the server:

services:
  speaches:
    container_name: speaches
    image: ghcr.io/speaches-ai/speaches:latest-cuda
    restart: unless-stopped
    ports:
      - "8000:8000"
    environment:
      # Unload the model after 10 min idle to free VRAM.
      # -1 = keep loaded forever, 0 = unload immediately after each request.
      - STT_MODEL_TTL=600
    volumes:
      - hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  hf-hub-cache:

No GPU? Use the ghcr.io/speaches-ai/speaches:latest-cpu image and delete the deploy: block. CPU transcription is slower but usable with the turbo model.

Start it:

docker compose up -d

2. Download a model on the server

Models do not auto-download on the first transcription request, so pull one explicitly (one-time; it persists in the hf-hub-cache volume):

curl -X POST http://localhost:8000/v1/models/deepdml/faster-whisper-large-v3-turbo-ct2

deepdml/faster-whisper-large-v3-turbo-ct2 is a good default: large-v3 accuracy class at turbo speed, about 1.6 GB download, roughly 3.7 GB VRAM loaded. Browse other CTranslate2 models on Hugging Face.

Verify the server is healthy and the model is installed:

curl http://localhost:8000/health
curl http://localhost:8000/v1/models

3. Point Handy at it

In Handy: Settings → Models, switch the selector at the top from Local Models to Remote API, then fill in the card:

Remote API settings

Field Value
Server URL http://<server-ip>:8000
Model deepdml/faster-whisper-large-v3-turbo-ct2 (must match what you downloaded)
Timeout 10 seconds is fine on a LAN; raise it for CPU servers or slow links
API key Leave empty unless your server requires auth (sent as a Bearer token if set)

Hit Test Connection. It checks the server's /health endpoint, confirms your model is in its model list, and reports round-trip latency. Green means dictate away.

The language selector works in remote mode, and words you add to Handy's dictionary are sent along as the Whisper prompt, so custom vocabulary still helps accuracy.

What to expect

Real numbers from the deployment this was built against (RTX 3080 Ti server, gigabit LAN, 11 seconds of speech): about 0.25 s of server-side inference when the model is warm, about 1 s end-to-end from shortcut release to pasted text. A cold request (model was unloaded by STT_MODEL_TTL) adds roughly 1.5 s for the model load.

Other servers

Anything that implements the OpenAI audio transcriptions API works, for example faster-whisper-server or LocalAI. Handy sends a multipart POST {url}/v1/audio/transcriptions with file (16 kHz mono WAV), model, response_format=json, plus language and prompt when set, and an Authorization: Bearer header if an API key is configured.

Troubleshooting

  • "could not connect": wrong URL or port, server not running, or a firewall between you. Test from the dictating machine: curl http://<server-ip>:8000/health.
  • "request timed out": server busy or cold-loading on a slow disk; raise the timeout.
  • Test Connection reaches the server but the model check fails: model ID typo, or the model was never downloaded on the server (step 2).
  • Dictation produces nothing and there is no error: check Handy's log file (Settings → Debug) for the underlying request error.
  • Security: the server speaks plain HTTP and, with Docker's default networking, published ports bypass simple ufw rules on Linux. Keep it on a trusted LAN (or a VPN such as Tailscale), or put a reverse proxy with TLS and auth in front and set the API key.