🎙️ ESKADENIA STT

Local Arabic ⇄ English Speech-to-Text

A fully local, GPU-accelerated speech recognition system that transcribes Arabic and English — including audio that switches between the two — without sending a single byte to the cloud.

Developed for the ESKADENIA Software Student Challenge — AI Speech-to-Text Solution.

---

📌 Overview

ESKADENIA STT is an offline speech-to-text platform purpose-built for bilingual Arabic/English speech. It runs entirely on a single consumer GPU (developed and validated on an NVIDIA RTX 4060 8 GB), which means:

• 🔒 Privacy by design — audio never leaves the machine. No external API, no internet dependency at inference time.
• 🌍 Arabic and English — a single multilingual engine handles both languages, and correctly routes each utterance to the right one.
• 🔀 Code-switching aware — it detects and tags audio that alternates between Arabic and English within the same recording.
• ⚡ Real-time — live microphone streaming with sub-real-time latency, plus batch file transcription.

The system is delivered as a web application: a FastAPI inference backend and a modern React frontend, launched together with a single double-click.

---

✨ Key Features

Feature	Description
📂 File Transcription	Upload any audio file and get a full transcript with per-segment timestamps, detected language, and confidence.
🔀 Mixed-Language Mode	Splits the audio on silence, identifies the language of each segment independently, and tags the output [AR] / [EN].
🔴 Live Streaming	Real-time transcription straight from the microphone over a WebSocket, with partial and finalized segments.
📊 Built-in Evaluation	An in-app benchmark screen computes WER / CER against reference transcripts.
🎚️ Optional Denoising	DeepFilterNet noise suppression for low-quality recordings.
🛡️ AR/EN Hard Lock	The engine is constrained to only ever output Arabic or English — hallucinated third languages are structurally impossible.

---

🏗️ Architecture

flowchart LR
    subgraph Client["🖥️ Frontend — React + Vite (port 3000)"]
        UI["Home · Upload · Live · Evaluate"]
    end

    subgraph Server["⚙️ Backend — FastAPI (port 8000)"]
        API["REST /transcribe · WS /stream · /health"]
        PIPE["Pipeline: load → denoise → VAD → LID → route → transcribe"]
    end

    subgraph Models["🧠 AI Models (local, GPU)"]
        LID["ECAPA VoxLingua107<br/>(fast language ID)"]
        ASR["Whisper large-v3<br/>(faster-whisper · int8_float16)"]
    end

    UI -- "HTTP / WebSocket" --> API
    API --> PIPE
    PIPE --> LID
    PIPE --> ASR

Loading

How transcription works

                     ┌─────────────┐
   audio ──▶ load ──▶│  denoise?   │──▶ trim silence ──▶ ┌──────────────────┐
                     └─────────────┘                     │  Language ID     │
                                                         │  (2-stage)       │
                                                         └────────┬─────────┘
                                                                  │
                             ┌────────────────────────────────────┤
                             ▼                                    ▼
                      confident ar/en                     uncertain
                             │                                    │
                             ▼                                    ▼
                     force-decode in                    auto-decode, then
                     that language                      verify guess ∈ {ar,en}
                             └──────────────┬─────────────────────┘
                                            ▼
                                     Whisper large-v3
                                            ▼
                              transcript + timestamps + RTF

Two-stage language identification. A lightweight ECAPA model gives a fast first guess; when it isn't confident enough (short or ambiguous audio), Whisper's own detector arbitrates. If even that lands outside {ar, en}, the system re-decodes forced to the closer of the two — this is what makes the Arabic/English-only guarantee hold.

---

🖥️ Screenshots

Dashboard	Studio (file upload)

Live	Evaluate

---

🧰 Tech Stack

Layer	Technologies
Frontend	React 19, TypeScript, Vite 6, TailwindCSS 4, Motion
Backend	Python 3.11, FastAPI, Uvicorn, WebSockets
Speech Engine	faster-whisper (CTranslate2) · Whisper large-v3 · int8_float16
Language ID	SpeechBrain ECAPA-TDNN (VoxLingua107)
Audio	Silero VAD, DeepFilterNet, SoundFile, SciPy
Compute	PyTorch 2.5.1 + CUDA 12.1 (NVIDIA RTX 4060 8 GB)

---

📁 Project Structure

Ahmad Arafat GP2 Project/
├── Start ESKADENIA SST.bat        # one-click launcher (backend + frontend + browser)
│
├── backend/                       # FastAPI inference server
│   ├── main.py                    # entry point (uvicorn on :8000)
│   ├── config.yaml                # models, thresholds, AR/EN constraint
│   ├── settings.py                # typed config loader
│   ├── api/                       # server.py (routes) + schemas.py (wire format)
│   ├── preprocessing/             # audio_loader, vad, denoise
│   ├── language_id/               # lightweight (ECAPA) + whisper_fallback + decision
│   ├── transcription/             # router + transcriber (faster-whisper)
│   ├── file_pipeline.py           # whole-file & mixed-language pipelines
│   ├── streaming/live_pipeline.py # real-time streaming + accent arbitration
│   └── tests/                     # runnable checks + benchmark_accuracy.py
│
└── frontend/                      # React + Vite web UI
    └── src/
        ├── App.tsx                # screen router
        ├── api.ts                 # REST + WebSocket client
        └── components/            # Home · Upload · Live · Evaluate screens

---

🚀 Getting Started

Prerequisites

• Windows with an NVIDIA GPU (CUDA 12.1 capable) — CPU also works, slower
• Python 3.11
• Node.js 18+

1 · Backend setup

# create an isolated virtual environment (kept OUTSIDE the project / OneDrive)
python -m venv C:\dev\gp2-venv
C:\dev\gp2-venv\Scripts\activate

# install pinned dependencies (CUDA build of PyTorch is resolved automatically)
pip install -r backend/requirements.txt

⚠️ The versions in requirements.txt are pinned deliberately (torch + CUDA, speechbrain, numpy<2). Upgrading casually has broken CUDA and the ECAPA import in the past.

2 · Frontend setup

cd frontend
npm install

3 · Run — the easy way

Double-click Start ESKADENIA SST.bat in the project root. It launches the backend, the frontend, and opens http://localhost:3000 for you.

The backend needs ~1 minute to load the models on first start. The “Engine” badge in the UI turns green when it's ready.

3 · Run — manually

# terminal 1 — backend
cd backend && python main.py            # → http://localhost:8000

# terminal 2 — frontend
cd frontend && npm run dev              # → http://localhost:3000

---

🔁 Reproducing this from a fresh clone

backend/whisper-ct2-finetuned/model.bin is not checked into git (.gitignore: *.bin, ~1.5 GB). A fresh git clone will have every file except the model weights. To make the clone runnable:

1. Get the fine-tuned weights. Obtain whisper-ct2-finetuned.zip (distributed separately — see reproducibility-package/) and extract it into backend/whisper-ct2-finetuned/ so you end up with:

   backend/whisper-ct2-finetuned/
   ├── config.json
   ├── model.bin
   ├── preprocessor_config.json
   ├── tokenizer.json
   └── vocabulary.json
   

2. Verify integrity against reproducibility-package/CHECKSUMS.txt:

   Get-FileHash backend\whisper-ct2-finetuned\model.bin -Algorithm SHA256

3. Install pinned dependencies exactly as in Getting Started above (pip install -r backend/requirements.txt, npm install).
4. Launch via Start ESKADENIA SST.bat (Windows) or the manual two-terminal steps above. /health returning {"status": "ok", ...} confirms the model loaded correctly.

For the full technical review package — exact base-model/dataset references, LoRA hyperparameters, the training script, dependency-version caveats, the deterministic held-out evaluation split, and checksums — see reproducibility-package/REPRODUCIBILITY.md. The 200-clip held-out evaluation split used for the WER numbers below can be regenerated independently with reproducibility-package/generate_eval_manifest.py (fixed seed, fully deterministic).

Known setup issues on a fresh Windows machine

Hit and fixed all three while validating this reproducibility package on a freshly formatted/reinstalled Windows machine — worth checking first if a clone "should work" but doesn't:

• ssl.SSLCertVerificationError on first backend start. The Silero VAD load (torch.hub.load(...)) and other first-run model downloads use raw urllib/ssl, which relies on the OS certificate trust store — incomplete on a fresh Windows install (pip install works fine regardless, since pip vendors its own certifi bundle). Fix: point SSL_CERT_FILE and REQUESTS_CA_BUNDLE at the venv's certifi bundle —

  setx SSL_CERT_FILE "C:\dev\gp2-venv\Lib\site-packages\certifi\cacert.pem"
  setx REQUESTS_CA_BUNDLE "C:\dev\gp2-venv\Lib\site-packages\certifi\cacert.pem"

  (open a new terminal afterward so the variables take effect).
• FileNotFoundError: [WinError 2] from subprocess.run inside audio_loader.py. ffmpeg is a system binary, not a pip package, so pip install -r requirements.txt never installs it. Fix: winget install --id Gyan.FFmpeg -e (adds it to PATH automatically; restart any already-running backend process afterward so it picks up the new PATH).
• POST /transcribe feels ~2-3x slower than the documented RTF. /health returning 200 doesn't mean uploads are fast — Windows Defender's real-time protection scans every freshly written file on first access, and each upload is written to disk before processing. Confirmed by comparing an existing file (fast, matches documented RTF) against a freshly-written copy of the exact same bytes (~0.6-0.9s slower, regardless of which folder it landed in). Fix: uploads are written to a project-local backend/tmp_uploads/ folder (not the OS temp dir) specifically so a narrow exclusion can be scoped to just that folder, rather than excluding all of %TEMP% (which would weaken AV coverage for every other app) or disabling Defender outright:

  # run in an elevated (Administrator) PowerShell
  Add-MpPreference -ExclusionPath "<repo>\backend\tmp_uploads"

---

🔌 API Reference

The FastAPI server exposes three endpoints (interactive docs at http://localhost:8000/docs):

Method	Endpoint	Description
GET	/health	Engine status, device, compute type, loaded models.
POST	/transcribe	Multipart file upload. Optional form fields: language (ar/en, omit to auto-detect), mixed (per-segment language tagging), denoise.
WS	/stream	Send 16 kHz mono float32 PCM chunks; receive JSON transcription segments in real time.

Example — file transcription:

curl -X POST http://localhost:8000/transcribe \
     -F "file=@meeting.wav" \
     -F "mixed=true"

Response (abridged):

{
  "text": "[AR] مرحبا كيف حالك [EN] let's start the demo",
  "detected_language": "mixed",
  "duration": 6.42,
  "rtf": 0.31,
  "model_used": "large-v3",
  "segments": [
    { "start": 0.0, "end": 2.1, "language": "ar", "lid_confidence": 0.94, "text": "مرحبا كيف حالك" },
    { "start": 2.4, "end": 6.4, "language": "en", "lid_confidence": 0.88, "text": "let's start the demo" }
  ]
}

rtf = Real-Time Factor (processing time ÷ audio duration). Values below 1.0 mean faster than real time.

---

⚙️ Configuration

All tunables live in backend/config.yaml — no code changes required:

models:
  whisper: large-v3
  language_id: speechbrain/lang-id-voxlingua107-ecapa

api:
  max_upload_mb: 10                    # hard cap on POST /transcribe uploads (Studio only, not /stream)
  max_concurrent_transcriptions: 1     # one GPU, one model instance — extra requests queue

language_id:
  confidence_threshold: 0.7      # trust ECAPA alone above this
  uncertain_threshold: 0.55      # below this → safe auto-decode
  languages: [ar, en]            # HARD CONSTRAINT — only these two, ever

---

🧪 Testing & Benchmarking

Every component ships with a self-contained, dependency-free check — each script prints PASS / FAIL, no test framework required:

cd backend
python tests/test_environment.py     # CUDA / torch / model availability
python tests/test_language_id.py      # ECAPA + Whisper LID accuracy
python tests/test_transcription.py    # ASR sanity
python tests/test_api.py              # HTTP + WebSocket round-trip

Accuracy benchmark — computes WER / CER and LID accuracy over a labelled sample set:

python tests/benchmark_accuracy.py

Real recordings and the developer's own voice dumps live in backend/tests/audio_samples/ and backend/debug_dumps/, kept out of any training set so evaluation stays honest.

---

💡 Engineering Highlights

A few of the harder problems solved along the way:

• 🎯 Arabic-accented English. Strongly accented English fools every pre-transcription language detector — even Whisper labels it "Arabic" with 0.9+ confidence. The live pipeline solves this with accent arbitration: it runs two cheap decode probes (one per language) and picks the winner by decode quality rather than by any detector. Measured on the developer's hand-labeled recordings (identical clip set for both), this lifted live language-ID accuracy on the developer's accented speech from 13% to 75%.

  Scope note (multi-speaker testing): accent arbitration currently runs only in the Live pipeline. Testing file uploads (Studio) with other speakers' accented English showed the same mislabeling — a decode-quality-based fix was prototyped for Studio too (46.7% → 67.2% language-tag accuracy on 122 labelled real recordings) but costs 2 extra decode passes per file, which roughly doubles Studio latency on short clips. Given file transcription doesn't need live's sub-second responsiveness but the extra wait was still judged not worth it for the accuracy gained, the fix was kept live-only; Studio still forces whatever the two-stage LID guesses.

• 🚫 No hallucinated languages. Off-the-shelf Whisper auto-detection occasionally "detects" a random language (e.g. Norwegian) on unclear audio and produces a fake translation. The AR/EN hard lock verifies every detection against {ar, en} and re-decodes forced to the nearest of the two.
• ⚡ Warm models. All models are pre-loaded on startup so no user request — or live stream — ever pays the cold-start cost.

---

🗺️ Roadmap

• Within-utterance code-switching fine-tune — a LoRA fine-tune of Whisper large-v3 on the MohamedRashad/arabic-english-code-switching dataset (trained on Kaggle, deployed locally as a CTranslate2 model) to handle Arabic and English mixed inside a single sentence. See results below.
• Speaker diarization — “who spoke when” labelling (deliberately postponed to keep the core robust first).
• Levantine accent adaptation — additional Jordanian/Palestinian data to further improve dialect handling.

---

🧠 Model Fine-Tuning (Code-Switching)

The enhancement track targets the one remaining weakness: sentences that mix Arabic and English inline (e.g. "بدي أعمل Deploy للـ Backend"). The approach:

1. LoRA adapters on Whisper large-v3 (only ~2% of parameters trained) — trained on Kaggle's free dual-T4 GPUs.
2. Merge → convert to CTranslate2 (int8_float16) → run 100% locally with the same faster-whisper engine already in production.
3. Evaluate fine-tuned vs. base WER on a held-out split to prove the improvement before shipping.

Results — measured on 200 held-out code-switched clips (never seen during training):

Model	WER ↓	Word Accuracy ↑
Base Whisper large-v3	47.6%	52.4%
Fine-tuned (LoRA)	21.9%	78.1%

The fine-tune cut the Word Error Rate by more than half — a 25.7-point absolute drop (−54% relative) on mixed Arabic/English speech, training only 3.6% of the parameters.

Language-ID accuracy — perfect on clean, balanced speech; the harder, honest test is the developer's own accented English, fixed via accent arbitration (see Engineering Highlights):

Only training uses the cloud; inference remains fully offline, consistent with the project's privacy-first design.

---

👤 Author

Ahmad Arafat & Rashid Shehadeh — Graduation Project (GP2) Developed for the ESKADENIA Software Student Challenge.

---

_{Built with a privacy-first, fully-local philosophy — your voice stays on your machine.}