Real-time AI-powered gesture recognition
Built with TensorFlow, FastAPI, and React
Converts sign language into text and speech instantly
An AI powered system that uses computer vision and deep learning to recognize American Sign Language (ASL) gestures in real time and convert them into text and speech.
Designed as a real-world AI application for assistive communication and human-computer interaction.
Live demo coming soon (deployment in progress.)
This project demonstrates a full-stack AI system integrating:
- Computer Vision (MediaPipe, OpenCV)
- Deep Learning (TensorFlow CNN)
- Backend APIs (FastAPI)
- Frontend UI (React + Vite)
It enables real-time interaction between humans and machines using gesture-based communication.
- Real-time gesture recognition system
- Real-time webcam-based gesture detection
- Full-stack AI application
- Production-ready architecture
- Modular and scalable design
Sign Language Detection • ASL Recognition • Gesture Recognition • Computer Vision Project • Deep Learning Application • Real-Time AI System • TensorFlow CNN • FastAPI Backend • React Frontend
Interactive module to learn ASL alphabet gestures visually.
- Displays A–Z sign gestures
- Click any letter to start practicing
- Beginner-friendly learning interface
Real-time AI-based gesture recognition using webcam input.
Workflow:
- Open camera
- Perform sign gesture
- AI detects the gesture
- System verifies correctness
Output includes:
- Detected letter
- Confidence score
- Correct / Incorrect feedback
Search and explore sign gestures.
- Instant letter search
- Sign image display
- Gesture description
- Audio pronunciation
Converts detected gestures into real-time spoken audio using Google Text-To-Speech (gTTS).
Webcam Frame
↓
MediaPipe Hand Detection
↓
Hand Cropping
↓
Image Resize (64×64)
↓
CNN Model Prediction
↓
Prediction Stabilization
↓
Output Letter + Confidence
React Frontend
│
│ HTTP API
▼
FastAPI Backend
│
▼
TensorFlow CNN Model
│
▼
Prediction Result
│
▼
Text → Speech Output
- Model Type: CNN (TensorFlow/Keras)
- Framework: TensorFlow / Keras
- Classes: 26 (A–Z)
- Input Size: 64x64
- Accuracy: ~95% (approximate during testing)
- Dataset: Custom ASL dataset
- Problem: Unstable outputs frame-by-frame
- Solution: Frame averaging (prediction smoothing)
- Problem: Reduced accuracy in different environments
- Solution: Data augmentation
- Problem: Delay in inference
- Solution: Image resizing and optimized pipeline
- Problem: Model struggled with unseen hand variations
- Solution: Improved dataset diversity and augmentation
This project demonstrates:
- Integration of AI models into production systems
- Real-time data processing and inference
- Scalable backend architecture using FastAPI
- Frontend interaction with live AI predictions
- Deployment-ready architecture for real-world applications
- Assistive communication system for hearing and speech-impaired individuals
- Educational ASL learning platform
- Gesture-based human-computer interaction
- React
- Vite
- TailwindCSS
- React Router
- Axios
- Python
- FastAPI
- Uvicorn
- TensorFlow
- Keras
- OpenCV
- MediaPipe
- NumPy
- gTTS (Google Text-To-Speech)
sign-language-translator
│
├── ai-training
│ ├── dataset
│ └── train_model.py
│
├── backend
│ ├── requirements.txt
│ ├── run.py
│ │
│ ├── app
│ │ ├── main.py
│ │ │
│ │ ├── api
│ │ │ └── routes.py
│ │ │
│ │ ├── services
│ │ │ ├── gesture_detection.py
│ │ │ ├── sentence_builder.py
│ │ │ ├── translator.py
│ │ │ └── tts_service.py
│ │ │
│ │ ├── core
│ │ │ └── model_loader.py
│ │ │
│ │ └── models
│ │ └── asl_model.h5
│ │
│ └── static
│
├── frontend
│ ├── package.json
│ ├── vite.config.js
│ │
│ ├── public
│ │ └── signs
│ │ ├── A.png
│ │ ├── B.png
│ │ └── ...
│ │
│ └── src
│ ├── components
│ │ ├── camera
│ │ │ ├── CameraFeed.tsx
│ │ │ └── GestureStatus.tsx
│ │ │
│ │ └── ui
│ │ │ ├── Button.jsx
│ │ │ ├── Card.jsx
│ │ │ └── Section.jsx
│ │ │
│ │ ├── Footer.jsx
│ │ └── Navbar.jsx
│ │
│ ├── layout
│ │ └── AppLayout.jsx
│ │
│ ├── pages
│ │ ├── Alphabet.jsx
│ │ ├── Detector.jsx
│ │ ├── Dictionary.jsx
│ │ ├── Home.jsx
│ │ ├── Practice.jsx
│ │ └── Spelling.jsx
│ │
│ ├── sections
│ │ ├── CallToAction.jsx
│ │ ├── Features.jsx
│ │ ├── Hero.jsx
│ │ └── HowItWorks.jsx
│ │
│ ├── services
│ │ └── api.ts
│ │
│ ├── App.jsx
│ ├── main.jsx
│ └── index.css
│
├── images
│ ├── alphabet.png
│ ├── practice.png
│ └── dictionary.png
│
├── README.md
└── .gitignore
git clone https://github.com/vipulsystems/real-time-asl-sign-language-detection.git
cd real-time-asl-sign-language-detectioncd backend
python -m venv venvActivate virtual environment:
venv\Scripts\activate # Windows
source venv/bin/activate # Linux / MacInstall dependencies:
pip install -r requirements.txtRun backend server:
python run.pyBackend will run at: http://localhost:8000
Ensure backend is running before starting frontend
cd frontend
npm install
npm run devFrontend will run at: http://localhost:5173
- Open the application in your browser
- Navigate to the Alphabet Trainer
- Select a letter to practice
- Perform the gesture using your webcam
- View real-time prediction and feedback
- Use the dictionary for reference and learning
- Real-time continuous gesture detection
- Word-level sign recognition
- Sentence translation
- Mobile device support
- Transformer-based gesture models
- Large-scale sign language datasets
Contributions are welcome.
- Fork the repository
- Create a new branch (
feature/your-feature) - Commit your changes
- Push to your branch
- Open a Pull Request
MIT License
Vipul Paighan
Full Stack Developer | AI & Computer Vision Engineer
- Backend: Spring Boot, FastAPI, REST APIs
- Frontend: React.js, Tailwind CSS
- AI/ML: TensorFlow, OpenCV, NLP, GenAI APIs
- Databases: MySQL, PostgreSQL
- Git, GitHub
- Docker
- Postman
- Selenium, BeautifulSoup
- Data Analysis (Pandas, NumPy)
- Power BI, Tableau
Email: vipul.paighan@gmail.com
GitHub: https://github.com/vipulsystems
Focused on building scalable AI-powered applications and solving real-world problems using modern technologies.
If you like this project:
⭐ Star the repository
🍴 Fork the repository
📢 Share the project