VitalSense is a high-performance, webcam-based physiological monitor that extracts vital signs using remote Photoplethysmography (rPPG). Built as a modular "Skill-Based" system, it integrates 3D computer vision, digital signal processing (DSP), and Large Language Models (LLMs) to provide real-time heart rate, stress analysis, and health feedback.
VitalSense follows a strict Producer-Consumer multithreaded architecture to maintain a consistent 30 FPS webcam feed while performing heavy computation in the background.
- Orchestrator (
main.py): Owns the webcam capture and MediaPipe face tracking. - Signal Processor (
dsp_pipeline.py): Isolates the cardiac waveform using time-domain filters. - AI Feedback Layer (
ai_feedback.py): Communicates asynchronously with the Groq API for health insights. - Live Dashboard (
dashboard.py): Renders a three-panel Matplotlib visualization of the raw, filtered, and spectral data.
The system utilizes the MediaPipe Tasks API (0.10.x) for asynchronous face landmarking.
- Forehead ROI: Tracks specific landmarks (IDs: 10, 109, 67, etc.) to isolate the most stable skin region.
-
G-Mean Extraction: Calculates the average green-channel intensity (
$G_{mean}$ ) because human blood absorbs green light most effectively.
Acts as a "Gatekeeper" to prevent "silent failures" where noisy data leads to incorrect readings.
- Luminance Analysis: Converts frames to YCrCb color space to isolate brightness (Y) from skin tone.
-
Lighting Classifiers: Detects Low Light (
$\mu < 60$ ), Backlit ($\sigma > 80$ ), and Flicker ($Var > 25$ ) to pause extraction if conditions are poor.
Converts raw pixel fluctuations into a clean cardiac waveform.
-
SOS Butterworth Filter: A 2nd-order bandpass filter (
$0.7–4.0$ Hz) using Second-Order Sections to prevent numerical rounding errors. -
Zero-Phase Filtering: Employs
sosfiltfiltto ensure the filtered peaks align perfectly with physical heartbeats without time delay. - Spectral Analysis: Uses Fast Fourier Transform (FFT) to identify the dominant pulse frequency.
-
SNR Confidence Score: Calculates a trust metric as the spectral Signal-to-Noise Ratio in decibels:
$C_{dB} = 10\log_{10}(\frac{P_{peak}}{P_{noise}})$ . A reading is treated as reliable at or above a 3 dB gate (the standard "half-power" significance point).
Beyond average BPM, the system analyzes the variance between individual beats.
- Metrics: Computes RMSSD (Root Mean Square of Successive Differences) and SDNN.
- Classifier: A rule-based logic gate that labels user states as Calm, Moderate Stress, or High Stress based on RMSSD thresholds.
Integrates the Groq API (llama-3.3-70b-versatile) to provide health advice.
- Async Daemon Threads: Ensures API latency does not stutter the 30 FPS monitor.
- Safety Guardrails: Enforces a 60-word limit and prevents medical diagnosis.
- Code Hardening: Optimized signal padding (min 20 samples) and lazy-loading for the API client to ensure "buttery smooth" performance.
- Target Frame Rate: 30 FPS (Locked).
- Sampling Requirement: Minimum 8 Hz (Nyquist-compliant for heart rates up to 240 BPM).
- API Throttle: Every 30–60 seconds to respect Groq/Gemini RPM limits.
- Enhanced UI: Moving from OpenCV/Matplotlib to a custom-themed GUI.
- Multimodal Vitals: Researching SpO2 (oxygen saturation) and Respiratory Rate (RR) extraction.
- Edge Deployment: Further optimization for mobile and low-power hardware.
- Verkruysse et al. (2008)[cite_start]: Why the green channel is superior for rPPG.
- Poh et al. (2010)[cite_start]: Foundations of FFT-based heart rate estimation.
- Zhao et al. (2023) / Fontes et al. (2024): HRV and stress-level interpretation.
Developed by: Muhammad Arham
Affiliation: Dawood University of Engineering & Technology