Real-time AR interaction (left) synchronized with a digital twin simulation (right).
A real-time, computer vision–driven digital twin of a cantilever beam that visualizes load application and bending behavior using only a webcam and hand tracking.
This project was developed as part of a Strength of Materials (SOM) application to bridge theoretical beam bending concepts with interactive, visual intuition—without using physical sensors.
Traditional cantilever beam analysis relies on formulas, simulations, or strain sensors to study deflection behavior.
This project demonstrates an alternative approach:
Use computer vision to detect a real beam and human interaction, then simulate bending behavior through a synchronized digital twin.
The system detects a physical cantilever beam marked with a colored strip, tracks fingertip interaction as a point load, and renders:
- an AR bending overlay on the real beam, and
- a separate digital twin simulation that mirrors the same behavior in real time.
Physical cantilever beam marked with a green strip for computer vision–based detection.
- Webcam (built-in or external)
- Physical cantilever beam (thin strip or ruler)
- Green tape or marker for beam detection
- Adequate ambient lighting
Classical cantilever behavior under a point load. Deflection increases quadratically toward the free end.
- Cantilever beam behavior under a point load
- Load position–dependent deflection
- Fixed-end vs free-end constraints
- Qualitative bending behavior inspired by classical SOM theory
- Digital twin synchronization between physical and simulated systems
⚠️ Note:
This is a qualitative, physics-inspired visualization, not a numerically exact EI-based solver.
The system integrates computer vision, hand tracking, and custom physics logic in a single real-time loop.
- Beam is wrapped with a green strip for robust detection
- HSV color segmentation is used to isolate the beam
cv2.minAreaRectidentifies beam geometry- The fixed end and free end are inferred automatically
- Pressing
clocks the beam geometry as the reference (undeflected state)
- MediaPipe Hands detects 21 landmarks
- Only Landmark 8 (index fingertip) is used
- Fingertip is treated as a moving point load
- Finger position is projected:
- along the beam axis (load location)
- perpendicular to the beam (load direction & magnitude)
Finger interaction is classified into three mechanical states:
- Near – finger close to the beam (no contact)
- Touching – contact detected, no bending yet
- Load Applied – downward push detected, bending triggered
Hover filtering and directional checks prevent false load application.
- Deflection curve increases quadratically from fixed end to free end
- Inspired by classical cantilever behavior:
δ ∝ x² - Deflection magnitude depends on:
- normalized load position (x/L)
- downward finger displacement
- tunable sensitivity parameters
- A separate simulation panel renders a digital cantilever beam
- Displays:
- bending shape
- load position (x%)
- applied force direction
- Updates synchronously with real-world interaction
c→ Lock detected beam geometryu→ Unlock beamq→ Quit application
- Python
- OpenCV – image processing, geometry, rendering
- MediaPipe – real-time hand landmark tracking
- NumPy – vector math and projections
pip install opencv-python mediapipe numpypython main.pyEnsure:
- A webcam is connected
- The beam is clearly marked with a green strip
- Adequate lighting for color segmentation
- Stable beam detection using color segmentation
- Reliable finger-based load classification
- Real-time bending visualization
- Digital twin accurately mirrors qualitative cantilever behavior
- Integrate force sensors for quantitative analysis
- Support multiple or distributed loads
- Auto-generate SFD & BMD plots
- Extend to other structural elements
- Mobile deployment for classroom demonstrations
This project was developed as part of a Strength of Materials course to provide an interactive, visual understanding of beam bending concepts using computer vision.
- Pranav Anil