Project

SecondTouchReality — VR hand interaction system

XR · Computer vision · Natural language · Hardware feedback

A camera-based interaction prototype that connects hand tracking to virtual objects and a hardware-feedback path. SecondTouchReality is the software iteration of the team's 2025 OneTouchReality prototype.

My contribution is the Python and Unity software: tracking, hand reconstruction, bone updates and object interaction. Teammates handled the Arduino, servos and mechanical integration.

Python Unity / C# MediaPipe Arduino scikit-learn
Hand-tracking concept and the real team-built haptic glove
Concept illustration. Original project evidence follows below.

Implementation details

Hand tracking & depth: MediaPipe Hands detects 21 keypoints per hand. Wrist-to-camera distance in meters is estimated from palm width and palm length through a one-time calibration, with the two channels fused by weights based on finger curl and hand side-pose, plus median/EMA filtering. Each frame, the wrist 3D position, 20 bone direction vectors, and the pinch state are streamed to Unity as UDP JSON.

Interaction in Unity: HandFromVectors rebuilds the 3D skeleton from bone directions and lengths; PinchGrabBall makes any object pinch-grabbable with a release grace period to survive tracking dropouts; HandOrbitCamera turns pinch-and-move into orbit/zoom camera control when nothing is held.

Language → objects: a character n-gram HashingVectorizer + SGDClassifier maps description inputs to object labels behind a TCP server. Typing "a green apple" in the Unity dialog returns a label that ModelLibrary / RuntimeModelLoader map to a prefab or a GLB model loaded at runtime via GLTFast.

Hardware integration: the team prototype forwards pinch-state transitions over serial to an Arduino driving a servo.

Architecture

The software separates perception, text classification and scene interaction. UDP JSON carries hand state, TCP carries model labels, and a serial bridge connects the hardware-feedback path.

SecondTouchReality system architecture from camera input to Arduino haptic feedback

Prototype photographs

OneTouchReality five-servo force-feedback glove prototype
The three-person OneTouchReality force-feedback glove prototype.
MediaPipe hand tracking running on a laptop
Real-time hand landmark and gesture tracking during the prototype build.

Perception → interaction

Tracking the human hand.

Real-time hand landmark and gesture tracking during the prototype build. My contribution connects Python perception with Unity interaction.

Prototype demo

OneTouchReality, the earlier team prototype that led to SecondTouchReality.

Current scope

The repository contains the complete prototype path across Python, Unity, and Arduino. It remains an experimental system: the current haptic bridge is a prototype channel, and the media above documents the earlier team-built glove rather than a production-ready SecondTouchReality device.