XR · Computer vision · Natural language · Hardware feedback
A camera-based interaction prototype that connects hand tracking to virtual objects and a hardware-feedback
path. SecondTouchReality is the software iteration of the team's 2025 OneTouchReality prototype.
My contribution is the Python and Unity software: tracking, hand reconstruction, bone updates and
object interaction. Teammates handled the Arduino, servos and mechanical integration.
PythonUnity / C#MediaPipeArduinoscikit-learn
Concept illustration. Original project evidence follows below.01 /
Implementation details
Hand tracking & depth: MediaPipe Hands detects 21 keypoints per hand. Wrist-to-camera
distance in meters is estimated from palm width and palm length through a one-time calibration, with the two
channels fused by weights based on finger curl and hand side-pose, plus median/EMA filtering. Each frame,
the wrist 3D position, 20 bone direction vectors, and the pinch state are streamed to Unity as UDP JSON.
Interaction in Unity:HandFromVectors rebuilds the 3D skeleton from bone
directions and lengths; PinchGrabBall makes any object pinch-grabbable with a release grace
period to survive tracking dropouts; HandOrbitCamera turns pinch-and-move into orbit/zoom camera
control when nothing is held.
Language → objects: a character n-gram HashingVectorizer + SGDClassifier maps description
inputs to object labels behind a TCP server. Typing "a green apple" in the Unity dialog
returns a label that ModelLibrary / RuntimeModelLoader map to a prefab or a GLB
model loaded at runtime via GLTFast.
Hardware integration: the team prototype forwards pinch-state transitions over serial to
an Arduino driving a servo.
02 /
Architecture
The software separates perception, text classification and scene interaction. UDP JSON carries hand state,
TCP carries model labels, and a serial bridge connects the hardware-feedback path.
03 /
Prototype photographs
The three-person OneTouchReality force-feedback glove prototype.
Real-time hand landmark and gesture tracking during the prototype build.
Perception → interaction
Tracking the human hand.
Real-time hand landmark and gesture tracking during the prototype build. My contribution connects Python perception with Unity interaction.
04 /
Prototype demo
OneTouchReality, the earlier team prototype that led to SecondTouchReality.
05 /
Current scope
The repository contains the complete prototype path across Python, Unity, and Arduino. It remains an
experimental system: the current haptic bridge is a prototype channel, and the media above documents the
earlier team-built glove rather than a production-ready SecondTouchReality device.