SecondTouchReality — VR hand interaction system

XR · Computer vision · Natural language · Hardware feedback

An end-to-end prototype that turns a plain webcam into a VR input device: Python tracks your hand in 3D, Unity reconstructs a virtual hand you can pinch and grab with, a lightweight text model spawns 3D objects from English descriptions ("a small red apple"), and an Arduino servo closes the loop with physical feedback when you pinch. The full chain is Perception → Semantics → Interaction → Hardware, evolved from my earlier OneTouchReality gesture-controlled robotic arm project.

Problem and design

Most hand-tracking demos stop at drawing landmarks on a screen. I wanted a complete, hackable loop where a single camera is enough to manipulate virtual objects naturally — no headset, no glove, no depth sensor. The key design problems were: estimating real-world hand depth from a monocular camera, making grabbing feel stable despite noisy tracking, and letting users create scene content by simply describing it in words.

The architecture is deliberately modular: Python handles perception (hand keypoints, depth, pinch state) and semantics (text → object label), Unity handles reconstruction and interaction, and an Arduino bridge handles physical feedback. Each module talks over simple protocols (UDP JSON, line-based TCP, single-byte serial), so any part can be swapped or tested in isolation.

Implementation details

Hand tracking & depth: MediaPipe Hands detects 21 keypoints per hand. Wrist-to-camera distance in meters is estimated from palm width and palm length through a one-time calibration, with the two channels fused by weights based on finger curl and hand side-pose, plus median/EMA filtering. Each frame, the wrist 3D position, 20 bone direction vectors, and the pinch state are streamed to Unity as UDP JSON.

Interaction in Unity: HandFromVectors rebuilds the 3D skeleton from bone directions and lengths; PinchGrabBall makes any object pinch-grabbable with a release grace period to survive tracking dropouts; HandOrbitCamera turns pinch-and-move into orbit/zoom camera control when nothing is held.

Language → objects: a HashingVectorizer + SGDClassifier model (trained on my own collected dataset of description/label pairs) runs behind a TCP server. Typing "a green apple" in the Unity dialog returns a label that ModelLibrary / RuntimeModelLoader map to a prefab or a GLB model loaded at runtime via GLTFast.

Hardware: pinch-state transitions are forwarded over serial to an Arduino driving a servo — a minimal but complete haptic channel, designed to scale toward a multi-servo force-feedback glove.

Demo video

可直接上传视频，或嵌入外部视频链接。

Results and future work

The prototype runs as a single python main.py plus one Unity scene: the virtual hand follows in real time, objects can be pinched, moved and released reliably, text queries spawn the right model with confidence scores around 0.9 on the trained categories, and the servo responds to pinch with low latency.

Next steps: replace the linear text classifier with embedding-based semantic retrieval (and Chinese input support), add more dynamic gestures beyond pinch (fist, point, wave), generate whole teaching scenes instead of single objects, and extend the single-servo channel to a multi-servo force-feedback glove.