Sprk3lzZ/HandTracking

HandTracking for Apple Vision Pro

A research and engineering project exploring high-fidelity hand tracking on Apple Vision Pro with custom gestures and physically‑accurate interactions in visionOS. The goal is to recreate a complete physics layer and build expressive gesture primitives that feel natural, reliable, and extensible.

Why this project

Following Apple's Vision Pro keynote and the announcement of new visionOS capabilities, this app was created to push beyond the default interaction patterns and experiment with:

Custom gesture design: building domain‑specific gestures on top of hand tracking data
Robust physics: recreating a complete and stable physics stack for tactile, believable interactions
Prototyping future use cases: testing how richer hand input changes UX across domains

Potential applications

Medical: hands‑free sterile workflows, virtual anatomy manipulation, simulation training
Aerospace: complex assembly guidance, cockpit procedure rehearsal, spatial checklists
Concept & design: rapid 3D ideation, model review, spatial sketching, collaborative critique
Art & performance: gestural instruments, live visuals, spatial choreography, installation art

Features

High‑quality hand tracking integration targeting visionOS on Apple Vision Pro
Custom gesture recognizers for nuanced, multi‑stage interactions (pose, motion, intent)
Complete physics layer with tuned parameters for stability and realism
Interaction primitives: grab, pinch, push, flick, throw, stretch, press, and compound gestures
Extensible architecture for adding new gestures, colliders, and interaction rules
RealityKit integration for rendering, collision shapes, and dynamic simulation

How it works (high level)

Sensing: read hand skeleton, joints, and pinch/pose signals provided by visionOS
Signal processing: filter, smooth, and normalize joint positions/orientations over time
Gesture pipeline: infer intent using temporal state machines and thresholds with hysteresis
Physics mapping: convert gesture intent to forces/constraints on RealityKit entities
State feedback: provide visual and physical cues to improve user confidence and control

Project structure

HandTracking/ — App sources (Swift, visionOS UI, interaction logic)
HandTracking/Assets.xcassets/ — App icons and assets
Packages/RealityKitContent/ — RealityKit content and supporting assets
HandTracking.xcodeproj/ — Xcode project

Requirements

Hardware: Apple Vision Pro (recommended) or visionOS Simulator
OS: macOS (Apple silicon)
Xcode: Xcode with visionOS SDK (ensure the latest stable release)
Languages & frameworks: Swift, visionOS, RealityKit
Apple Developer account: required for device deployment

Getting started

Open HandTracking.xcodeproj in Xcode.
Select the HandTracking target and choose a visionOS destination (Vision Pro or Simulator).
Ensure signing is configured under Project Settings > Signing & Capabilities.
Build and run.

Running on device

Use a registered Apple Developer account and a provisioned device profile.
In the scheme selector, pick your Vision Pro and press Run.

Usage

Launch the app and follow on‑screen guidance for neutral hand pose calibration.
Interact with visible entities using supported gestures (grab, pinch, press, flick, throw, etc.).
For best results, maintain comfortable lighting and keep hands within the tracked volume.

Custom gestures

This project implements a modular gesture pipeline that allows you to:

Define a gesture state machine (e.g., Idle → Candidate → Active → Cooldown)
Specify joint/pose constraints, velocity thresholds, and dwell times
Combine primitives into compound gestures (e.g., two‑hand stretch, rotate‑and‑press)

To add a new gesture:

Create a recognizer conforming to your internal gesture protocol
Subscribe to hand tracking updates and compute intent confidence over time
Emit high‑level actions consumed by the interaction/physics layer

Physics system

Rigid body dynamics with tuned damping and restitution
Stable one‑hand and two‑hand constraints for grabbing, pinning, and stretching
Impulse mapping for flicks/throws with velocity clamping and safety limits
Collision filtering and shape selection for reliable grasping

Roadmap

Expand gesture library (two‑hand sculpt, precision tools, bimanual macros)
Haptics and audio feedback models mapping intent confidence to cues
Shared spatial sessions for multi‑user interactions
Dataset collection and evaluation tools for gesture robustness
Performance profiling and battery‑aware scheduling

Contributing

Issues and pull requests are welcome. Please describe:

The scenario (device/simulator, lighting, distance)
Expected vs actual behavior
Repro steps and any relevant logs or screen captures

License

This project does not currently include an explicit open‑source license. If you intend to use the code beyond local experimentation, please open an issue to discuss licensing options.

Acknowledgments

Inspired by Apple's keynote announcing Apple Vision Pro and the evolving visionOS capabilities
Built with Swift, visionOS, and RealityKit