Dynamics Based 3D Skeletal Hand Tracking (1705.07640v1)

Abstract: Tracking the full skeletal pose of the hands and fingers is a challenging problem that has a plethora of applications for user interaction. Existing techniques either require wearable hardware, add restrictions to user pose, or require significant computation resources. This research explores a new approach to tracking hands, or any articulated model, by using an augmented rigid body simulation. This allows us to phrase 3D object tracking as a linear complementarity problem with a well-defined solution. Based on a depth sensor's samples, the system generates constraints that limit motion orthogonal to the rigid body model's surface. These constraints, along with prior motion, collision/contact constraints, and joint mechanics, are resolved with a projected Gauss-Seidel solver. Due to camera noise properties and attachment errors, the numerous surface constraints are impulse capped to avoid overpowering mechanical constraints. To improve tracking accuracy, multiple simulations are spawned at each frame and fed a variety of heuristics, constraints and poses. A 3D error metric selects the best-fit simulation, helping the system handle challenging hand motions. Such an approach enables real-time, robust, and accurate 3D skeletal tracking of a user's hand on a variety of depth cameras, while only utilizing a single x86 CPU core for processing.

Dynamics Based 3D Skeletal Hand Tracking: A Technical Evaluation

Abstract Overview

"Dynamics Based 3D Skeletal Hand Tracking" presents a computational approach for tracking the full skeletal pose of human hands using depth sensors. The paper introduces an augmented rigid body simulation model for 3D hand tracking that operates on a single x86 CPU core in real-time. The proposed method frames hand tracking as a linear complementarity problem resolved by a Gauss-Seidel solver, integrating constraints from spatial data, prior motion, collisions, and joint mechanics.

Technical Contributions

The core innovation in this paper is the formulation of 3D hand tracking using rigid body dynamics, a stark departure from typical kinematic approaches. Through impulse capping and multiple heuristic-driven simulations per frame, the system robustly handles complex hand motions. The proposed model uniquely generates 3D surface constraints directly from depth sensor data, employing a voxel-based subsampling method to improve efficiency while minimizing sensor noise artifacts.

Numerical Results and Claims

The research asserts that the method operates comfortably at over 60 Hz on a single core of an x86 processor, despite the compute-intensive nature of rigid body dynamics simulation. This claim highlights the system's scalability and efficiency, potentially offering compatibility across various depth cameras, irrespective of the specific sensor or its resolution. The simulations, which can handle fast and subtle motions, maintain tracking robustness without requiring additional hardware beyond the sensor itself.

Implications

Practically, this tracking solution can significantly advance user interaction modalities across applications requiring intricate hand motions, such as virtual reality and gesture recognition systems. Theoretically, the introduction of physics-based constraints into tracking systems suggests a pathway to enhancing model accuracy by exploiting mechanical properties, potentially inspiring similar applications in other areas of articulated motion capture.

Future Directions

The paper's emphasis on modular adaptability suggests future work exploring improvements in softbody representations and RGB-based tracking enhancements. It also opens the possibility for extending the approach to full-body tracking demanding similar computational efficiency and robustness. Additionally, applications leveraging 3D hand tracking in multitouch interfaces could redefine user interaction paradigms, albeit requiring innovative solutions to overcome the absence of haptic feedback.

Conclusion

The methodology proposed by Melax et al. pushes the boundaries of markerless hand tracking by integrating dynamics with advanced computational strategies to produce a robust system capable of operating in real-time on consumer-grade hardware. This approach beckons further research to realize its full potential in enriching human-computer interaction and possibly transcending current limitations observed in gesture recognition and motion gaming applications.