Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor (1704.02201v2)

Abstract: We present an approach for real-time, robust and accurate hand pose estimation from moving egocentric RGB-D cameras in cluttered real environments. Existing methods typically fail for hand-object interactions in cluttered scenes imaged from egocentric viewpoints, common for virtual or augmented reality applications. Our approach uses two subsequently applied Convolutional Neural Networks (CNNs) to localize the hand and regress 3D joint locations. Hand localization is achieved by using a CNN to estimate the 2D position of the hand center in the input, even in the presence of clutter and occlusions. The localized hand position, together with the corresponding input depth value, is used to generate a normalized cropped image that is fed into a second CNN to regress relative 3D hand joint locations in real time. For added accuracy, robustness and temporal stability, we refine the pose estimates using a kinematic pose tracking energy. To train the CNNs, we introduce a new photorealistic dataset that uses a merged reality approach to capture and synthesize large amounts of annotated data of natural hand interaction in cluttered scenes. Through quantitative and qualitative evaluation, we show that our method is robust to self-occlusion and occlusions by objects, particularly in moving egocentric perspectives.

• The paper introduces a dual-CNN framework that separates hand localization and 3D pose regression to robustly track hands even in occluded scenes.
• It leverages a photorealistic synthetic dataset, SynthHands, to overcome data scarcity and ensure diverse hand pose training.
• The system’s real-time performance and improved accuracy in cluttered environments enhance VR/AR applications and human-computer interaction.

Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor

The paper "Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor" presents an innovative approach for accurate hand pose estimation using egocentric RGB-D cameras, particularly in complex, cluttered environments. Recognizing the inadequacies of existing methods when dealing with occlusions typical in first-person views, the authors propose a system leveraging convolutional neural networks (CNNs) to enhance the robustness and accuracy of hand tracking.

The key contribution of this research lies in its dual-CNN framework designed to tackle hand localization and 3D pose estimation separately. This two-tier architecture significantly improves performance in environments with substantial occlusions and background clutter. The first CNN (termed HALNet) localizes the hand by estimating the 2D position of the hand center, thus overcoming challenges posed by cluttered scenes. The output from HALNet facilitates the cropping of the image around the hand, which is then processed by the second CNN (JORNet) to regress the relative 3D positions of hand joints in real time. Moreover, the incorporation of a kinematic pose tracking energy ensures temporal stability and refined pose estimates.

For training, the authors introduce a novel photorealistic dataset called SynthHands. This dataset simulates real-world diversity through comprehensive variations in hand poses, shapes, colors, and interactions with virtual objects. The dataset addresses the challenge of labeled data scarcity by using synthetic images that are annotated automatically, providing a rich source for training machine learning models.

Quantitative and qualitative evaluations demonstrate that this methodology is robust in handling self-occlusions and scenarios where the hands interact with objects while maintaining real-time performance. The authors further validate their system against a newly curated benchmark dataset, EgoDexter, featuring hand interactions from egocentric viewpoints in cluttered settings. The evaluations underscore the robustness of the system, as it outperforms state-of-the-art methods prepared for third-person tracking in clutter-free backgrounds.

From a theoretical perspective, this work extends the capabilities of CNNs in complex perceptual tasks by showing how decomposing the problem into localization and regression can lead to more robust models. The findings suggest a potential pathway for advancing human-computer interaction in VR/AR contexts by enabling seamless and natural interaction through accurate hand tracking. This has direct utility in enhancing immersive experiences and can be extended to other applications like activity recognition and motion control.

Practically, the system holds promise for VR/AR applications where precise hand movements are critical, meeting the demands for intuitive interaction in augmented environments. As VR/AR technologies gain prevalence, methodologies like this one will likely become foundational components of user interface design.

Future work may extend this research by refining the kinematic models to accommodate more dynamic and complex hand-object interactions or introducing domain adaptation techniques to further improve generalization across diverse camera setups. Additionally, while the RGB-D sensor and synthetic dataset address current challenges effectively, improvements in synthetic realism using advanced computer vision techniques such as GANs for generating even more lifelike training datasets might enhance system performance.

Overall, this work offers significant advancements in the field of hand pose estimation, demonstrating how integrated approaches utilizing CNNs and synthetic datasets can tackle the challenges of occlusion in real-time tracking applications.