EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

Abstract: Egocentric 3D hand pose estimation and gesture recognition are essential for immersive augmented/virtual reality, human-computer interaction, and robotics. However, conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets. To overcome these limitations, we propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. Central to our approach is KeypointBEV, a flexible stereo fusion module that lifts features into a canonical bird's-eye-view space and employs an iterative reprojection-guided refinement loop to progressively resolve depth uncertainty and enforce kinematic consistency. In addition, we introduce EgoEVHands, the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance with an MPJPE of 30.54mm and 86.87% Top-1 gesture recognition accuracy, significantly outperforming RGB-based stereo and prior event-camera methods, particularly in low-light and bimanual occlusion scenarios, thereby setting a new benchmark for event-based egocentric perception. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose.

• The paper presents a KeypointBEV fusion module that iteratively refines 3D hand pose estimates to overcome depth ambiguity and enforce kinematic consistency.
• It achieves state-of-the-art accuracy with a mean per joint error of 30.54 mm and 86.87% Top-1 gesture recognition across 38 gesture classes.
• The integration of the EgoEVHands dataset and efficient multi-task optimization paves the way for robust egocentric hand perception in AR/VR and wearable HCI systems.

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

Introduction

"EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras" (2605.12297) presents an end-to-end framework for joint 3D bimanual hand pose estimation and gesture recognition utilizing stereo event cameras from an egocentric viewpoint. The paper identifies limitations of conventional RGB/depth-based and monocular event-based sensing—primarily motion blur, illumination sensitivity, depth ambiguity, and insufficient real-world datasets for stereo event streams. EgoEV-HandPose addresses these through the KeypointBEV fusion module, which iteratively refines fused features in a canonical bird's-eye-view space to resolve depth uncertainty and enforce kinematic consistency. The contribution is augmented by the EgoEVHands dataset, a large-scale real-world stereo event-camera benchmark for egocentric hand perception featuring dense 3D/2D keypoints across 38 gesture classes.

EgoEVHands Dataset

EgoEVHands is the first large-scale stereo event-camera dataset tailored for egocentric bimanual hand perception. Acquisition utilizes a head-mounted rig combining a RealSense D435 RGB-D camera and dual Prophesee EVK4 event cameras, facilitating synchronized high-dynamic-range recording via wide-angle lenses.

Figure 1: The head-mounted capture system integrates stereo event cameras and RGB-D for synchronous egocentric recording.

The annotation pipeline combines semi-automatic segmentation (Grounded-SAM) and keypoint detection (MediaPipe) on infrared images, followed by manual refinement, 3D projection using depth values, and alignment with event-camera views. This results in dense multimodal labels—segmentation masks and synchronized 3D pose data.

Figure 2: Hand segmentation mask generated by Grounded-SAM.

EgoEVHands comprises 5,419 annotated samples from 10 participants under diverse illumination and occlusion conditions, covering 38 gesture types, partitioned into single-hand, bimanual (with/without occlusion), self-occlusion, and mutual occlusion. Manifold analysis demonstrates strong intra-class consistency and structural richness despite occlusions and variable lighting.

Figure 3: t-SNE visualization shows clear separation and intra-class consistency across semantic interaction groups in EgoEVHands dataset.

EgoEV-HandPose Framework

The pipeline processes synchronized stereo event streams through three stages: (1) EgoBlaze backbone for modality-aware 2D feature extraction with ego-motion suppression, (2) KeypointBEV module for geometry-aware stereo fusion and iterative 3D pose refinement, (3) a wrist-normalized temporal transformer classifier for gesture recognition.

Figure 4: Overview of EgoEV-HandPose pipeline: stereo event streams are lifted to BEV via EgoBlaze and KeypointBEV, enabling refined 3D pose estimation and gesture classification.

Stereo Event Fusion (KeypointBEV)

KeypointBEV leverages calibrated intrinsics/extrinsics to triangulate initial 3D pose estimates. Iterative refinement combines reprojection-guided sampling from backbone features with gated spatial encodings, using transformer decoders to update pose estimates adaptively. This BEV-based fusion exploits explicit cross-view geometry and local feature correspondences, directly addressing depth ambiguity and occlusion.

Temporal Gesture Recognition

Normalized joint coordinates are fed into a transformer encoder with learned temporal positional encodings, enabling robust long-term gesture classification. Cross-entropy loss and multi-stage curriculum ensure stable optimization jointly with pose estimation.

Multi-task Optimization

The training pipeline utilizes a staged curriculum: (1) pretraining 2D feature heads, (2) freezing backbone for KeypointBEV training with reprojection and geometric losses, (3) trajectory-based gesture classifier pretraining, (4) end-to-end fine-tuning for unified kinematic and semantic representations.

Experimental Results

EgoEV-HandPose achieves state-of-the-art accuracy on EgoEVHands: MPJPE of 30.54 mm in heterogeneous test splits and 86.87% Top-1 gesture recognition accuracy. The framework is robust across lighting and occlusion scenarios, maintaining low error rates even in mutual occlusion and low-light conditions.

Figure 5: Qualitative results under normal light scenarios.

Qualitative visualizations illustrate precise and temporally consistent reconstructions under challenging illumination and motion conditions, outperforming RGB and monocular event-based methods.

Comparative Analysis

Compared with HandMvNet (RGB-based stereo), Ev2Hands (monocular event-based), and EventEgo3D (monocular event-based), EgoEV-HandPose reduces MPJPE by 57.5% and 73.5%, respectively, with significantly improved robustness and efficiency (8.44M params, 19.86 GFLOPs, 13.6 FPS). Ablation studies confirm KeypointBEV's superiority over alternative fusion strategies, demonstrating 30.54 mm MPJPE versus >42.98 mm for cross-view matching baselines.

Figure 6: Qualitative comparison highlighting superior finger joint localization, especially under occlusion.

Confusion matrices and PCK curves demonstrate high discriminative power for complex gesture categories and consistent error reduction through iterative refinement.

Figure 7: Confusion matrix on gesture recognition illustrating clear separability across 38 classes.

Figure 8: Iterative refinement stages show steady MPJPE reduction.

Figure 9: PCK curve demonstrates significant improvement in correct keypoints vs. baseline triangulation.

Generalization and Efficiency

Cross-architecture transfer to the DHP19 human pose dataset reduces DEV-Pose baseline MPJPE from 55.53 mm to 47.88 mm, validating KeypointBEV's plug-and-play geometric fusion capabilities across domains. Module-wise complexity analysis shows optimal trade-off between stereo geometric precision and computational cost, positioning EgoEV-HandPose for AR/VR and mobile HCI deployment.

Figure 10: Qualitative comparison on DHP19 dataset, showing improved GT alignment and pose accuracy with KeypointBEV integration.

Implications and Future Directions

EgoEV-HandPose sets a new benchmark for event-based egocentric bimanual pose estimation and gesture recognition, addressing the deficiencies of monocular and frame-based approaches. The geometry-aware, stereo BEV fusion strategy is theoretically compelling—providing explicit physical grounding—and practically efficient for power-constrained wearable systems.

Potential future directions include:

• Self-supervised pretraining for reduced manual annotation overhead.
• Dynamic structural gating to optimize single vs. dual-hand processing efficiency.
• Tracking-based optimization to reduce redundant computations for static frames.
• Integration into real-time AR/VR pipelines and mobile platforms.

The release of EgoEVHands will enable further research in event-based egocentric perception, spurring development of resource-efficient, geometry-consistent AI systems.

Conclusion

EgoEV-HandPose demonstrates decisive advances in 3D hand pose estimation and gesture recognition with stereo event cameras. Its geometry-anchored BEV fusion resolves depth ambiguities and occlusions, surpassing prior RGB and event-based baselines quantitatively and qualitatively. The EgoEVHands dataset establishes rigorous ground truth for this domain, catalyzing future work in egocentric event-based perception pivotal for next-generation AR/VR and wearable HCI systems.