EgoEVHands: Egocentric Event-Based Hand Dataset

• EgoEVHands is an egocentric event-based hand dataset that provides dense 3D/2D keypoints and gesture labels from stereo event streams.
• It comprises 5,419 annotated sequences spanning 38 gesture classes under normal and low-light conditions with rigorous stereo calibration and annotation protocols.
• The accompanying EgoEV-HandPose framework employs iterative BEV fusion and geometry-aware refinement to substantially reduce pose estimation errors amid occlusion and ego-motion challenges.

Searching arXiv for the cited topic and closely related papers to ground the article. EgoEVHands denotes, in its most specific usage, the dataset introduced with "EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras" (Wang et al., 12 May 2026): the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. In a broader research sense, the term also refers to an emerging area of egocentric event-based hand perception for XR, HCI, and robotics, where stereo event streams, monocular first-person event cameras, and geometry-aware filtering are used to address ego-motion interference, depth ambiguity, self-occlusion, mutual occlusion, and low-light capture (Wang et al., 12 May 2026, Xu et al., 17 Sep 2025, Hara et al., 25 May 2025).

1. Definition and problem setting

EgoEVHands is situated at the intersection of egocentric vision, event-based sensing, and hand perception. Its central problem is 3D bimanual hand pose estimation and, in the dataset’s originating paper, joint gesture recognition from head-mounted stereo event streams. The motivation is explicit: conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets (Wang et al., 12 May 2026).

The event-camera premise is technically specific. Event sensors asynchronously emit polarity events with microsecond-level timestamps and very high dynamic range, which is directly relevant to fast egocentric hand motion and low-light capture. In first-person settings, the hands appear at close range, frequently self-occlude or mutually occlude, and are observed under substantial camera motion. The literature around EgoEVHands treats these as primary failure modes rather than edge cases. EventEgoHands, for example, identifies camera-motion-induced background events as the defining obstacle for egocentric event-based 3D hand mesh reconstruction (Hara et al., 25 May 2025), while EvHand-FPV frames first-person event-based 3D hand tracking as an on-device XR problem requiring simultaneous accuracy, low latency, and low compute (Xu et al., 17 Sep 2025).

A terminological ambiguity is intrinsic to the topic. "EgoEVHands" is the dataset name in (Wang et al., 12 May 2026), but nearby work uses closely related names for distinct artifacts: "EventEgoHands" denotes an egocentric event-based 3D hand mesh reconstruction method (Hara et al., 25 May 2025), and "EgoExo-Hands" denotes an ego–exo evaluation set for in-the-wild 3D hand tracking rather than an event-camera resource (Rim et al., 2 Oct 2025). The literature therefore uses the term both as a proper dataset name and as shorthand for a broader egocentric event-based hand-perception agenda.

2. Dataset composition and annotation protocol

The EgoEVHands dataset was created to supply real-world stereo event data with dense hand-pose supervision. It contains 5,419 fully annotated samples from 10 participants across 38 gesture classes. The classes are grouped into single-hand interaction, bimanual interaction without occlusion, self-occlusion, and mutual occlusion. Illumination includes normal-light and low-light subsets. Each sequence includes left and right stereo event streams, RealSense D435 infrared frames and depth maps, pixel-level hand masks, dense 3D keypoints and their 2D projections, gesture labels, and split files for homogeneous and heterogeneous protocols (Wang et al., 12 May 2026).

The head-mounted capture rig integrates two Prophesee EVK4 event cameras with 4 mm wide-angle lenses and one Intel RealSense D435 RGB-D camera. The stereo module and annotation pipeline assume known intrinsics and extrinsics, and the cameras are rigidly time-synchronized for egocentric recording. The paper states that the dataset “Res” is 720, matching the RealSense IR/depth 720p streams; the event-camera pixel resolution is not explicitly reported (Wang et al., 12 May 2026).

The annotation pipeline is anchored in the RealSense IR and depth channels. Grounded-SAM is applied to RealSense IR frames to generate hand segmentation masks. MediaPipe detects 2D hand keypoints on three IR variants—original, masked, and background-removed—and the most reliable set is auto-selected and manually screened or refined by expert annotators. These 2D IR keypoints are back-projected using per-pixel depth and intrinsics to obtain sparse 3D joints. Neighborhood averaging is used for missing depth, limited 3D interpolation fills small temporal gaps, and frames with large gaps are left invalid. The resulting 3D joints are projected into the left and right event-camera planes using calibrated $K$, $R$, and $T$ to produce aligned 2D keypoints and masks. Visibility flags label points as original, interpolated, or invalid (Wang et al., 12 May 2026).

The hand model is a 21-keypoint per hand skeleton, yielding 42 keypoints for two hands. Unlike MANO-based pipelines, the dataset annotation does not fit a parametric hand mesh; bone lengths are implicitly determined by the 3D labels from depth. This design keeps the released supervision close to the measured IR-depth geometry rather than to a fitted template. The 3D keypoints are defined in the RealSense camera or world frame used during back-projection, and the reported pose metrics are in millimeters (Wang et al., 12 May 2026).

Two evaluation protocols are defined. In the heterogeneous, cross-subject split, participants 1, 2, and 9 form the test set and the remaining participants are used for training. In the homogeneous, within-subject split, the first three repetitions per gesture are used for training and the remaining repetitions form the test set. These protocols separate identity generalization from within-subject repetition generalization (Wang et al., 12 May 2026).

3. EgoEV-HandPose and the KeypointBEV formulation

The method introduced with the dataset, EgoEV-HandPose, is an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. Its central module, KeypointBEV, lifts stereo features into a canonical bird’s-eye-view space and performs iterative reprojection-guided refinement to resolve depth ambiguity and enforce kinematic consistency (Wang et al., 12 May 2026).

Event pre-aggregation uses Locally Normalized Event Surfaces over a temporal window $\Delta T$. The accumulation given in the paper is

$$S(x) = \sum_{i \in E, \Delta T} p_i \cdot \max\left(0, 1 - \frac{t - t_i}{\Delta T}\right).$$

Stereo geometry is then incorporated in an iterative update loop. At refinement iteration $k$, projected points are obtained by

$$\tilde{u}_v^{(k)} = K_v [R_v \mid T_v] \tilde{X}^{(k)},$$

left and right features are sampled by bilinear interpolation at the current projections, and the pose estimate is updated as

$P^{(k+1)} = P^{(k)} + \eta^{(k)} \Delta P^{(k)}.$

The method-context section of the paper characterizes this as feature-level BEV fusion with explicit geometry, rather than dense cost-volume construction (Wang et al., 12 May 2026).

Gesture recognition is coupled to pose through wrist-centric and scale normalization. For joint $j$ at time $t$, the normalized pose is

$R$0

followed by

$R$1

where $R$2 is the MCP of the middle finger. This normalization is explicitly described as part of the gesture pipeline in (Wang et al., 12 May 2026).

Training follows a staged curriculum: 2D heatmap and mask pretraining, BEV iterative 3D refinement with reprojection penalties at intermediate iterations, cross-entropy gesture training, and joint fine-tuning. The paper summarizes the objective as

$R$3

Practical training details are also reported: Adam optimizer, initial learning rate $R$4, joint fine-tuning learning rate $R$5, step decay of $R$6 every 10 epochs, total 25 epochs, and gradient clipping with max-norm $R$7 (Wang et al., 12 May 2026).

4. Evaluation metrics and reported performance

The principal pose metric for EgoEVHands is 3D mean per-joint position error in millimeters,

$R$8

The paper also reports PA-MPJPE, 2D pixel error, PCK AUC over thresholds from 0 to 50 mm, and Top-1 gesture recognition accuracy. Evaluation is reported on both homogeneous and heterogeneous splits, and on scenario subsets including normal vs. low-light and single-hand vs. bimanual with no occlusion, self-occlusion, and mutual occlusion (Wang et al., 12 May 2026).

On the heterogeneous cross-subject split, EgoEV-HandPose reports an MPJPE of 30.54 mm and a Top-1 gesture recognition accuracy of 86.87%. On the homogeneous split, the reported MPJPE is 22.03 mm. The method also improves PCK AUC from 0.436 for initial triangulation to 0.589 after KeypointBEV refinement. Scenario breakdown on the heterogeneous split reports 31.49 mm in normal light and 29.50 mm in low light; for bimanual settings the overall MPJPE is 29.86 mm, and for mutual occlusion it is 31.02 mm (Wang et al., 12 May 2026).

The comparative gains are reported in relative terms. The method achieves a 57.5% error reduction versus stereo RGB HandMvNet and a 73.5% error reduction versus monocular event Ev2Hands on the heterogeneous split. Efficiency is also quantified: 8.44M parameters, 19.86 GFLOPs, and approximately 13.6 FPS with stereo processing and a shared backbone (Wang et al., 12 May 2026).

These results are notable because they tie together three properties that the egocentric event literature often trades off against one another: depth recovery, low-light robustness, and bimanual occlusion handling. The paper attributes a substantial part of the gain to iterative stereo refinement: a coarse triangulation result of 52.72 mm is reduced to 30.54 mm after the final KeypointBEV iteration (Wang et al., 12 May 2026).

5. Relation to monocular first-person event methods

The EgoEVHands dataset and EgoEV-HandPose framework belong to a broader line of egocentric event-based hand methods, but the design choices differ sharply across papers. EventEgoHands targets event-based egocentric 3D hand mesh reconstruction from a single egocentric event stream. Its pipeline combines a U-Net Hand Segmentation Module on Locally-Normalized Event Surfaces with a PointNet++ hand reconstruction module operating on filtered event clouds, and uses cross-attention between two hand branches to avoid requiring per-event left/right labels (Hara et al., 25 May 2025). On its eventized HOT3D-derived test set, EventEgoHands reports R-AUC 0.450, MPJPE 59.51 mm, and MPVPE 42.92 mm, compared with 0.243/105.80 mm/65.23 mm for EventHands and 0.251/106.75 mm/69.66 mm for Ev2Hands. Removing segmentation degrades the method to R-AUC 0.392, MPJPE 66.24 mm, and MPVPE 47.11 mm (Hara et al., 25 May 2025).

EvHand-FPV addresses a different subproblem: efficient first-person-view 3D hand tracking from a single event camera for XR. It constructs an event-based FPV dataset with synthetic 3D labels for training and real event data with 2D labels for evaluation, compresses events into a single-channel representation, applies LNES-Fast accumulation with early-stop thresholding, and localizes the hand using a wrist-based ROI. Its backbone is MobileViT V2 with ROI-offset embedding at the final fully connected layer and an auxiliary geometric feature head used only during training (Xu et al., 17 Sep 2025). On the real FPV test set, EvHand-FPV improves 2D-AUCp from 0.77 to 0.85 while reducing parameters from 11.2M to 1.2M and FLOPs per inference from 1.648G to 0.185G; on synthetic data it reports a 3D-AUCp of 0.84. A 160×160 ROI reduces FLOPs by 42.72% with only a 0.01 drop in 2D-AUCp, from 0.85 to 0.84 (Xu et al., 17 Sep 2025).

The relationship among these systems is therefore methodological rather than nominal. EgoEV-HandPose uses synchronized stereo event views and explicit geometry; EventEgoHands uses segmentation plus point-cloud reconstruction for bimanual mesh recovery; EvHand-FPV emphasizes wrist-based localization and compact mobile deployment. All three, however, are explicitly motivated by the same first-person constraints: ego-motion, background events, self-occlusion, rapid wrist rotation, and the need to preserve high temporal fidelity under limited compute budgets (Wang et al., 12 May 2026, Hara et al., 25 May 2025, Xu et al., 17 Sep 2025).

6. Naming, adjacent research directions, and limitations

A recurring source of confusion is that several neighboring resources are not EgoEVHands, even when they address egocentric hands. "Ego-Exo 3D Hand Tracking in the Wild with a Mobile Multi-Camera Rig" introduces an evaluation set named EgoExo-Hands, not EgoEVHands. It uses eight exocentric monochrome fisheye cameras, two egocentric Meta Quest 3 views, and five OptiTrack PrimeX 13W cameras for headset tracking. Its contribution is a marker-less ego–exo pipeline for 42 3D hand keypoints per frame with sub-centimeter median MPJPE against a 30-camera dome, not event-based sensing (Rim et al., 2 Oct 2025). "EgoEMG" is likewise adjacent but distinct: it synchronizes bilateral wrist EMG, IMU, egocentric RGB, external RGB-D, and mocap-derived 22-DoF hand joint-angle labels for multimodal pose estimation, rather than event-based perception (Xi et al., 7 May 2026).

The limitations of EgoEVHands as a dataset are also explicit. Subject diversity is 10 participants, which the paper describes as adequate for a first benchmark but still modest. Illumination diversity is documented, but the number and types of environments are not detailed. The scope is centered on free-hand gestures rather than explicit hand–object interaction annotation. The event-camera resolution and detailed sensor metadata are not fully enumerated in the paper (Wang et al., 12 May 2026).

Related methods expose additional practical limits that matter for interpreting the field. EvHand-FPV does not report end-to-end latency or energy measurements, even though its 1.2M parameters and 0.185 GFLOPs per inference are presented as compatible with mobile NPUs and GPUs; it can fail under extreme occlusions or when wrist geometry is insufficiently visible, and wrist localization can degrade under dense background events at wrist height (Xu et al., 17 Sep 2025). EventEgoHands identifies severe hand-object occlusions and fine fingertip articulation as continuing failure cases, and it does not introduce an explicit temporal smoothness loss (Hara et al., 25 May 2025).

The broader implication is that EgoEVHands should not be treated as a single closed benchmark problem. In the literature, it denotes a technically specific stereo event dataset and method (Wang et al., 12 May 2026), but it also names a wider research direction in which annotation strategy, event representation, geometry, mobile efficiency, and first-person robustness are co-designed rather than optimized independently.