EgoEV-HandPose: 3D Bimanual Estimation

• The paper introduces EgoEV-HandPose, a unified framework that jointly estimates 3D bimanual hand poses and recognizes gestures from synchronized stereo event streams.
• It employs a stereo event backbone with 2D heatmap and segmentation decoders, refined by the novel KeypointBEV module for iterative reprojection and depth ambiguity resolution.
• The system, trained via staged pretraining and fine-tuning on the EgoEVHands dataset, achieves superior MPJPE and gesture accuracy in challenging low-light and occluded settings.

Searching arXiv for the specified paper and closely related work to ground the article in current literature. EgoEV-HandPose is an end-to-end framework for joint egocentric 3D bimanual hand-pose estimation and gesture recognition from stereo event-camera streams. It is designed for settings in which conventional frame-based cameras are limited by motion blur and dynamic-range constraints, and in which prior event-based methods are affected by ego-motion interference, monocular depth ambiguity, severe self-occlusions, and the lack of large-scale real-world stereo datasets. The framework combines a stereo event backbone, 2D heatmap and segmentation decoding, a geometry-aware stereo fusion module called KeypointBEV, and a temporal transformer for gesture recognition, and it is introduced together with EgoEVHands, described as the first large-scale real-world stereo event-camera dataset for egocentric hand perception (Wang et al., 12 May 2026).

1. Problem setting and naming context

In its 2026 formulation, EgoEV-HandPose addresses joint egocentric 3D bimanual hand-pose estimation and gesture recognition from synchronized stereo event streams. The stated goal is to recover 3D keypoints and gesture labels directly from stereo event-camera input, without reliance on parametric hand meshes, while remaining robust to low light, ego-motion noise, and bimanual occlusion (Wang et al., 12 May 2026).

A source of potential confusion is that the same name has also been used for an earlier egocentric RGB framework centered on 2D hand pose and action recognition. That earlier system combines EffHandNet for single-hand estimation, EffHandEgoNet for egocentric two-hand estimation, and a lightweight transformer over 2D hand and object poses, achieving 91.32% accuracy on H2O and 94.43% on FPHA (Mucha et al., 2024). The shared label therefore refers to two distinct technical lineages: a 2D RGB pose-plus-action pipeline in 2024, and a stereo event-based 3D pose-plus-gesture pipeline in 2026. A plausible implication is that the later work reuses the broad egocentric hand-perception framing while shifting the sensing substrate and the geometric formulation.

The 2026 system identifies four main technical obstacles: motion blur and limited dynamic range of conventional RGB/D sensors; ego-motion noise and depth ambiguity in monocular event methods; severe self-occlusions in bimanual egocentric views; and the lack of large-scale real-world stereo event-camera datasets. Its central claim is that explicit stereo geometry and iterative reprojection-guided refinement are sufficient to make event streams competitive for both 3D pose recovery and gesture recognition in these conditions (Wang et al., 12 May 2026).

2. Input representation and end-to-end architecture

The input to EgoEV-HandPose is a pair of synchronized stereo event streams. Each stream $\mathcal{E}_v$ is converted into a Locally-Normalized Event Surface (LNES),

$$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$

yielding a 2-channel image of size $H\times W\times 2$. The temporal window used for LNES is $\Delta T = 50$ ms (Wang et al., 12 May 2026).

These LNES representations are processed by a shared-weight backbone named EgoBlaze. EgoBlaze consists of a stack of asymmetric depthwise convolutions and State-Space Model blocks, together with a “confidence gate” that suppresses head-motion spikes. Two lightweight decoders are attached to the backbone: a heatmap decoder that outputs 42D heatmaps for left and right 2D joints, and a segmentation decoder that outputs a hand mask. The 2D predictions are then consumed by KeypointBEV, which produces refined 3D keypoints. Gesture recognition is handled by a separate temporal head operating on normalized 3D keypoint trajectories (Wang et al., 12 May 2026).

The temporal gesture branch applies wrist-centering and palm-scale normalization. For joint $j$ at time $t$, it uses

$$\bar{\mathbf{p}}_{j,t}=\mathbf{p}_{j,t}-\mathbf{p}_{\mathrm{wrist},t}, \qquad \hat{\mathbf{p}}_{j,t}=\bar{\mathbf{p}}_{j,t}/(\|\bar{\mathbf{p}}_{9,t}\|_2+\epsilon).$$

The normalized pose is flattened to a 168D vector per frame, embedded to 512D, prepended with a learnable $[\mathrm{CLS}]$ token, augmented with positional encodings, and processed by a 3-layer Transformer encoder. An MLP followed by Softmax outputs a 38-way gesture distribution $\hat y$ (Wang et al., 12 May 2026).

3. KeypointBEV and stereo geometric refinement

KeypointBEV is the core stereo fusion module. Its defining operation is to lift 2D event-backbone features into a canonical bird’s-eye-view space and refine a coarse 3D pose by an iterative reprojection-guided loop. The initial 3D estimate is obtained by triangulating left and right 2D joint heatmaps $\{\mathbf{u}_L,\mathbf{u}_R\}$ using calibrated intrinsics $$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$0 and extrinsics $$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$1,

$$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$2

Left and right feature maps $$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$3 are passed through a $$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$4 convolution, flattened spatially, and concatenated with a learnable joint-identity embedding to form initial queries $$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$5 (Wang et al., 12 May 2026).

The system runs $$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$6 refinement iterations. At iteration $$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$7, the current estimate $$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$8 is projected back into both image planes, feature vectors are sampled bilinearly from the per-view feature maps, a spatial descriptor is computed from the current 3D coordinates, and a Transformer decoder predicts a residual update. The projected coordinates are

$$S(\mathbf{x}) = \sum_{i\in\mathcal{E}_v,\;\Delta T} p_i\,\max\!\bigl(0,\;1-\tfrac{t-t_i}{\Delta T}\bigr),$$9

with homogeneous coordinates $H\times W\times 2$0. For feature sampling, the projected coordinates are normalized to $H\times W\times 2$1:

$H\times W\times 2$2

The sampled features are

$H\times W\times 2$3

and the spatial descriptor is defined as

$H\times W\times 2$4

A Transformer decoder $H\times W\times 2$5 predicts a residual,

$H\times W\times 2$6

and the pose update is

$H\times W\times 2$7

where $H\times W\times 2$8 is a learned step size (Wang et al., 12 May 2026).

The role of this loop is explicitly twofold: progressively resolve depth uncertainty and enforce kinematic consistency. The ablation in which simple 2D cross-attention yields 42.98 mm MPJPE, while KeypointBEV yields 30.54 mm, suggests that the BEV lifting plus reprojection-guided refinement is not reducible to generic feature fusion alone (Wang et al., 12 May 2026).

4. Objectives, staged training, and optimization

EgoEV-HandPose is trained with a staged procedure. The 2D backbone is first pretrained using a 2D objective

$H\times W\times 2$9

where $\Delta T = 50$0 is MSE between predicted heatmaps and Gaussian targets, and $\Delta T = 50$1 is BCE between the predicted mask and the ground-truth mask. KeypointBEV is then trained with the 2D backbone frozen, the action head is pretrained with the geometry module frozen, and the full system is finally jointly fine-tuned (Wang et al., 12 May 2026).

The 3D pose loss is MPJPE over the final prediction,

$\Delta T = 50$2

At each refinement iteration $\Delta T = 50$3, a reprojection loss penalizes the difference between the projection of $\Delta T = 50$4 and the ground-truth 2D joints. The KeypointBEV-stage objective is

$\Delta T = 50$5

For joint fine-tuning, the total loss is

$\Delta T = 50$6

where $\Delta T = 50$7 is the gesture classification loss (Wang et al., 12 May 2026).

The reported typical weights are $\Delta T = 50$8, $\Delta T = 50$9, $j$0, and $j$1. Optimization uses Adam in PyTorch on a single NVIDIA A800 GPU, with learning rate $j$2 in the pretraining phases and $j$3 during joint fine-tuning, step decay by $j$4 every 10 epochs, total 25 epochs, gradient clipping with max norm $j$5, batch size 32, query embedding dimension $j$6, and gesture-head embedding dimension 512 (Wang et al., 12 May 2026).

5. EgoEVHands dataset and annotation pipeline

EgoEVHands is introduced together with the framework and described as the first real-world stereo event-camera dataset for egocentric hand perception. The capture rig uses two Prophesee EVK4 event cameras with 4 mm wide-angle lenses for the left and right event streams, plus an Intel RealSense D435 RGB-D device for the annotation back-end, mounted on a rigid head-mounted rig with synchronized recordings at 720 px resolution (Wang et al., 12 May 2026).

The annotation pipeline is semi-automatic. Hand masks are obtained via Grounded-SAM on RealSense IR. Two-dimensional keypoints, 21 per hand, are generated by Mediapipe on IR variants and then manually refined. These are back-projected to 3D using depth and intrinsics to obtain sparse 3D ground truth, followed by temporal interpolation over small gaps only to produce dense 3D annotations. The 3D keypoints are then re-projected to the left and right event planes, producing aligned 2D keypoints and masks, together with visibility flags distinguishing original, interpolated, and invalid annotations (Wang et al., 12 May 2026).

The dataset contains 5,419 sequences $j$7 2 views, 10 subjects, and 38 gesture classes. It is organized into four semantic groups: single-hand, bimanual no-occlusion, self-occlusion, and mutual-occlusion. It also includes normal and low-light scenarios. The reported data volume is approximately 300 m events, and the summary describes the total frames as approximately 5,419 at the sequence level (Wang et al., 12 May 2026). This design directly supports heterogeneous and homogeneous evaluation splits as well as scenario-wise error analysis.

6. Quantitative performance and comparative positioning

The main reported quantitative results are summarized below.

Evaluation setting	EgoEV-HandPose result	Context
HETER 3D pose	2D error = 16.64 px; MPJPE = 30.54 mm; PA-MPJPE = 21.08 mm	HandMvNet 71.87 mm; Ev2Hands 115.13 mm; EvHandPose 60.20 mm; EventEgo3D 72.76 mm
HOMO 3D pose	2D error = 11.48 px; MPJPE = 22.03 mm; PA-MPJPE = 18.96 mm	—
Gesture recognition	Top-1 accuracy = 86.87%	38 classes, HETER split
Iterative refinement	MPJPE 52.72 mm $j$8 30.54 mm	PCK AUC 0.589 vs 0.436 init triangulation
Complexity	8.44 M params; 19.86 GFLOPs; $j$913.6 FPS	NVIDIA A800 GPU

Scenario-wise, on the HETER split the method reports 31.49 mm in normal light and 29.50 mm in low light. It reports 33.88 mm for single-hand sequences and 29.86 mm for bimanual sequences overall, with 27.43 mm for interaction no-occlusion, 30.89 mm for self-occlusion, and 31.02 mm for mutual-occlusion (Wang et al., 12 May 2026). The paper’s abstract characterizes these gains as especially strong in low-light and bimanual occlusion scenarios.

The cross-architecture result on DHP19 is also notable: integrating KeypointBEV into DEV-Pose reduces MPJPE from 55.53 mm to 47.88 mm, corresponding to a 13.8% error reduction (Wang et al., 12 May 2026). This suggests that the KeypointBEV formulation is not tied exclusively to the EgoBlaze backbone.

In the broader event-hand-pose literature, the closest conceptual predecessor in the provided record is EvHandPose, which is a weakly supervised event-based 3D hand pose estimator built around MANO regression, hand-flow representations, and Pose-to-IWE self-supervision. EvHandPose is formulated for sparse supervision and reports strong robustness to fast motion and HDR scenes, including 19.8 mm / 4.4 px on EvRealHands normal fixed and 29.0 mm / 6.8 px under strong light fixed, but it is based on a different supervision regime and a different problem formulation than the stereo, mesh-free, bimanual EgoEV-HandPose setting (Jiang et al., 2023). A common misconception is that event sensing by itself resolves pose ambiguity; the contrast between monocular event baselines and the stereo KeypointBEV design indicates that explicit stereo geometry remains central to reducing depth ambiguity in egocentric 3D pose recovery.

7. Related extensions, adjacent modalities, and open directions

A related line of work proposes that dorsal-skin deformation can complement egocentric hand-pose estimation under self-occlusion. DeltaDorsal introduces a dual-stream delta encoder that contrasts a relaxed dorsal reference image $t$0 against a dynamic dorsal image $t$1, using a shared DINOv3 Vision Transformer featurizer and a small 4-layer ConvNet over $t$2. In self-occluded scenarios with fingers at least 50% occluded, it reports MPJAE 7.59 ± 4.38 versus 9.24 ± 5.54 for HaMeR, corresponding to an 18% reduction, and a related exposition explicitly describes how its ideas could slot into an “EgoEV-HandPose” framework by replacing a single-stream hand-silhouette backbone with a two-branch silhouette-plus-dorsal-delta design (Huang et al., 21 Jan 2026). This suggests a route toward hybrid egocentric systems in which event geometry and dorsal appearance cues address different occlusion regimes.

A second adjacent direction is multimodal sensing with surface electromyography. EgoEMG provides bilateral wristband EMG with 16 total channels, IMU, egocentric RGB, external RGB-D, and mocap-derived hand motion for a common 22-DoF joint-angle prediction target. Its benchmark includes EMG-to-pose, vision-to-pose, and EMG+vision fusion, and the residual fusion architecture consistently reduces MAE by approximately 0.4–0.6° over matched generic vision backbones, with the occlusion-stratified fusion gain growing from +0.24° at low self-occlusion to +0.52° at high occlusion (Xi et al., 7 May 2026). A plausible implication is that future egocentric event-camera systems could incorporate EMG when vision-derived cues become unreliable under persistent self-occlusion or poor visibility.

Finally, the earlier RGB-based EgoEV-HandPose framework remains relevant as a complementary point in the design space. It demonstrates that a purely 2D pipeline based on EffHandEgoNet and a compact transformer can be competitive for egocentric action understanding from RGB, with a full 20-frame action pipeline taking approximately 96 ms per action on an RTX 3090 (Mucha et al., 2024). Read alongside the stereo event formulation, this indicates that “EgoEV-HandPose” now denotes a small family of egocentric hand-perception systems spanning 2D RGB, stereo event-based 3D geometry, and proposed extensions toward dorsal-feature and multimodal fusion.