EvHand-FPV: Lightweight Egocentric Hand Tracking

• The paper introduces EvHand-FPV, a framework that reduces parameters by 89% and FLOPs by 89% while enhancing real-time 3D hand tracking accuracy.
• It leverages an event-based sensing model with a wrist-based ROI and early stopping in accumulation to handle rapid, nonrigid hand motion effectively.
• Combining synthetic and real FPV datasets, the method improves real-data 2D-AUCp from 0.77 to 0.85 and maintains competitive synthetic 3D-AUC performance.

EvHand-FPV denotes a lightweight framework for egocentric First-Person-View 3D hand tracking from a single event camera, designed for settings in which low latency, energy efficiency, and robustness to rapid nonrigid motion are jointly constrained, such as Extended Reality devices. The framework couples an event-based FPV dataset, a wrist-based region of interest, an end-to-end mapping strategy that embeds ROI offsets into the network, and a multi-task learning strategy with an auxiliary geometric feature head. On a real FPV test set, it improves 2D-AUCp from 0.77 to 0.85 while reducing parameters from 11.2M to 1.2M by 89% and FLOPs per inference from 1.648G to 0.185G by 89%, and it maintains a competitive 3D-AUCp of 0.84 on synthetic data (Xu et al., 17 Sep 2025).

1. Problem formulation and event-based sensing

EvHand-FPV is motivated by the observation that XR devices impose tight constraints: low end-to-end latency for responsive interaction, strict power envelopes, and high accuracy under rapid hand motion. Conventional RGB and depth pipelines are described as struggling in this regime because images are captured at fixed frame rates such as 30–60 FPS, which introduces motion blur and raises sensor and compute power requirements. Event cameras are used instead because they asynchronously report per-pixel brightness changes with microsecond temporal resolution and mW-level power, while eliminating motion blur and producing sparse signals only where motion or illumination changes occur (Xu et al., 17 Sep 2025).

The sensing model is expressed in terms of events

$e_i=(x_i,y_i,t_i,p_i),$

where $p_i\in\{+1,-1\}$ denotes the polarity of the brightness change. With

$L(x,y,t)=\log I(x,y,t),$

an event is triggered when

$|\Delta L(x,y,t)|\ge C,$

with polarity assignment $p_i=+1$ if $\Delta L\ge C$ and $p_i=-1$ if $\Delta L\le -C$. This formulation underlies the framework’s emphasis on sparse, temporally precise egocentric sensing rather than frame reconstruction.

The target task is single-camera FPV 3D hand tracking. The output space is a 12D vector consisting of MANO PCA coefficients, 3D translation, and hand rotation. A central design premise is that egocentric hand tracking can exploit geometric regularities specific to first-person imagery, particularly the fact that the forearm typically enters the lower part of the field of view and the wrist forms a narrow junction before the hand expands.

2. Event representation and wrist-based localization

EvHand-FPV adopts a single-channel, window-normalized event image derived from events within a temporal bin. Let $E_{[t_0-L,t_0]}$ denote the set of events arriving in a fixed window of length $L$ ending at time $p_i\in\{+1,-1\}$0. Channel compression merges both polarities to reduce complexity, so a pixel is marked if either polarity occurs, without double-counting. The accumulated representation is

$p_i\in\{+1,-1\}$1

with linear time-decay weight

$p_i\in\{+1,-1\}$2

This is described as a fast, window-normalized LNES-Fast variant (Xu et al., 17 Sep 2025).

To balance fast and slow motion regimes, the framework uses early stopping during accumulation. Events are processed in reverse time until either $p_i\in\{+1,-1\}$3 or the cumulative event count exceeds a threshold $p_i\in\{+1,-1\}$4; accumulation terminates once count $p_i\in\{+1,-1\}$5. Gaussian blur is then applied per bin,

$p_i\in\{+1,-1\}$6

to suppress isolated spurious events while smoothing the representation. The design goal is to avoid over-blurring during fast motion and over-sparsity during slow motion without incurring the cost of multi-bin temporal filtering.

Hand localization is performed through a wrist-based ROI defined by geometric cues in the accumulated event image. The algorithm initializes $p_i\in\{+1,-1\}$7, $p_i\in\{+1,-1\}$8, and $p_i\in\{+1,-1\}$9, then scans rows from bottom to top. For each row $L(x,y,t)=\log I(x,y,t),$0, it finds leftmost $L(x,y,t)=\log I(x,y,t),$1 and rightmost $L(x,y,t)=\log I(x,y,t),$2 event locations exceeding a small density threshold $L(x,y,t)=\log I(x,y,t),$3, computes

$L(x,y,t)=\log I(x,y,t),$4

and identifies the wrist location $L(x,y,t)=\log I(x,y,t),$5 near the local minimum width preceding re-expansion. The wrist center is

$L(x,y,t)=\log I(x,y,t),$6

Given a target ROI of size $L(x,y,t)=\log I(x,y,t),$7, the vertical span is defined as $L(x,y,t)=\log I(x,y,t),$8 and the horizontal span as $L(x,y,t)=\log I(x,y,t),$9.

This localization strategy is explicitly tied to egocentric anatomy. The data state that wrist-based ROI suppresses background and camera-motion events common in FPV, and that viewpoint assumptions matter: the wrist narrowing heuristic presumes typical FPV geometry, so atypical camera placements or rotated views may reduce ROI localization reliability.

3. Network architecture and learning objective

EvHand-FPV builds on MobileViT V2 and introduces two explicit architectural simplifications: SiLU is replaced with ReLU, and softmax in linear self-attention is approximated with a Taylor series. The processing pipeline begins with an ROI feature extractor consisting of two 2D convolutions and two inverted residual layers in the MobileViT V2 style. The network then branches into a main head and an auxiliary head (Xu et al., 17 Sep 2025).

The main head comprises a sequence of MobileViT V2 blocks, global average pooling, and two linear layers. The auxiliary head uses an inverted residual layer, global average pooling, and a linear layer that predicts seven geometric statistics: $|\Delta L(x,y,t)|\ge C,$0, $|\Delta L(x,y,t)|\ge C,$1, $|\Delta L(x,y,t)|\ge C,$2, $|\Delta L(x,y,t)|\ge C,$3, $|\Delta L(x,y,t)|\ge C,$4, $|\Delta L(x,y,t)|\ge C,$5, and $|\Delta L(x,y,t)|\ge C,$6. These correspond to the mean of event coordinates, standard deviations along axes, eigenvalues of the $|\Delta L(x,y,t)|\ge C,$7 covariance matrix $|\Delta L(x,y,t)|\ge C,$8, and the orientation of the principal axis. Formally,

$|\Delta L(x,y,t)|\ge C,$9

A distinctive component is the end-to-end mapping that embeds ROI offsets into the final prediction layer rather than reconstructing predictions explicitly from ROI coordinates back into the original image space. If $p_i=+1$0 denotes flattened ROI features and $p_i=+1$1 the ROI top-left offset, the final layer predicts

$p_i=+1$2

where $p_i=+1$3 is the 12D output. The output dimensions are: 1–6 MANO PCA coefficients, 7–9 3D translation, and 10–12 hand rotation.

Training uses weighted MSE losses for the main head and unweighted MSE for the auxiliary geometric features. The total loss is

$p_i=+1$4

The main loss is

$p_i=+1$5

with $p_i=+1$6, $p_i=+1$7, and $p_i=+1$8. The component losses are

$p_i=+1$9

The auxiliary loss is

$\Delta L\ge C$0

The auxiliary branch is pruned at test time, so it improves representation learning without inference overhead.

4. Dataset construction, labels, and implementation

The framework addresses the scarcity of egocentric event-based benchmarks by constructing a dataset that combines synthetic training data with 3D labels and real event data with 2D labels. Synthetic data are generated with evsim based on MANO and include synchronized RGB, event streams, 3D joints, and 12D PCA labels with $\Delta L\ge C$1s timestamps. The synthetic training split has size 720,000 ms, and the synthetic test split has size 60,000 ms. Diversity is introduced through randomized camera viewpoints, gesture transformations, translations, and rotations (Xu et al., 17 Sep 2025).

The real FPV evaluation set uses a DAVIS346 event camera with 346×240 event resolution, synchronized to 30 FPS RGB. Sequences are up to 60 s, with total real test set duration approximately 60,000 ms. The data include different hand shapes and sizes, and motion patterns spanning planar translation, depth motion, wrist rotation, and gesture variations. Annotation is limited to 2D hand joint coordinates via MediaPipe bootstrapping, refined by manual calibration; no real 3D labels are available.

Domain-gap mitigation is built into three parts of the design. First, single-channel compression reduces polarity-induced domain variability. Second, wrist-based ROI suppresses background and camera-motion events common in FPV. Third, the auxiliary geometric head encourages pose-aware features without extra inference cost. Training also includes LNES-Fast and denoising for robustness under sparse and noisy real events.

Implementation details are specified. Training uses PyTorch Lightning, the Adam optimizer, learning rate $\Delta L\ge C$2, batch size 32, and 20 epochs. Event preprocessing uses channel compression, LNES-Fast accumulation with window length $\Delta L\ge C$3 and threshold $\Delta L\ge C$4, and Gaussian blur with kernel $\Delta L\ge C$5 and variance $\Delta L\ge C$6. At inference time, the default ROI size is 160×160, and the reported footprint is 0.185 G FLOPs with 1.2 M parameters.

5. Evaluation protocol, empirical results, and ablations

Evaluation is split into real-data 2D assessment and synthetic-data 3D assessment. For 2D, palm-normalized PCK is defined by

$\Delta L\ge C$7

and the area under the curve is

$\Delta L\ge C$8

For synthetic 3D evaluation,

$\Delta L\ge C$9

with

$p_i=-1$0

3D evaluation is reported on synthetic data only (Xu et al., 17 Sep 2025).

The principal reported comparisons are summarized below.

Model or setting	Real 2D-AUCp	Synthetic 3D-AUC / 3D-AUCp
EventHands trained for third-person view	0.11	0.17
Retrained baseline (ResNet-18 on EvHand-FPV data)	0.77	0.82
EvHand-FPV	0.85	0.84

The table reflects three distinct observations. First, the transfer gap is large: EventHands trained for third-person view performs poorly without retraining on FPV, dropping to 0.11 in 2D-AUCp and 0.17 in 3D-AUC. Second, retraining on EvHand-FPV data raises the ResNet-18 baseline to 0.77 on real 2D-AUCp and 0.82 on synthetic 3D-AUC. Third, EvHand-FPV improves the real-data 2D result to 0.85 while maintaining 0.84 on synthetic 3D evaluation.

Ablations isolate the contribution of backbone selection, ROI cropping, and the auxiliary head. Among backbones pruned to approximately 0.1 M parameters for fairness, MobileViT V2 reaches 0.80 2D-AUCp, compared with 0.78 for ShuffleNet V2 and 0.76 for MobileNet V3. ROI size reduction to 160×160 lowers FLOPs from 0.322 G to 0.185 G, a 42.72% reduction, with only a 0.01 accuracy drop from 0.85 to 0.84. The ROI and auxiliary-head ablation is as follows.

Variant	2D-AUCp	FLOPs
No ROI, no AUX	0.85	0.322 G
ROI only	0.84	0.185 G
AUX only	0.87	0.322 G
ROI + AUX	0.85	0.185 G

The reported interpretation is narrow and empirical: the auxiliary head consistently improves feature quality, while ROI preserves accuracy within 0.01 and cuts FLOPs. Relative to the ResNet-18 EventHands baseline, the full framework reduces parameters from 11.2 M to 1.2 M and FLOPs from 1.648 G to 0.185 G, both by 89%.

6. Limitations, failure cases, and research significance

The framework’s limitations are stated explicitly. Extreme occlusions or self-occlusions degrade 2D and 3D accuracy because purely monocular egocentric events lack disambiguating cues. Very low-motion scenes can reduce feature richness because event sparsity limits information content. Background clutter and ego-motion can trigger spurious events; wrist-based ROI suppresses many but not all of them. The wrist narrowing heuristic also depends on viewpoint regularity, so atypical camera placements or rotated views can reduce localization reliability (Xu et al., 17 Sep 2025).

Several extensions are identified in the source material. These include multi-camera fusion for ambiguity resolution, IMU integration for ego-motion compensation, spiking neural networks for further energy reduction, learnable ROI via lightweight STN, and stronger kinematic priors such as MANO shape regularization and bone-length constraints. These are framed as future directions rather than as validated components of the present system.

Within event-based hand tracking, the significance of EvHand-FPV lies in its co-design of representation, geometry, and architecture for egocentric deployment constraints. The framework uses sparse, microsecond event sensing; channel-compressed LNES-Fast with early stopping; wrist-centric ROI plus end-to-end offset conditioning; and a MobileViT V2-based architecture with 1.2 M parameters. The reported result is accurate and efficient egocentric event-based hand tracking suitable for on-device XR applications (Xu et al., 17 Sep 2025).