E-3DPSM: A State Machine for Event-Based Egocentric 3D Human Pose Estimation

Abstract: Event cameras offer multiple advantages in monocular egocentric 3D human pose estimation from head-mounted devices, such as millisecond temporal resolution, high dynamic range, and negligible motion blur. Existing methods effectively leverage these properties, but suffer from low 3D estimation accuracy, insufficient in many applications (e.g., immersive VR/AR). This is due to the design not being fully tailored towards event streams (e.g., their asynchronous and continuous nature), leading to high sensitivity to self-occlusions and temporal jitter in the estimates. This paper rethinks the setting and introduces E-3DPSM, an event-driven continuous pose state machine for event-based egocentric 3D human pose estimation. E-3DPSM aligns continuous human motion with fine-grained event dynamics; it evolves latent states and predicts continuous changes in 3D joint positions associated with observed events, which are fused with direct 3D human pose predictions, leading to stable and drift-free final 3D pose reconstructions. E-3DPSM runs in real-time at 80 Hz on a single workstation and sets a new state of the art in experiments on two benchmarks, improving accuracy by up to 19% (MPJPE) and temporal stability by up to 2.7x. See our project page for the source code and trained models.

• The paper introduces a neural state machine that continuously evolves latent 3D pose states using event streams for precise estimation.
• It employs convolutional backbones with bidirectional SSM blocks and deformable attention to effectively handle fast motion and occlusions.
• The integration of a neural Kalman-style filter reduces pose jitter and drift, achieving up to 19% MPJPE reduction in benchmark tests.

E-3DPSM: A State Machine for Event-Based Egocentric 3D Human Pose Estimation

Motivation and Contributions

E-3DPSM introduces a new approach to monocular egocentric 3D human pose estimation, leveraging event cameras mounted on head-worn devices. Event cameras provide ultra-high temporal resolution, high dynamic range, and minimal motion blur, making them advantageous for wearable applications where fast motion and unpredictable lighting are common. Prior event-camera solutions suffer from substantial jitter, drift, and inadequate accuracy under occlusions, due to architectures not fully aligned with the asynchronous, change-driven nature of event streams.

E-3DPSM rethinks the estimation problem through three core insights:

1. Event streams encode changes in 2D, which should be mapped directly to continuous changes in 3D joint positions. The model accumulates these delta pose updates for robust trajectory estimation.
2. Pose prediction should be expressed as a continuous state evolution, aligning latent pose states with the dynamic event stream.
3. Intermediate supervision with 2D heatmaps and segmentation masks can be eliminated; all relevant features are learned end-to-end.

This leads to a novel architecture: E-3DPSM maintains continuous latent pose states, evolves them synchronously with incoming events, and fuses delta and direct pose estimates through a learned neural Kalman-style filter.

Architecture and Technical Approach

Event Representation and State-Space Modeling

Incoming asynchronous events are transformed into Locally Normalised Event Surface (LNES) frames, encoding spatial and temporal characteristics (Equation 2). This representation supports compatibility with convolutional backbones.

State-space models (SSMs), specifically S5 layers, provide explicit temporal context aggregation. These layers are deployed at hierarchical encoder stages to capture long-range dependencies crucial during occlusion or sparse event input.

Figure 1: Architecture of SPEM, integrating convolutional hierarchy, SSM blocks, deformable attention, and joint-query decoding.

Spatiotemporal Pose Encoder Module (SPEM)

SPEM combines convolutional pyramidal feature extractors with stage-wise deformable attention, enabling spatial focus on pose-relevant locations. Bidirectional SSM blocks operate on per-location feature sequences, producing temporally stabilized representations.

Joint-query transformer decoding extracts joint-specific features, associating learned query tokens with anatomical joints, facilitating robust pose regression across highly distorted ego-centric inputs.

Pose Regression Module (PRM) and Fusion

PRM produces direct 3D pose estimates and regresses delta updates between consecutive LNES frames. These delta estimates, inherently aligned with the event-driven nature of the inputs, are fused with direct pose predictions using a learnable (neural) Kalman-style filter. The filter maintains a pose state and uncertainty covariance, adaptively balancing between global stability and local motion cues.

Figure 2: Pose drift over time; learned fusion effectively mitigates temporal drift compared to naive or direct-only fusion.

Experimental Results

Main Results and Occlusion Robustness

On EE3D-R and EE3D-W benchmarks, E-3DPSM achieves up to 19% MPJPE reduction and 2.7× improvement in smoothness error ($e_\text{smooth}$) over EventEgo3D++ and EgoPoseFormer baselines. The causal variant (real-time deployment) maintains strong performance, with further gains under non-causal (batch) inference.

Figure 3: Qualitative results showing jitter improvement on EE3D-R sequences.

Figure 4: Qualitative results showing jitter improvement on EE3D-W, highlighting stability in wild, challenging conditions.

Occlusion-only evaluation demonstrates substantial error reduction in end-effector joints (knees, ankles, wrists), particularly for heavy occlusion scenarios.

Qualitative Comparisons

E-3DPSM predictions remain anatomically plausible and temporally consistent across actions, robust under occlusion, fast motion, and illumination changes, outperforming prior methods even under challenging visibility and motion conditions.

Figure 5: End-effector joint displacement plots for EE3D-R and EE3D-W, highlighting reduced jitter.

Figure 6: Per-action qualitative comparison on EE3D-W (challenging sequences).

Figure 7: Per-action qualitative comparison on EE3D-R (walk and challenge actions).

Ablation Studies

Ablation experiments validate that bidirectional SSM blocks, deformable attention, and learned fusion modules are key to accuracy and smoothness. Removal of SSM or fusion modules leads to severe performance degradation, confirming the critical role of continuous state evolution and adaptive fusion in handling event stream sparsity, occlusions, and drift.

Real-Time Device and Viewer

Deployed on head-mounted setups with NVIDIA Jetson Orin Nano, E-3DPSM operates fully stand-alone at up to 80 Hz, with visualizations rendered in real time on portable devices.

Figure 8: Head-mounted device setup: event camera, embedded processing, portable power system.

Figure 9: Real-time viewer displaying live event stream, RGB reference, and predicted 3D poses.

Implications and Future Directions

E-3DPSM substantially advances egocentric event-based 3D human pose estimation, delivering robust, real-time performance suitable for VR/AR, telepresence, and wearable interaction. The continuous state evolution model and bidirectional SSM context represent a general paradigm shift for event-based vision, showing superior resilience to occlusion, temporal sparsity, and rapid motion.

Practical implications include more reliable low-power and bandwidth-efficient pose tracking for next-generation wearable devices. Theoretically, the modular state-machine architecture opens avenues for more complex scene understanding, multi-person interaction, and adaptive modeling of physical uncertainty in event-driven perception.

Future developments may focus on explicit occlusion modeling, generative refinement of ambiguous poses, context-aware fusion with other modalities (RGB, IMU), and further reduction of computational requirements for embedded real-time applications.

Conclusion

E-3DPSM provides a principled, event-driven, and temporally coherent approach to egocentric 3D human pose estimation from event streams. Through continuous pose state evolution, adaptive neural fusion, and rigorous temporal context modeling, it sets a new reference for accuracy and stability in event-camera-based applications. The demonstrated improvements for occluded and distal joints suggest strong potential for both practical deployment and methodological extension in dynamic vision and wearable AI.