EgoInertia-MI: A Multimodal Egocentric Vision and IMU Benchmark for Motor Impairment Assessment

Abstract: Motor impairments, including tremor, bradykinesia, gait abnormalities, and postural instability, are common across many neurological and movement-related conditions. Conventional clinical assessments are often intermittent and may fail to capture subtle temporal variations in motor behavior. While wearable IMUs and third-person video have shown promise for objective motor assessment, third-person recordings raise privacy concerns and require constrained acquisition setups. In contrast, egocentric vision provides a more naturalistic and privacyaware alternative. In this work, we introduce EgoInertia-MI, a multimodal benchmark dataset combining synchronized egocentric video and wearable IMU signals for motor impairment analysis. The dataset contains 19 upper- and lower-body activities performed by healthy volunteers simulating varying levels of motor impairment severity levels: no impairment, mild impairment, and severe impairment. We establish two benchmark tasks: action recognition and motor impairment severity estimation, and evaluate multiple unimodal and multimodal baselines. Experimental results show that egocentric video provides strong cues for motor impairment assessment, while multimodal fusion achieves the best overall performance, reaching 0.78 Macro-F1 for severity estimation and 0.93 Macro-F1 for action recognition. These findings highlight the potential of combining egocentric vision and wearable sensing for ecologically valid and privacy-aware motor assessment. Code and data are available at:https://fatemah-alh.github.io/EgoInertia-MI-Page/.

• The paper introduces a multimodal dataset combining egocentric video and IMU data for nuanced motor impairment assessment.
• The paper details rigorous synchronization of video and IMU signals, enabling robust severity estimation and action recognition through fusion and specialized architectures.
• The paper demonstrates that multimodal fusion significantly improves macro-F1 performance, offering a privacy-aware, ecologically valid approach to movement analysis.

EgoInertia-MI: A Multimodal Benchmark for Egocentric Vision and IMU-Based Motor Impairment Assessment

Introduction and Motivation

EgoInertia-MI addresses a critical gap in quantitative motor impairment analysis by introducing a multimodal benchmark dataset encompassing synchronized egocentric video and wearable IMU signals. Traditional clinical assessment protocols often miss nuanced variations in patients’ motor behavior due to their intermittent nature and reliance on constrained environments. While third-person vision and wearable IMUs provide solutions, they suffer from privacy, ecological validity, and viewpoint limitations. By focusing on egocentric perspectives aligned with wearable inertial data, EgoInertia-MI establishes a privacy-conscious, naturalistic, and context-rich data resource conducive to robust movement analysis, particularly for healthcare settings.

Dataset Composition and Collection Protocol

EgoInertia-MI comprises roughly 1,150 multimodal segments from 17 healthy adult volunteers, collected under ethically approved protocols. Participants performed 19 upper- and lower-body activities, spanning a spectrum from standardized clinical assessments (e.g., finger tapping, TUG test) to daily living activities, each executed at three predefined severity levels: none, mild, and severe impairment. Simulated impairment enables systematic exploration of movement variability.

Wearable sensing leveraged Respeck IMUs on wrist and shin (25.5 Hz, capturing tri-axial accelerometer/gyroscope), as well as a chest-mounted GoPro (4K, 30 FPS, with extracted IMU traces). Synchronization was achieved via timestamp anchoring and manual alignment. Each data segment is annotated for activity, severity, handedness, and sensor configuration, supporting robust multimodal learning and subject-wise cross-validation for generalization assessment.

Benchmark Tasks and Models

EgoInertia-MI defines two main tasks:

• Motor Impairment Severity Estimation: Multi-class classification (three levels) to predict impairment severity from either unimodal or multimodal input.
• Action Recognition: Classification of 19 diverse activities executed under varying impairment conditions.

To accommodate different deployment scenarios, the benchmark evaluates unimodal (IMU, video) and multimodal (fusion) models. IMU-based architectures include CNN, LSTM, TCN, DeepConvLSTM, and the HARTransformer, modeling temporal dynamics and capturing biomechanical patterns. Video-based approaches utilize X3D, SlowFast, and V-JEPA models, evaluated in both frozen and fine-tuned configurations. Fusion is examined through late and cross-attention mechanisms, leveraging embeddings from both sensory branches to facilitate rich cross-modal integration.

Quantitative Results and Analysis

The results establish several key findings:

• Egocentric Video Encodes Rich Action and Impairment Cues: Fine-tuned video models, particularly SlowFast and X3D variants, outperform IMU-only models in both severity estimation (Macro-F1 up to 0.75) and action recognition (Macro-F1 up to 0.93).
• Multimodal Fusion Achieves Superior Performance: Late fusion improves severity estimation Macro-F1 to 0.78 and action recognition Macro-F1 to 0.93, underscoring the complementary strengths of spatial-contextual (video) and low-level dynamic (IMU) data.
• IMU-Based Models Remain Viable: Especially without video model fine-tuning, IMU architectures deliver competitive severity estimates (Macro-F1 ~0.69), validating inertial approach utility for low-cost, privacy-sensitive deployment.
• Sensor Placement Impact: Wrist and chest IMUs deliver more discriminative features than leg-mounted sensors for both tasks, highlighting modality-task alignment and simulation protocol limitations when mimicking impairment in healthy subjects.
• Residual Confusions: Analysis reveals challenges in distinguishing closely related actions and adjacent severity grades, reflecting inherent ambiguities at fine-grained motion boundaries.

Practical and Theoretical Implications

EgoInertia-MI sets a new baseline for ecologically valid, privacy-aware motor evaluation. Practically, it demonstrates that integrating egocentric vision with wearable IMUs enables more accurate action and impairment severity inference across unconstrained activities—a prerequisite for deploying AI-driven monitoring outside controlled laboratory or clinic environments. The strong performance of multimodal late fusion, especially in recognizing impairment intensity, suggests that future healthcare solutions should not be modality-exclusive but instead dynamically leverage contextually available data streams for robustness.

Theoretically, the dataset's structured variation in activity and impairment enables deeper probing of representation learning under cross-modal and temporal uncertainty. Its subject-disjoint split protocol and the absence of real patient-pathology data encourage further work in transfer learning, self-supervised methods, and real-world clinical cohort generalization. The simulation of impairments, while necessary for initial benchmarking, also points to a clear next step: extension to real neurodegenerative and movement disorder populations for authentic validation.

Study Limitations and Future Directions

The main limitation is reliance on simulated impairment by healthy volunteers, which, while controllable and reproducible, cannot fully capture the complex, multifactorial nature of real-world motor pathology (e.g., effects of rigidity, compensatory strategies, pharmacological states). Severity levels are protocol-defined and not directly mapped to established clinical scales, narrowing current clinical interpretability. The data also primarily reflects movement variations achievable by instruction, not genuine disability.

Future directions should prioritize clinical cohort data acquisition, deployment in naturalistic home and community settings, and exploration of self-supervised, domain-adaptive, and generative modeling paradigms to enhance transferability and diagnostic precision.

Conclusion

EgoInertia-MI provides the first public benchmark for simultaneous egocentric vision and IMU-based motor impairment analysis, emphasizing the feasibility and value of multimodal learning in movement assessment. The dataset, with its depth, annotation granularity, and methodological rigor, catalyzes research at the intersection of AI, wearable sensing, and healthcare. It lays the groundwork for the development of privacy-preserving, context-aware, and clinically relevant movement analytics across domains ranging from telemedicine to assisted living and beyond.