EgoLM: Multi-Modal Language Model of Egocentric Motions (2409.18127v1)

Abstract: As the prevalence of wearable devices, learning egocentric motions becomes essential to develop contextual AI. In this work, we present EgoLM, a versatile framework that tracks and understands egocentric motions from multi-modal inputs, e.g., egocentric videos and motion sensors. EgoLM exploits rich contexts for the disambiguation of egomotion tracking and understanding, which are ill-posed under single modality conditions. To facilitate the versatile and multi-modal framework, our key insight is to model the joint distribution of egocentric motions and natural languages using LLMs (LLM). Multi-modal sensor inputs are encoded and projected to the joint latent space of LLMs, and used to prompt motion generation or text generation for egomotion tracking or understanding, respectively. Extensive experiments on large-scale multi-modal human motion dataset validate the effectiveness of EgoLM as a generalist model for universal egocentric learning.

An Overview of EgoLM: Multi-Modal LLM of Egocentric Motions

The paper "EgoLM: Multi-Modal LLM of Egocentric Motions" addresses the essential task of learning egocentric motions using contextual AI frameworks. This research targets the domain of egocentric motion tracking and understanding by integrating multi-modal sensor inputs from wearable devices. The authors propose a versatile framework, EgoLM, which exploits LLMs to capture the rich contextual information inherent in egocentric videos and sparse motion sensors.

Core Contributions

1. Unification of Egocentric Motion Tracking and Understanding: EgoLM combines the tasks of motion tracking and motion understanding from an egocentric perspective. Motion tracking aims to recover full-body motions from sparse motion sensors, while motion understanding seeks to describe human motions using natural language based on the input data from motion sensors and egocentric videos.
2. Innovative Use of Multi-Modal Inputs: To address the challenges inherent in egocentric motion learning, the proposed framework integrates multi-modal sensor inputs, such as three-points (head and wrists) and one-point (head) 6-DoF poses, alongside egocentric videos. This multi-modal data helps disambiguate the context, which often presents ill-posed problems when using single modalities.
3. Joint Learning Through LLMs: A key insight of this work is to leverage LLMs to model the joint distribution of egocentric motions and natural languages. The EgoLM framework encodes multi-modal inputs into a shared latent space aligned with LLMs. This encoded data can then prompt motion generation or text generation, facilitating both tracking and understanding tasks.
4. Extensive Experimental Validation: Utilizing a large-scale multi-modal human motion dataset, Nymeria, the authors perform comprehensive experiments to validate EgoLM. The results demonstrate the model's effectiveness, showing that it outperforms state-of-the-art methods in both egocentric motion tracking and understanding tasks.

Methodological Details

EgoLM's methodology unfolds in three critical steps:

1. Motion Tokenization: The authors initialize by training a VQ-VAE to act as a motion tokenizer. This component discretizes continuous motion data into token sequences compatible with LLMs. The motion data, represented as poses, global translations, and rotations, are encoded into latent features split across multiple codebooks for fine-grained quantization.
2. Motion Pre-training: Leveraging a pre-trained LLM (GPT-2 Medium), the authors conduct motion pre-training wherein the model learns the distribution of human motions from the tokenized motion data. This stage outputs an unconditional motion generator, solidifying the backbone’s capacity for subsequent movements.
3. Multi-Modal Instruction Tuning: In the final step, the framework employs instruction tuning on multi-modal data, guiding the LLM to perform specific tasks. Inputs include sparse motion sensor data and egocentric videos, which are encoded into the LLM’s feature space using pre-trained vision encoders and custom-designed temporal encoders. Training involves multiple tasks, including motion tracking from sensor inputs and egocentric videos, and understanding motions from combined modalities.

Experimental Highlights

The quantitative results emphasize EgoLM's strong performance across several metrics. Key results include:

• Motion Tracking: The inclusion of egocentric videos significantly improves tracking accuracy. For instance, incorporating video inputs reduces full-body joint position errors by approximately 10mm in the three-points tracking scenario and 20mm in the one-point setting.
• Motion Understanding: Using metrics such as BERT, BLEU, and ROUGE, EgoLM demonstrates superior performance in generating accurate narrations of human motions. The combination of three-points data and egocentric videos yields the highest scores in understanding tasks, highlighting the model's contextual grasp.

Implications and Future Directions

The implications of EgoLM are multifaceted:

• Practical Applications: The enhanced accuracy in tracking and understanding egocentric motions could significantly benefit applications in VR/AR, personal assistants, and context-aware AI systems, improving user-agent interactions.
• Methodological Use: The introduced method of leveraging LLMs in multi-modal sensor data fusion can be extended to other domains requiring context-rich understanding and generation tasks, such as robotics and autonomous driving.

Going forward, researchers might investigate augmenting the LLM backbone with more complex and finely tuned architectures, exploring scalability with larger models or alternative network designs. Additionally, addressing the limitations related to motion reconstruction errors and improving the granularity of information extracted from egocentric videos could drive further advancements in egocentric motion learning.

In summary, EgoLM sets a robust foundation in exploring the fusion of multi-modal sensor data within the framework of LLMs, advancing the field of egocentric AI in significant ways.