MobileEgo Anywhere

• MobileEgo Anywhere is an infrastructure for capturing, processing, and utilizing long-duration egocentric video and multimodal sensor data on commodity devices.
• It integrates ARKit/ARCore-based 6 DoF tracking with LiDAR and IMU sensor fusion, enabling precise pose estimation and drift mitigation over extended recordings.
• The open-source suite facilitates scalable research in vision-language-action tasks by offering accessible data collection, automatic language annotations, and reproducible pipelines.

MobileEgo Anywhere is an end-to-end infrastructure designed for capturing, processing, and utilizing long-horizon egocentric video and multimodal sensor data using only commodity mobile hardware. It encompasses a head-worn mobile application, a large-scale 6 Degree-of-Freedom (DoF) egocentric dataset, and an open-source Python processing suite. The system provides robust, hour-plus, 6 DoF-registered first-person trajectories, removing the traditional barriers of specialized hardware for large-scale data collection in robotics and vision-language-action (VLA) tasks (Palanisamy et al., 7 May 2026).

1. System Architecture and Data Capture

MobileEgo Anywhere employs a single commodity device, typically an ARKit-enabled iPhone (such as the 11 Pro or later), rigidly mounted to a helmet or head strap. The mobile device simultaneously streams and records RGB frames, per-pixel LiDAR depth maps, IMU data (accelerometer and gyroscope), magnetometer, GPS (for outdoor settings), camera intrinsics, and 6 DoF camera poses calculated via ARKit's visual-inertial odometry (VIO). Data is stored in an open-source, timestamp-synchronous MCAP log format.

The design is hardware-agnostic, running on any suitable ARKit (iOS, with LiDAR) or ARCore (Android, RGB + IMU, LiDAR optional) device, requiring only basic head mounting and no external rigs or robotics interfaces. The application supports hands-free voice commands for controlling recording sessions and provides real-time feedback on tracking quality and battery/storage status (Palanisamy et al., 7 May 2026).

Pose Tracking and Drift Mitigation

Long-term camera pose is tracked using ARKit's tightly-coupled VIO, additionally fused with LiDAR depth to constrain drift to less than 1 cm over 100 m trajectories. A lightweight loop-closure mechanism with sparse ArUco markers enables sub-millimeter drift correction over hour-long sessions. Every captured frame is consistently anchored in a persistent world-frame, maintaining global scene coherence (Palanisamy et al., 7 May 2026).

2. Dataset Composition and Structure

MobileEgo Anywhere introduces a dataset comprising 200 hours of egocentric video, recorded in 354 continuous sessions by 16 contributors. Scenarios cover a diversity of environments—kitchens, living rooms, garages, parks, and shops—and activities including cooking, cleaning, DIY and outdoor errands. The mean session length is 21.2 minutes, with the longest recording reaching 108 minutes (Palanisamy et al., 7 May 2026).

Each timestamped frame records the following:

• RGB image (1920×1440 at 15 fps)
• LiDAR depth map (256×192 at 15 fps)
• Camera intrinsics ($K$)
• Camera extrinsics ($(R_i, t_i)$) in the world frame
• IMU measurements ($\alpha_i$, $\omega_i$) at 100–200 Hz
• Global GPS fix for outdoor sessions

The dataset provides comprehensive action annotations, with 45,415 atomic action spans (≈8 words/span) and a three-level hierarchical episode breakdown: 5,570 episodes (mean ≈42 s), 1,298 sub-goals (mean ≈3.9 min), and one top-level session goal (mean ≈15.5 min). Contributor-based splitting (70% train, 15% validation, 15% test) prevents user overlap across partitions (Palanisamy et al., 7 May 2026).

Data Formats

• Raw logs: MCAP (binary, synchronized)
• Metadata: Per-session JSON, including intrinsics, extrinsics, and split assignment
• Processed features: HDF5 (image/depth), NPZ (hand trajectories), and plain text (captions, instruction trees)

3. Mobile Application and User Workflow

The recording application is open-source and operates on both iOS and Android platforms. Core features are tailored for robust, user-friendly, prolonged egocentric capture:

• Voice commands ("start", "stop") for hands-free operation
• Real-time preview of tracking quality and resource status
• Automatic segmentation if battery drops below 15% or storage nears capacity
• Secure local encryption of session logs
• Optional Wi-Fi upload functionality for centralized data collection

A standardized workflow comprises mounting the phone, initiating recording via voice, hands-free activity, and eventual upload of the encrypted session log (Palanisamy et al., 7 May 2026). Sensor synchronization is maintained via a high-resolution system clock, and typical storage requirements are 2 GB/hour—permitting extensive offline recording on devices with ≥128 GB storage.

4. Data Processing Pipeline: From Raw Streams to Training

The Python processing suite (installable via pip as fpvlabs) performs a multi-stage transformation:

Step 1: MCAP Parsing

• Demultiplex and decode RGB, LiDAR, IMU, and pose streams.

Step 2: Intrinsic Calibration Refinement

• Multi-view checkerboard calibration using the Zhang method estimates $K$ and lens distortion ($k_1$, $k_2$).

Step 3: Pose Graph Optimization

• Node poses $(R_i, t_i)$ initialized from ARKit.
• IMU pre-integration via iterated Extended Kalman Filter (EKF).
• Depth reprojection enforced by minimizing

$\sum_{i,j} \| u_{ij} - \pi(K, R_i, t_i, X_j) \|^2$

where $\pi(K, R, t, X)$ projects 3D world points into the image plane.

Step 4: 3D Hand Trajectory Extraction

• 2D keypoints detected via WiLoR, then unprojected to 3D in the camera frame and transformed to the world frame by $(R_i, t_i)$0.

Step 5: Automatic Language Annotation

• Video is divided into ≈5 s atomic spans.
• Vision-LLMs generate imperative action labels from cropped RGB frames.
• Text grouping yields hierarchical instruction trees.

Sensor fusion is performed using device-calibrated camera-IMU transforms. Propagation follows standard IMU pre-integration equations, and all coordinates conform to an $(R_i, t_i)$1right, $(R_i, t_i)$2down, $(R_i, t_i)$3forward world frame (Palanisamy et al., 7 May 2026).

5. Innovations and Comparative Analysis

MobileEgo Anywhere significantly extends prior datasets in egocentric vision and robotics:

• Long-Horizon Consistency: Enables continuous, hour-plus 6 DoF trajectories with persistent world-frame registration and depth.
• Commodity Hardware: Utilizes ARKit/ARCore smartphones, eliminating the need for gimbals, MoCap studios, or custom robot interfaces.
• Low-Burden Annotations: State and language labels generated at negligible compute cost ($(R_i, t_i)$41.29$ for the full 200 h dataset).
• Comprehensive Data Modalities: Delivers synchronized RGB, depth, IMU, global pose, hand trajectories, and language.

A summarizing table highlights differences with related datasets (Palanisamy et al., 7 May 2026):

Dataset	Duration (h)	6 DoF	Depth	Max Episode	Hand Tracking
Ego4D	3,670	Partial	No	∼20 min	Partial
EPIC-KITCHENS	100	No	No	Short	No
HOI4D / HOT3D / ARCTIC	<10	Yes	Yes	2 min	Proprietary HW
MobileEgo Anywhere	200	Full	Yes	108 min	WiLoR

Compared to alternatives, MobileEgo Anywhere achieves true long-horizon, multi-modal, and hardware-accessible data collection, closing a gap in scalable VLA dataset infrastructure.

6. Research Applications and Reproducibility

MobileEgo Anywhere's open releases include:

• iOS app: https://fpvlabs.ai/app-code
• Python suite: https://fpvlabs.ai/python-package
• Dataset download: https://fpvlabs.ai/data
• Visualization dashboard: https://fpvlabs.ai/dataset-visualization

These tools enable:

• Pretraining and fine-tuning of vision-language-action models requiring long-horizon, state-consistent data
• Hierarchical policy learning for robotic manipulation and household tasks
• Imitation learning using precise 3D hand demonstrations
• Grounded instruction-following experiments

Code for loading, accessing, and preprocessing the dataset is provided, allowing for customizable multimodal batching and point cloud extraction. The open infrastructure ensures transparent reproduction of the entire pipeline, from egocentric recording to standardized training-ready data (Palanisamy et al., 7 May 2026).

7. Impact and Outlook

By democratizing long-duration, 6 DoF egocentric data collection using commodity mobile devices and open-source tools, MobileEgo Anywhere substantially lowers barriers to large-scale VLA research. It offers unprecedented tractability for acquiring diverse, persistent state, and language-annotated data, previously constrained by hardware costs and logistical complexity. The infrastructure's extensibility and accessibility position it as a foundational resource for advancing generalizable robot learning and multimodal AI research (Palanisamy et al., 7 May 2026).