Egocentric2Embodiment Dataset (E2E-3M)

• The dataset transforms first-person videos into hierarchical VQA pairs to support embodied planning and physical interaction modeling.
• A four-stage, schema-driven translation pipeline ensures robust, deterministic, and temporally consistent supervision across diverse egocentric scenarios.
• PhysBrain training on E2E-3M demonstrates state-of-the-art performance in planning and action prediction, enhancing robotic vision–language capabilities.

The Egocentric2Embodiment (E2E-3M) dataset is a large-scale, structured corpus of egocentric video-derived question–answer (VQA) supervision targeting embodied planning, hand–object interaction, and physical intelligence. Developed to address the viewpoint discrepancy between third-person VLM pretraining and the egocentric demands of humanoid robot perception and action, E2E-3M leverages automated translation pipelines to transform raw human first-person video into hierarchical, schema-driven VQA data, with rigorous evidence grounding and temporal consistency. This dataset underpins the training of PhysBrain, an egocentric-aware vision–LLM that demonstrates state-of-the-art performance in egocentric benchmarks and enhances transfer to downstream robot control (Lin et al., 18 Dec 2025).

1. Egocentric2Embodiment Translation Pipeline

The E2E-3M dataset is constructed via a four-stage pipeline designed to convert extensive first-person video into structured, verifiable VQA supervision explicitly aligned with tasks in physical intelligence and embodied planning.

1. Data Intake & Pre-processing: Raw first-person video episodes, each annotated with scene and activity metadata, are temporally segmented into shorter clips using a hybrid of fixed-interval slicing, event-based detection (object motion or hand keypoints), and kinematic-aware analysis (hand–object velocity peaks). Each resultant clip, indexed by $[t_s, t_e]$, retains episode context for downstream supervision.
2. Schema-driven VQA Generation: For each clip, the system samples from seven VQA modes: Temporal, Spatial, Attribute, Mechanics, Reasoning, Summary, and Trajectory. Templated questions are populated, and a small VLM (e.g., GPT variant) yields a natural language answer. Generation enforces domain conventions such as explicit left/right hand references and contact verbs (“grasp,” “slide,” “insert”).
3. Deterministic Rule-Based Validation:

To mitigate LLM hallucinations and enforce factual grounding, each Q/A pair is checked for: - Evidence grounding: All referenced objects and hand contacts must appear in at least one frame. - Egocentric consistency: Hand assignments match pose detections; no spurious limbs. - Temporal logic: Temporal descriptors (“before,” “after,” “then”) must align with video ordering. Failed samples are regenerated iteratively until all criteria are satisfied.

1. Dataset Assembly: Each validated example records clip indices, VQA mode/template, Q/A content, and a “passed” verdict, ensuring traceability and reproducibility.

Two diversity metrics quantify coverage:

$$\mathrm{ObjectDiv}(s) = \frac{|\mathcal{V}^{\text{noun}_s}|}{T^{\text{noun}_s}} \times 1000$$

$$\mathrm{VerbDiv}(m) = \frac{|\mathcal{V}^{\text{verb}_m}|}{N_m} \times 1000$$

where $\mathcal{V}^{\rm noun}_s$ and $\mathcal{V}^{\rm verb}_m$ count unique noun or verb lemmas, $T^{\rm noun}_s$ is total noun tokens, and $N_m$ is total Q/A pairs in mode $m$.

2. Dataset Characteristics

E2E-3M is constructed from three egocentric video sources:

• Ego4D: ∼3,000 hours, household and open-world scenarios.
• BuildAI: Factory workflows, emphasizing dense hand–tool interactions.
• EgoDex: Laboratory manipulations, high resolution.

The final dataset comprises approximately 3 million rigorously validated VQA pairs spanning thousands of hours. Data modalities include RGB video frames (depth is not included), temporally localized metadata, and human-readable Q/A annotations. Annotation is 100% automated, with deterministic rule-driven filtering; no manual intervention occurs at any stage. The schema features seven VQA modes, each decomposing object interactions (e.g., “cup–spoon contact”) and hierarchical task structures (e.g., sub-action decomposition).

3. PhysBrain Supervised Training

E2E-3M is utilized for supervised fine-tuning (SFT) of visual LLMs, culminating in the PhysBrain model. The regime includes:

• Base Models: Qwen2.5-VL-7B, VST-RL-7B, LLaVA-1.5-7B.
• Data Mixture: 3 million E2E-3M pairs (egocentric VQA), 3 million general vision–language instruction samples (FineVision subset), totaling 6M SFT pairs.
• Objectives: Standard cross-entropy loss over answer tokens,

$$\mathcal{L}_{\rm VQA} = -\sum_{(q,a)} \sum_{t=1}^{|a|} \log P\bigl(a_t \mid q,\,\text{video‐features}\bigr)$$

No auxiliary or contrastive losses are used.

For action prediction in Vision–Language Action (VLA) fine-tuning, a Flow-Matching diffusion objective is adopted: $$\mathcal{L}_{\rm FM} = \mathbb{E}_{\tau,\epsilon} \left[ \left\| f_\theta(\tilde a, \tau; Z_t, s_t) - (a - \epsilon) \right\|_2^2 \right]$$ PhysBrain serves as an egocentric-aware initialization for efficient downstream policy tuning.

4. Evaluation and Empirical Analysis

4.1 Egocentric VLM Benchmarks

On the EgoThink evaluation suite (leakage-free: Ego4D held out), PhysBrain achieves state-of-the-art scores across six subtasks: Activity, Forecast, Localization, Object, Planning, and Reasoning.

Method	Activity	Forecast	Localization	Object	Planning	Reasoning	Average
Qwen2.5-VL-7B	56.5	54.0	71.5	64.7	32.0	60.0	57.3
RoboBrain2.0	36.0	49.5	78.0	61.3	37.0	52.7	53.1
PhysBrain	70.0	53.5	77.0	65.3	64.5	58.0	64.3

The most substantial increase is observed in Planning (32.0 → 64.5), with PhysBrain outperforming even GPT-4 baselines in this dimension.

4.2 Ablation: Spatial Aptitude Training

Fine-tuning VST-RL-7B on E2E-3M (no new SAT data) yields significant gains, particularly for egocentric movement (+65.21 points), overall accuracy (+14.00), and action consequence (+10.81), with some neutral or negative shifts in other categories. This demonstrates E2E-3M’s complementary value for targeted egocentric improvement.

4.3 VLA Simulation in SimplerEnv

In VLA simulation on SimplerEnv (WidowX), PhysBrain→VLA using PhysGR00T architecture yields a 53.9% success rate (mean of five tasks), surpassing the best prior LLM baseline by 8.8 percentage points and RoboBrain2.0 by 16.1 points.

5. Advancements in Embodied Robot Learning

The E2E-3M approach underscores the following principles:

• Large-scale human egocentric video offers naturally scalable supervision aligned with embodied planning and manipulation, circumventing the cost and diversity constraints of robot-only data.
• Multi-mode, template-driven, rule-validated VQA translation produces high-quality, reproducible supervision capable of teaching decomposition, state identification, and causality in planning.
• PhysBrain, trained on E2E-3M, exhibits elevated sample efficiency and VLA success, demonstrating superior transfer to robot control relative to both third-person VLMs and robot-only pretraining.
• Pretraining VLM backbones on egocentric VQA data (E2E-3M or similar) is recommended for vision–language–action pipelines in embodied robots, as this egocentric initialization accelerates and improves downstream policy tuning.
• The pipeline is domain-agnostic and fully automatable, amenable to extension into diverse domains (including outdoor and medical). This suggests a plausible implication for rapid adaptation in future physically intelligent agents.

6. Context and Implications

E2E-3M provides a foundation for bridging the gap between large-scale vision–language representations and the exigencies of robot embodiment. By transforming human-performed, first-person interactions into a rich supervision stream with verifiable, multi-level semantics, it enables VLMs to internalize planning structures required for generalization in physical environments. The dataset's architecture, metrics, and empirical outcomes set benchmarks for rigor and transfer efficacy in the development of physically intelligent, egocentric-aware robots (Lin et al., 18 Dec 2025).