---
title: 'E2E-3M: Egocentric VQA Dataset'
url: https://www.emergentmind.com/topics/e2e-3m-dataset
type: topic
---

# E2E-3M: Egocentric VQA Dataset

The Egocentric2Embodiment Dataset (E2E-3M) is a large-scale, rule-validated corpus of multi-level visual question answering (VQA) instances derived from human first-person videos. Developed as the foundation for constructing egocentric-aware embodied vision–language systems, E2E-3M enables learning physical intelligence—reasoning about state changes, contact-rich interactions, and long-horizon planning—from richly annotated, temporally grounded, and physically coherent supervision extracted from egocentric perception and action [2512.16793].

## 1. Translation Pipeline and Dataset Construction

E2E-3M is generated by the Egocentric2Embodiment translation pipeline, which consists of four sequential stages designed to convert unstructured human egocentric video into actionable VQA supervision for embodied learning:

**1. Data Intake & Pre-processing:**  
Videos are sourced from three major corpora: Ego4D (~1,500 h; household), BuildAI (~700 h; factory), and EgoDex (~300 h; laboratory). Temporal segmentation is performed using fixed-interval, event-driven (e.g., scene change, hand–object contact), or kinematic-aware (hand motion peaks) strategies to produce short, scenario-aware clips $c_i$, each indexed with precise start/end times and metadata, including location type and object inventory.

**2. Schema-Driven Annotation:**  
For each clip $c_i$, a VQA mode $m$ is randomly sampled from the set $\{\text{Temporal}, \text{Spatial}, \text{Attribute}, \text{Mechanics}, \text{Reasoning}, \text{Summary}, \text{Trajectory}\}$. A question template $T_{m,j}$ is instantiated based on domain metadata, while answers $A_i$ are generated by a dedicated VLMAnnotator, yielding provisional tuples $(c_i, m, j, Q_i, A_i)$.

**3. Quality Assurance & Validation:**  
A deterministic rule checker $f_{\text{check}}$ enforces three constraint families:

- **Evidence grounding ($f_g$):** All referenced entities (object, hand, action verb) must be visually present in the selected frames.
- **Egocentric consistency ($f_e$):** Hand designations match observed wrist pose; references to non-visible limbs are disallowed.
- **Temporal logic ($f_t$):** Temporal relations in questions and answers (e.g., “before”) are validated against annotated event timestamps.

Only records passing $f_g \wedge f_e \wedge f_t$ are admitted; otherwise, automatic regeneration occurs with corrective error messaging.

**4. Structured Output:**  
Validated instances are structured records:  
$\{\text{visual clip frames}, m, j, Q_i, A_i, \text{validation\_log}\}$, enabling full traceability to original video context.

## 2. Dataset Scale, Modalities, and Diversity

After complete pipeline execution, E2E-3M comprises approximately 1.2 million temporal clips and 3.0 million uniquely validated VQA pairs, each encoding both visual and linguistic modalities:

- **Visual:** Each clip includes 3–5 RGB frame crops capturing key temporal moments.
- **Language:** Each instance contains a natural-language question $Q_i$ and an answer $A_i$ corresponding to the annotation schema.

The annotation schema encompasses 7 distinct modes, each with its finite template set $\mathcal{T}_m$. Object and verb lexical diversity is quantified as follows:

| Domain (s)      | Object Diversity per 1k Tokens | Verb Diversity per 1k QAs |
|:----------------|:------------------------------:|:------------------------:|
| Household       | 200–400 distinct nouns         | 80–160 verbs             |
| Factory         | 200–400 distinct nouns         | 80–160 verbs             |
| Laboratory      | 200–400 distinct nouns         | 80–160 verbs             |

This ensures broad coverage of physical entities and interactional verbs across diverse egocentric scenarios.

## 3. Schema-Driven, Multi-Level VQA Supervision

Each VQA mode targets a discrete facet of embodied planning or interaction:

- **Temporal:** Ordering of actions and events (e.g., “What did the agent do before placing the cup?”)
- **Spatial:** Egocentric spatial relations (“Where is the onion relative to the towel?”)
- **Attribute:** Object properties perceivable in context (“What color is the tool in the left hand?”)
- **Mechanics:** Contact and manipulation dynamics (“Which hand lifts the lid?”)
- **Reasoning:** Causal or motivational explanations (“Why did the agent push the button?”)
- **Summary:** High-level task progression (“What is the next step?”)
- **Trajectory:** Motion path or region traversal (“Through which region does the slider move?”)

Critically, schema-driven rules restrict language to visible, temporally consistent phenomena, enforcing alignments such as $ \tau(e_1) < \tau(e_2) $ for “before” relations and $ \{\text{hand}, \text{object}\} \subseteq \text{Vis}(c_i) $ for contact events, where $\text{Vis}(c_i)$ is the set of entities annotated as visible in the clip.

## 4. Model Training and Benchmark Evaluation

PhysBrain, the embodied vision–language model, is trained via supervised fine-tuning (SFT) on a balanced mixture of E2E-3M and FineVision data. The SFT objective is standard cross-entropy loss over answer tokens:

$$
\mathcal{L}_{\mathrm{SFT}} = -\mathbb{E}_{(c,Q,A)\sim D_{E}\cup D_{F}} \sum_{t=1}^{|A|} \log p_\phi(a_t \mid a_{<t}, Q, c)
$$

where $D_{E}$ is the E2E-3M corpus, $D_{F}$ is FineVision, and $\phi$ are the model parameters. Ego4D-derived clips are excluded from $D_{E}$ during evaluation on EgoThink to prevent data leakage. Optimization is performed using AdamW, deepspeed ZeRO, and a cosine learning rate schedule.

**Benchmarks:**
- **EgoThink:** Six subtasks (Activity, Forecast, Localization, Object, Planning, Reasoning).
  - Planning: 64.5% for PhysBrain versus 32.0% for Qwen2.5-VL-7B.
  - Average across all subtasks: 64.3% versus 57.3%.
- **SimplerEnv (WidowX):** Four robot manipulation tasks under the PhysGR00T VLA head.
  - Aggregate success rate: 53.9% (with task-level rates: 65.6%, 37.5%, 33.3%, 79.2%).
  - Observed +9% absolute improvement over the next-best VLM-initialized vision–language actor (VLA).

## 5. Downstream Applications and Significance

E2E-3M is the first empirically grounded, large-scale egocentric VQA dataset optimized for injecting planning structure and hand–object interaction semantics into vision–language models (VLMs). Rule-based validation ensures that annotated language aligns precisely with visible phenomena and temporal structure, suppressing hallucinations and facilitating physically meaningful affordance learning.

Fine-tuning on E2E-3M confers several empirical benefits:

- Improved sample efficiency for downstream VLA adaptation.
- Enhanced reliability for long-horizon planning under partial observability.
- Increased robotic manipulation success rates when combined with even limited robot-collected data.

Anticipated future directions include curriculum learning for transfer from human to robotic tasks, hybrid supervision frameworks, and real-world deployment of VLM-based controllers in environments spanning household assistance, industrial assembly, and laboratory automation.

## 6. Context in Embodiment and Vision–Language Research

E2E-3M addresses a fundamental viewpoint mismatch between existing third-person VLM training data and the egocentric nature of robotic perception. By leveraging the scalability and diversity of human first-person video, and enforcing physically grounded, temporally coherent annotation schemas, E2E-3M enables new directions for research in embodied intelligence, bridging the gap between large-scale vision–language pretraining and the practical demands of physical agents in diverse, unstructured settings [2512.16793].

Source: https://www.emergentmind.com/topics/e2e-3m-dataset