---
title: 'EgoRe-5M: Egocentric QA Dataset'
url: https://www.emergentmind.com/topics/egore-5m
type: topic
---

# EgoRe-5M: Egocentric QA Dataset

EgoRe-5M is a **5M-question-answer egocentric QA dataset** introduced in the EgoThinker framework to train multimodal large language models for **egocentric reasoning** rather than only visible-event recognition. It is constructed from **13M diverse egocentric video clips** and organized around four supervision axes—**short-term perception**, **long-term causal reasoning**, **chain-of-thought (CoT) rationales**, and **fine-grained spatio-temporal grounding**—with multi-minute segments, dense hand-object grounding, and explicit reasoning traces as first-class annotation targets [2510.23569]. The dataset is designed for settings in which the decisive latent variable is the camera wearer’s hidden intention and evolving interaction with objects, rather than a third-person observer’s description of externally visible actions.

## 1. Problem setting and design rationale

EgoRe-5M is motivated by the claim that **standard video QA data mostly teaches visible event understanding from a third-person viewpoint**, whereas egocentric video reasoning must infer the behavior of an **unobservable agent behind the camera** who dynamically shapes the scene. In this formulation, the core difficulty is not merely action recognition, but reconstruction of **hidden intentions**, **fine-grained interactions**, and **action sequences over long time spans** [2510.23569].

The dataset was created because existing egocentric QA resources, while useful, are described as insufficient for “reasoning” in the specifically egocentric sense. The paper identifies three missing ingredients: **explicit causal chains**, **long-horizon annotations over multi-minute context**, and **fine-grained grounding of hands and manipulated objects**. Since hand-object interaction is treated as the primary visible signal in first-person video, EgoRe-5M makes it a first-class supervision source rather than an auxiliary attribute.

Accordingly, EgoRe-5M is positioned as a unified corpus for **egocentric reasoning chat**, **hand-object grounding**, and **temporal grounding**. Its scope spans immediate object and action perception, temporally extended causal inference, step-by-step rationalization, and localized “where/when” evidence extraction. This design differentiates it from datasets that supervise only classification or captioning.

## 2. Source corpus and clip filtering

The raw video collection underlying EgoRe-5M is assembled from web video and existing egocentric datasets, yielding a final pool of **13 million egocentric video clips** [2510.23569]. The mining process begins with instructional video sources from **HowTo100M**, especially **HTM-AA** and **Howto-Interlink7M**, on the premise that instructional content often contains head-mounted or handheld viewpoints and rich hand-object manipulation. The initial web-scale pool contains **30M clips**, with durations ranging from a few seconds to several minutes.

A dedicated **ego-vs-exo filtering** stage removes third-person footage. The paper trains a binary classifier on manually labeled ego/exo clips using an **InternVideo backbone** followed by a **2-layer MLP**. Reported performance is **92% accuracy** and **89% AUC**, reducing the pool to **12M clips** that exhibit clear first-person motion and perspective.

A second stage performs **dynamic interaction filtering**. Because not all first-person clips are useful for egocentric reasoning, the pipeline retains clips with visible hands, active objects, limited multi-person interference, sufficient interaction density, and a minimum duration of **2 seconds**. This produces **8.7M high-quality web-sourced egocentric clips**. These are then merged with **Ego4D**, **EPIC-Kitchens**, **EgoExoLearn**, and **EgoExo4D** to obtain the final **13M-clips egocentric video collection**. The appendix further reports inclusion of **4M clips from Ego4D**, **56K frames from EK-Visor / EPIC-Kitchens-related data**, **10K L1 clips from EgoExoLearn**, **400 samples for RES**, and **500 samples from EgoExo4D for RES**.

The filtering logic is formalized in the appendix as three rules:

\[
\text{Filter Clip} \iff N_{hands} > 2
\]

\[
\text{Filter Clip} \iff \sum_{i=1}^{N} N_{objects}^{(i)} < \alpha \times N, \quad \alpha = 0.7
\]

\[
\text{Keep Clip} \iff \max \text{ hand-center displacement} > 0.1 \times \min(H, W)
\]

These constraints operationalize the dataset’s premise that egocentric reasoning should be trained on clips containing manipulations, motion, and relatively unambiguous hand-object structure, rather than on first-person footage in general.

## 3. Annotation pipeline and dataset structure

EgoRe-5M is **automatically generated** rather than manually labeled in the traditional sense [2510.23569]. For clips lacking sufficient text, the pipeline uses **Videochat2-HD** sampled at **1 fps** to produce dense captions. These captions serve as the substrate for question generation. The QA and rationale synthesis stages then use **DeepSeek-V3** and **DeepSeek-R1**, depending on the split.

The dataset is partitioned into four complementary splits:

| Split | Temporal span | Scale |
|---|---:|---:|
| Short-term | 1–10 seconds | 2.4M QA pairs |
| Long-term | 15–120 seconds | 2.5M QA pairs |
| CoT | 15–200 seconds | 50K QA pairs |
| Fine-grained grounding | temporal + hand-object | 10K temporal + 56K hand-object |

The **short-term split** is intended to teach basic egocentric perception. Its question types include **object existence**, **object attribute**, **object count**, **object interaction**, **action description**, **action reasoning**, and **background attribute**. The appendix reports substantial type-wise coverage, with per-type counts such as **302K**, **326K**, and **444K**, summing to the short-term total.

The **long-term split** targets extended causal chains and temporal context. Consecutive clips are concatenated into coherent segments and their narrations merged into a single caption. The supervised tasks include **action sequence**, **temporal grounding**, **object count**, **action prediction**, **action summary**, and **action reasoning**. This split produces **2.5M QA pairs** and is the primary vehicle for training over multi-step activity trajectories rather than isolated frames.

The **CoT split** is the highest-fidelity reasoning subset. The pipeline selects clips with dense captions, concatenates them into longer descriptions, prompts **DeepSeek-R1** to generate both a question and a **step-by-step rationale**, and retains only clips suitable for multi-step inference. The resulting **50K high-fidelity QA pairs** are the dataset’s main source of detailed reasoning supervision.

The **fine-grained grounding split** provides explicit localization targets. For **hand-object grounding**, it uses **EK-Visor pixel-level masks** for hands and active objects and asks the model to output a **normalized bounding box** after textual reasoning. For **temporal grounding**, it uses **EgoExoLearn fine-grained temporal annotations** and requires a **start–end time interval in seconds** after reasoning.

A common misconception is that EgoRe-5M is merely a large short-clip QA corpus. The split design contradicts that interpretation: the long-term, CoT, and temporal grounding components are explicitly built around **multi-minute segments** and long-horizon integration.

## 4. Grounding interface, quality control, and training-facing formalism

The fine-grained supervision is exposed through a structured output format using the special tokens:

- `<think> ... </think>`
- `<answer> ... </answer>`

This format is directly tied to the reward functions used during reinforcement fine-tuning [2510.23569]. The **format reward** assigns 1 when the reasoning and answer conform to the required template and 0 otherwise. The **IoU reward** is task-specific: hand-object grounding uses **spatial IoU** between predicted and ground-truth boxes, while temporal grounding uses **temporal IoU** between the predicted interval and ground truth. The paper states that the hand-object reward combines format reward with spatial IoU, and the temporal grounding reward combines format reward with temporal IoU.

The paper also gives the **GRPO advantage normalization** used to exploit these annotations during reinforcement fine-tuning:

\[
A_i= \frac{r_i-\mathrm{mean}(\{r_i\}_{i=1}^N)}{\mathrm{std}(\{r_i\}_{i=1}^N)}
\]

This defines the relative quality of sampled responses for a question. The accompanying GRPO objective is described as maximizing advantage-weighted likelihood with a KL penalty to a reference model, following the intended DeepSeek-style update.

Quality control is reported through manual inspection of **500 QA pairs**, of which **over 95%** are judged correct and logically coherent. This is important because EgoRe-5M relies on captioning plus LLM-based synthesis rather than direct human annotation. The dataset therefore occupies an intermediate regime: automatically generated at scale, but explicitly checked for logical consistency.

## 5. Role in EgoThinker’s two-stage curriculum

EgoRe-5M is used in a **two-stage learning curriculum** consisting of **supervised fine-tuning (SFT)** followed by **reinforcement fine-tuning (RFT)** [2510.23569]. In Stage 1, the model is SFT-trained on a balanced mixture of general captions, general VQA, ego-related datasets, and EgoRe-5M’s short-term, long-term, and CoT splits. The paper states that the full SFT mixture contains **1.5M samples**. It also notes an internal accounting difference: the main table attributes **860K** of these to EgoRe-5M, while the appendix specifies **410K short-term**, **400K long-term**, and **50K CoT**, totaling **810K EgoRe-5M samples**.

This SFT phase is intended to instill **object perception**, **causal inference**, **long-horizon context integration**, and **reasoning patterns**. Stage 2 then performs RFT on **EgoRe-5M-FG** using **GRPO**, with **hand-object grounding trained first** and **temporal grounding trained second**. The stated purpose is to sharpen localization while preserving the reasoning competence acquired in SFT.

The curriculum reflects a division of labor within the dataset itself. The short-term, long-term, and CoT splits supervise semantic and causal reasoning, while the fine-grained split supplies explicit spatio-temporal localization pressure. This suggests a deliberate decomposition of egocentric reasoning into a semantic stage and a localization stage, though the paper presents both within a single unified framework.

## 6. Empirical impact and position in the broader egocentric landscape

The paper attributes substantial downstream gains to EgoRe-5M and the associated two-stage curriculum [2510.23569]. On egocentric QA and reasoning benchmarks, EgoThinker is reported to set new SOTA on **EgoTaskQA**, **QAEgo4D**, **ERQA**, **EgoPlan**, **EgoSchema**, **EgoMCQ**, **VLN-QA**, and **RES**. Relative to **Qwen2-VL-7B**, the reported gains include **+4.4% on EgoPlan**, **+3.4% on EgoSchema**, and **+8.0% on VLN-QA**. On **RES**, the paper reports **39.5** for EgoThinker versus **26.3** for Qwen2-VL and states that EgoThinker beats the second best by **8.4%**.

On the grounding benchmarks, the effect of the fine-grained split is especially pronounced. On **EK-Visor**, the base model reports **28.6 mIoU / 64.5 Loc-Acc**, Stage 1 SFT on EgoRe-5M-FG reports **38.9 / 74.1**, and Stage 2 with RFT reaches **53.7 mIoU / 80.3 Loc-Acc**. On **EgoExoLearn**, EgoThinker reports **25.2 mIoU / 63.9 R1@0.05**. The paper interprets this as evidence that **RFT produces the major jump** in grounding performance. At the same time, general video capability is reported as preserved: **MVBench 70.0 vs 68.2 for Qwen2-VL**, **Perception Test 72.7 vs 70.3**, and **VideoMME 62.9**, matching the baseline.

Within the broader egocentric research ecosystem, EgoRe-5M occupies the reasoning-and-grounding stratum rather than the full space of first-person understanding. Adjacent work on **egoEMOTION** argues that existing egocentric benchmarks such as **EPIC-KITCHENS, Ego4D, Ego-Exo4D, and Nymeria** largely model external behavior while ignoring **emotion and personality**, and introduces synchronized self-reported affect, personality, and physiological signals for egocentric perception [2510.22129]. A plausible implication is that EgoRe-5M supplies large-scale supervision for causal reasoning and grounding, whereas egoEMOTION addresses the missing **internal state of the wearer**.

Similarly, work on **EgoLoc-v1** shows that robust geometry remains a separate challenge: adding **camera relocalization via 2D–3D matching** to SfM improves VQ3D performance from **87.12** to **88.64** in overall success rate and from **90.53** to **92.05** in **QwP**, indicating that egocentric reasoning over text and hand-object evidence does not replace the need for strong camera-pose estimation in 3D localization tasks [2407.08023]. In that sense, EgoRe-5M is best understood not as a universal egocentric benchmark, but as a large-scale training substrate for the specific problem of **spatio-temporal, causal, first-person reasoning**.

Source: https://www.emergentmind.com/topics/egore-5m