EgoRe-5M: Egocentric QA Dataset

• EgoRe-5M is a 5M-question egocentric QA dataset derived from over 13M video clips, emphasizing hidden intentions and dynamic hand-object interactions.
• The dataset is organized around four axes—short-term perception, long-term causal reasoning, chain-of-thought rationales, and fine-grained spatio-temporal grounding—to ensure comprehensive egocentric supervision.
• Its dual-stage training framework, combining supervised fine-tuning and reinforcement learning, demonstrates significant performance improvements on multiple egocentric reasoning benchmarks.

EgoRe-5M is a 5M-question-answer egocentric QA dataset introduced in the EgoThinker framework to train multimodal LLMs for egocentric reasoning rather than only visible-event recognition. It is constructed from 13M diverse egocentric video clips and organized around four supervision axes—short-term perception, long-term causal reasoning, chain-of-thought (CoT) rationales, and fine-grained spatio-temporal grounding—with multi-minute segments, dense hand-object grounding, and explicit reasoning traces as first-class annotation targets (Pei et al., 27 Oct 2025). The dataset is designed for settings in which the decisive latent variable is the camera wearer’s hidden intention and evolving interaction with objects, rather than a third-person observer’s description of externally visible actions.

1. Problem setting and design rationale

EgoRe-5M is motivated by the claim that standard video QA data mostly teaches visible event understanding from a third-person viewpoint, whereas egocentric video reasoning must infer the behavior of an unobservable agent behind the camera who dynamically shapes the scene. In this formulation, the core difficulty is not merely action recognition, but reconstruction of hidden intentions, fine-grained interactions, and action sequences over long time spans (Pei et al., 27 Oct 2025).

The dataset was created because existing egocentric QA resources, while useful, are described as insufficient for “reasoning” in the specifically egocentric sense. The paper identifies three missing ingredients: explicit causal chains, long-horizon annotations over multi-minute context, and fine-grained grounding of hands and manipulated objects. Since hand-object interaction is treated as the primary visible signal in first-person video, EgoRe-5M makes it a first-class supervision source rather than an auxiliary attribute.

Accordingly, EgoRe-5M is positioned as a unified corpus for egocentric reasoning chat, hand-object grounding, and temporal grounding. Its scope spans immediate object and action perception, temporally extended causal inference, step-by-step rationalization, and localized “where/when” evidence extraction. This design differentiates it from datasets that supervise only classification or captioning.

2. Source corpus and clip filtering

The raw video collection underlying EgoRe-5M is assembled from web video and existing egocentric datasets, yielding a final pool of 13 million egocentric video clips (Pei et al., 27 Oct 2025). The mining process begins with instructional video sources from HowTo100M, especially HTM-AA and Howto-Interlink7M, on the premise that instructional content often contains head-mounted or handheld viewpoints and rich hand-object manipulation. The initial web-scale pool contains 30M clips, with durations ranging from a few seconds to several minutes.

A dedicated ego-vs-exo filtering stage removes third-person footage. The paper trains a binary classifier on manually labeled ego/exo clips using an InternVideo backbone followed by a 2-layer MLP. Reported performance is 92% accuracy and 89% AUC, reducing the pool to 12M clips that exhibit clear first-person motion and perspective.

A second stage performs dynamic interaction filtering. Because not all first-person clips are useful for egocentric reasoning, the pipeline retains clips with visible hands, active objects, limited multi-person interference, sufficient interaction density, and a minimum duration of 2 seconds. This produces 8.7M high-quality web-sourced egocentric clips. These are then merged with Ego4D, EPIC-Kitchens, EgoExoLearn, and EgoExo4D to obtain the final 13M-clips egocentric video collection. The appendix further reports inclusion of 4M clips from Ego4D, 56K frames from EK-Visor / EPIC-Kitchens-related data, 10K L1 clips from EgoExoLearn, 400 samples for RES, and 500 samples from EgoExo4D for RES.

The filtering logic is formalized in the appendix as three rules:

$\text{Filter Clip} \iff N_{hands} > 2$

$$\text{Filter Clip} \iff \sum_{i=1}^{N} N_{objects}^{(i)} < \alpha \times N, \quad \alpha = 0.7$$

$$\text{Keep Clip} \iff \max \text{ hand-center displacement} > 0.1 \times \min(H, W)$$

These constraints operationalize the dataset’s premise that egocentric reasoning should be trained on clips containing manipulations, motion, and relatively unambiguous hand-object structure, rather than on first-person footage in general.

3. Annotation pipeline and dataset structure

EgoRe-5M is automatically generated rather than manually labeled in the traditional sense (Pei et al., 27 Oct 2025). For clips lacking sufficient text, the pipeline uses Videochat2-HD sampled at 1 fps to produce dense captions. These captions serve as the substrate for question generation. The QA and rationale synthesis stages then use DeepSeek-V3 and DeepSeek-R1, depending on the split.

The dataset is partitioned into four complementary splits:

Split	Temporal span	Scale
Short-term	1–10 seconds	2.4M QA pairs
Long-term	15–120 seconds	2.5M QA pairs
CoT	15–200 seconds	50K QA pairs
Fine-grained grounding	temporal + hand-object	10K temporal + 56K hand-object

The short-term split is intended to teach basic egocentric perception. Its question types include object existence, object attribute, object count, object interaction, action description, action reasoning, and background attribute. The appendix reports substantial type-wise coverage, with per-type counts such as 302K, 326K, and 444K, summing to the short-term total.

The long-term split targets extended causal chains and temporal context. Consecutive clips are concatenated into coherent segments and their narrations merged into a single caption. The supervised tasks include action sequence, temporal grounding, object count, action prediction, action summary, and action reasoning. This split produces 2.5M QA pairs and is the primary vehicle for training over multi-step activity trajectories rather than isolated frames.

The CoT split is the highest-fidelity reasoning subset. The pipeline selects clips with dense captions, concatenates them into longer descriptions, prompts DeepSeek-R1 to generate both a question and a step-by-step rationale, and retains only clips suitable for multi-step inference. The resulting 50K high-fidelity QA pairs are the dataset’s main source of detailed reasoning supervision.

The fine-grained grounding split provides explicit localization targets. For hand-object grounding, it uses EK-Visor pixel-level masks for hands and active objects and asks the model to output a normalized bounding box after textual reasoning. For temporal grounding, it uses EgoExoLearn fine-grained temporal annotations and requires a start–end time interval in seconds after reasoning.

A common misconception is that EgoRe-5M is merely a large short-clip QA corpus. The split design contradicts that interpretation: the long-term, CoT, and temporal grounding components are explicitly built around multi-minute segments and long-horizon integration.

4. Grounding interface, quality control, and training-facing formalism

The fine-grained supervision is exposed through a structured output format using the special tokens:

• > ...
• <answer> ... </answer>

This format is directly tied to the reward functions used during reinforcement fine-tuning (Pei et al., 27 Oct 2025). The format reward assigns 1 when the reasoning and answer conform to the required template and 0 otherwise. The IoU reward is task-specific: hand-object grounding uses spatial IoU between predicted and ground-truth boxes, while temporal grounding uses temporal IoU between the predicted interval and ground truth. The paper states that the hand-object reward combines format reward with spatial IoU, and the temporal grounding reward combines format reward with temporal IoU.

The paper also gives the GRPO advantage normalization used to exploit these annotations during reinforcement fine-tuning:

$$A_i= \frac{r_i-\mathrm{mean}(\{r_i\}_{i=1}^N)}{\mathrm{std}(\{r_i\}_{i=1}^N)}$$

This defines the relative quality of sampled responses for a question. The accompanying GRPO objective is described as maximizing advantage-weighted likelihood with a KL penalty to a reference model, following the intended DeepSeek-style update.

Quality control is reported through manual inspection of 500 QA pairs, of which over 95% are judged correct and logically coherent. This is important because EgoRe-5M relies on captioning plus LLM-based synthesis rather than direct human annotation. The dataset therefore occupies an intermediate regime: automatically generated at scale, but explicitly checked for logical consistency.

5. Role in EgoThinker’s two-stage curriculum

EgoRe-5M is used in a two-stage learning curriculum consisting of supervised fine-tuning (SFT) followed by reinforcement fine-tuning (RFT) (Pei et al., 27 Oct 2025). In Stage 1, the model is SFT-trained on a balanced mixture of general captions, general VQA, ego-related datasets, and EgoRe-5M’s short-term, long-term, and CoT splits. The paper states that the full SFT mixture contains 1.5M samples. It also notes an internal accounting difference: the main table attributes 860K of these to EgoRe-5M, while the appendix specifies 410K short-term, 400K long-term, and 50K CoT, totaling 810K EgoRe-5M samples.

This SFT phase is intended to instill object perception, causal inference, long-horizon context integration, and reasoning patterns. Stage 2 then performs RFT on EgoRe-5M-FG using GRPO, with hand-object grounding trained first and temporal grounding trained second. The stated purpose is to sharpen localization while preserving the reasoning competence acquired in SFT.

The curriculum reflects a division of labor within the dataset itself. The short-term, long-term, and CoT splits supervise semantic and causal reasoning, while the fine-grained split supplies explicit spatio-temporal localization pressure. This suggests a deliberate decomposition of egocentric reasoning into a semantic stage and a localization stage, though the paper presents both within a single unified framework.

6. Empirical impact and position in the broader egocentric landscape

The paper attributes substantial downstream gains to EgoRe-5M and the associated two-stage curriculum (Pei et al., 27 Oct 2025). On egocentric QA and reasoning benchmarks, EgoThinker is reported to set new SOTA on EgoTaskQA, QAEgo4D, ERQA, EgoPlan, EgoSchema, EgoMCQ, VLN-QA, and RES. Relative to Qwen2-VL-7B, the reported gains include +4.4% on EgoPlan, +3.4% on EgoSchema, and +8.0% on VLN-QA. On RES, the paper reports 39.5 for EgoThinker versus 26.3 for Qwen2-VL and states that EgoThinker beats the second best by 8.4%.

On the grounding benchmarks, the effect of the fine-grained split is especially pronounced. On EK-Visor, the base model reports 28.6 mIoU / 64.5 Loc-Acc, Stage 1 SFT on EgoRe-5M-FG reports 38.9 / 74.1, and Stage 2 with RFT reaches 53.7 mIoU / 80.3 Loc-Acc. On EgoExoLearn, EgoThinker reports 25.2 mIoU / 63.9 [email protected]. The paper interprets this as evidence that RFT produces the major jump in grounding performance. At the same time, general video capability is reported as preserved: MVBench 70.0 vs 68.2 for Qwen2-VL, Perception Test 72.7 vs 70.3, and VideoMME 62.9, matching the baseline.

Within the broader egocentric research ecosystem, EgoRe-5M occupies the reasoning-and-grounding stratum rather than the full space of first-person understanding. Adjacent work on egoEMOTION argues that existing egocentric benchmarks such as EPIC-KITCHENS, Ego4D, Ego-Exo4D, and Nymeria largely model external behavior while ignoring emotion and personality, and introduces synchronized self-reported affect, personality, and physiological signals for egocentric perception (Jammot et al., 25 Oct 2025). A plausible implication is that EgoRe-5M supplies large-scale supervision for causal reasoning and grounding, whereas egoEMOTION addresses the missing internal state of the wearer.

Similarly, work on EgoLoc-v1 shows that robust geometry remains a separate challenge: adding camera relocalization via 2D–3D matching to SfM improves VQ3D performance from 87.12 to 88.64 in overall success rate and from 90.53 to 92.05 in QwP, indicating that egocentric reasoning over text and hand-object evidence does not replace the need for strong camera-pose estimation in 3D localization tasks (Mai et al., 2024). In that sense, EgoRe-5M is best understood not as a universal egocentric benchmark, but as a large-scale training substrate for the specific problem of spatio-temporal, causal, first-person reasoning.