EgoLifeQA: Egocentric Video QA Benchmark

Updated 3 July 2026

EgoLifeQA is an egocentric video QA benchmark that uses multi-modal data from week-long recordings to test long-term memory and causal inference in AI systems.
It comprises core sub-tasks—EntityLog, EventRecall, HabitInsight, RelationMap, and TaskMaster—designed for multi-entity tracking and temporal reasoning over extensive daily activities.
Advanced modeling approaches such as retrieval-augmented generation, dynamic temporal graphs, and compression-based memory have yielded significant performance gains over traditional methods.

EgoLifeQA is an egocentric video question answering (QA) benchmark and reference task suite, grounded in multi-modal, week-long recordings of everyday human life. It fundamentally evaluates an AI system's capacity for long-term memory, temporal reasoning, multi-entity tracking, and causal inference across ultra-long egocentric sequences. By presenting questions that require resolving temporally distant events, identifying personal habits, and integrating entity-centric and causal evidence, EgoLifeQA poses challenges central to the development of practical, lifelong personal AI assistants and embodied systems.

1. Benchmark Definition, Data, and Task Taxonomy

EgoLifeQA originated as the principal evaluation suite for the EgoLife project, which recorded six subjects living together for seven days, each equipped with wearable cameras, audio recorders, and motion sensors (Yang et al., 5 Mar 2025). The resulting dataset encompasses approximately 1,000 hours (≈1 TB) of continuous egocentric video, paired with 500 publicly released question–answer pairs. The full EgoLifeQA question set covers five core sub-tasks:

EntityLog: Object-attribute logging (e.g., "What color was the mug I used yesterday morning?")
EventRecall: Temporal localization (e.g., "When did I first enter the kitchen today?")
HabitInsight: Habitual aggregation across days (e.g., "Which day do I usually water the plants?")
RelationMap: Multi-entity, multi-hop contextual linking (e.g., "Who was I talking to when I turned on the stove?")
TaskMaster: Causal and temporal chaining ("Why did I bring an umbrella on Tuesday afternoon?")

These categories reflect core requirements for long-term, context-aware assistants: temporal grounding ("first/last time"), cross-day aggregation ("usually"), causal inference, and multi-entity chaining (Sun et al., 27 Feb 2026, Rege et al., 26 Jan 2026).

2. Dataset Construction and Annotation Protocol

Data was collected in a controlled real-world environment, with each participant wearing Meta Aria AI glasses for egocentric capture, supplemented by 15 synchronized GoPro cameras for multi-view reference and mmWave radars for room-scale motion tracking (Yang et al., 5 Mar 2025). Annotation proceeded in multiple stages:

Video Segmentation: The seven-day footage was chunked into ~5-minute clips, yielding several thousand segments per participant.
Dense Captioning: Human annotators provided hundreds of thousands of brief narrations (2.65 sec. avg.) per clip. Merged by GPT-4o-mini into coarse-grained visual-audio captions.
ASR & Diarization: Transcripts obtained by ASR (Whisper), diarized and manually refined for speaker identity and timing.
QA Curation: GPT-4o was used to seed tens of thousands of candidate questions per participant, which were then filtered for real-world relevance, minimum 5-minute look-back, and diversity, producing 500 validated multiple-choice QAs per individual (Yang et al., 5 Mar 2025).

Each question is paired with four distractors, is timestamped, and flagged for necessary modalities (e.g., "audio needed?"). Certificate-length annotation bins classify the temporal retrieval horizon required for evidence (<2h, 2–6h, 6–24h, >24h).

3. Formal Task Formulation and Evaluation Metrics

The EgoLifeQA task is formalized as follows:

Input: Stream $\mathcal{X} = \{(F_i, A_i, T_i)\}_{i=1}^N$ where $F_i$ are frame sets, $A_i$ are audio segments, $T_i$ are transcript tokens, segmented into $N$ contiguous clips.
Question: A natural-language query $q$ , multiple-choice with candidate set $\{a_1, ..., a_M\}$ .
Objective: The system selects $\hat a(q) = \arg\max_{a_j} P(a_j | q, \mathcal{X})$ .
Metric: QA accuracy, both overall and per sub-task: $|\{\hat a(q) = a^*(q)\}| / |\mathcal{Q}|$ .
Temporal breakdown: Certificate-length accuracy subdivides queries by temporal reason span (Yang et al., 5 Mar 2025, Sun et al., 27 Feb 2026).

No explicit train/validation/test splits are mandated; the 500 QAs are a public test set. Zero-shot evaluation is standard, prohibiting finetuning or access to gold answers (Sun et al., 27 Feb 2026).

4. Modeling Approaches and System Architectures

A wide spectrum of modeling paradigms have been applied to EgoLifeQA:

Retrieval-Augmented Generation (RAG): Systems like EgoRAG construct a memory bank of visual-audio captions; at query time, captions most relevant to the question are retrieved via embedding similarity and passed to an LLM (e.g., GPT-4o) for answer generation (Yang et al., 5 Mar 2025).
Temporal Knowledge Graphs: EgoGraph constructs a dynamic graph $G=(V,E)$ , with nodes $F_i$ 0 representing Persons, Locations, Objects, or Events. Edges $F_i$ 1 encode temporally anchored relations. Temporal filtering and memory roll-up mechanisms support efficient, temporally scoped reasoning (Sun et al., 27 Feb 2026).
Scene Graphs + Agentic Planning: EGAgent integrates visual, audio, and entity scene graph tools via an LLM planner, orchestrating multi-hop, cross-modal reasoning to tackle compositional queries (e.g., combining kitchen presence with conversation logs) (Rege et al., 26 Jan 2026).
Compression-Based Memory: Imprint applies online memory compression, structuring all interactions as five-tuples $F_i$ 2, weighting their retention by frequency, recency, and distinctiveness, resulting in a highly compact and retrieval-effective memory store (Das et al., 1 Jul 2026).
Multimodal LLMs and Pointer Mechanisms: Recent systems utilize memory pointer architectures to select and ground relevant frames, improving retrieval and evidence localization for long-context questions (Ye et al., 2024).

All approaches converge on an explicit memory representation (episodic summary, knowledge-graph, or interaction-centric), a retrieval stage, and an LLM (or equivalent) reasoning module.

5. Comparative Performance and Empirical Findings

EgoLifeQA benchmarks not just retrieval but full pipeline efficacy—memory formation, targeted retrieval, and complex LLM reasoning. Key results include:

Model	EntityLog	EventRecall	HabitInsight	RelationMap	TaskMaster	Avg Acc.
Gemini-1.5-Pro	36.0	37.3	45.9	30.4	34.9	36.9
GPT-4o	34.4	42.1	29.5	30.4	44.4	36.2
LightRAG	40.8	40.4	36.0	32.0	50.7	39.2
EgoGraph	46.4	46.8	45.9	35.2	60.3	45.8
EGAgent (SOTA 2026)	—	—	—	74.4*	66.7*	57.5

*EGAgent achieves substantial gains on RelationMap (+20.8%) and TaskMaster (+22.2%) over non-agentic baselines (Rege et al., 26 Jan 2026).

Ablation studies show +2.2 pts from an egocentric schema, +4.4 pts from temporal filtering, with the full EgoGraph pipeline yielding a +6.6 pt gain over static-graph baselines (Sun et al., 27 Feb 2026).
Imprint raises evidence-grounded QA from 10.8% to 64.8%, achieves 2.3× memory compression, and 11.8× retrieval speedup over traditional RAG, underscoring the centrality of interaction-centric memory for multi-day QA (Das et al., 1 Jul 2026).
Temporal robustness: EgoGraph and EGAgent maintain performance as query time increases, while clip-based and plain-text models degrade below 10% for day-long horizons (Sun et al., 27 Feb 2026, Rege et al., 26 Jan 2026).
Even state-of-the-art LLMs (e.g., GPT-5, Gemini-2.5 Pro) remain 20–40 points below human annotator accuracy in individualized, long-span personalized QA scenarios (Xiao et al., 2 Apr 2026).

6. Integration with, and Extensions Beyond, Prior Benchmarks

EgoLifeQA is both informed by and extends previous egocentric benchmarks:

TeleEgo introduces a streaming, real-time, multimodal evaluation protocol with real-time accuracy and memory persistence time metrics for tasks across work, lifestyle, social, and cultural domains (Yan et al., 28 Oct 2025). Its architecture—timestamp-aligned memory, cross-modal grounding, and differentiable retrieval—directly informs EgoLifeQA design.
MM-Ego and LifelongMemory focus on short (≤1h) to moderate (≤20h) context windows, utilizing debiasing schemes and pointer-based retrieval, but lack EgoLifeQA’s multi-day, multi-entity, and causal chaining complexity (Ye et al., 2024, Wang et al., 2023).
MyEgo targets ego-grounding (“my…” questions) but exposes systematic weaknesses in MLLMs tracking “me” and my past across long video, especially when distractor identities and ambiguous referents are present (Xiao et al., 2 Apr 2026).
EgoLifeQA tasks routinely require evidence look-back beyond 24 hours, exceeding the temporal demands of previous datasets (Yang et al., 5 Mar 2025).

7. Challenges, Limitations, and Future Directions

EgoLifeQA highlights significant unsolved research challenges:

Speech and Audio Understanding: Emotion and prosody in diarized transcripts are poorly handled (Yang et al., 5 Mar 2025).
Identity and Habit Tracking: Current systems overfit early personalization data or lose context across days; multi-modal or “virtual user ID” representations are needed (Xiao et al., 2 Apr 2026).
Retrieval and Reasoning Decoupling: Most systems separate retrieval from final LLM reasoning, with little support for iterative, multi-hop, or chain-of-thought retrieval (Yang et al., 5 Mar 2025, Das et al., 1 Jul 2026).
Compression vs. Expressivity: Aggressive temporal summarization may erase crucial evidence for multi-hop or causal queries (Das et al., 1 Jul 2026).
Human Performance Gap: The top models lag 20–50 points behind human QA accuracy on person-specific and long-horizon queries (Xiao et al., 2 Apr 2026).

Planned directions include end-to-end retriever–reader training, hierarchical or federated memory stores, robust multi-entity tracking, integration of gesture/gaze/IMU cues, and scaling to multi-language, multi-environment deployments (Yang et al., 5 Mar 2025, Sun et al., 27 Feb 2026, Das et al., 1 Jul 2026).

EgoLifeQA constitutes a rigorous, multi-faceted testbed for evaluating agents’ persistent, retrieval-oriented memory, complex temporal reasoning, and causal inference in lifelog-scale first-person data. Its benchmarks and analysis pipeline have catalyzed architectures grounded in structured memory, entity-aware scene graphs, online behavioral compression, and contextually filtered reasoning, providing a foundation for the next generation of embodied and assistive AI (Yang et al., 5 Mar 2025, Sun et al., 27 Feb 2026, Rege et al., 26 Jan 2026, Das et al., 1 Jul 2026).