---
title: Ego4D Episodic Memory Challenge
url: https://www.emergentmind.com/topics/ego4d-episodic-memory-challenge
type: topic
---

# Ego4D Episodic Memory Challenge

The Ego4D Episodic Memory Challenge is the past-oriented component of the Ego4D benchmark suite, centered on retrieving or localizing previously observed events, objects, and moments from long egocentric video. In the original Ego4D benchmark, episodic memory comprises Visual Queries (VQ), Natural Language Queries (NLQ), and Moment Queries (MQ); later challenge reports also focus on GoalStep grounding, in which a procedural step description is localized in a long, untrimmed first-person video [2110.07058][2406.15778]. Across these formulations, the shared problem is to identify a response track or temporal interval that matches a query under long-horizon, first-person, real-world conditions.

## 1. Benchmark setting and motivation

Ego4D frames egocentric video as a substrate for episodic memory in Tulving’s sense: memory for specific lived experiences rather than general facts. The full dataset contains **3,670 hours** of egocentric video from **931 unique camera wearers**, collected across **74 locations worldwide** and **9 countries**; participants wore cameras for **1–10 hours at a time**. Portions of the dataset include **audio**, **3D scans / meshes**, **gaze**, **stereo**, **IMU**, and **synchronized multi-camera views**. The corpus is also densely narrated, with **3.85 million sentences** and **13.2 sentences/minute on average** [2110.07058].

Within that larger benchmark suite, episodic memory is explicitly about **querying the past**. The benchmark is designed around a wearable or augmented-reality assistant that can index prior egocentric experience and later answer questions such as where an object was last seen, when an event occurred, or which moments instantiate a queried activity [2110.07058]. This emphasis distinguishes episodic memory from present-focused tasks such as hand-object manipulation or social interaction analysis, and from future-focused forecasting tasks.

A recurrent misconception is to equate episodic memory in Ego4D with answer generation alone. The challenge is primarily a localization problem. VQ requires a spatio-temporal response track, NLQ and GoalStep require temporal grounding intervals, and MQ requires retrieval of all temporal segments corresponding to a queried moment category [2110.07058][2605.20818].

## 2. Task taxonomy and formal problem definitions

The original benchmark defines **three query types**. In **Visual Queries**, the input is an egocentric video, a static image crop of an object, and a query time; the system must identify where the object was last seen in the past video. Formally, given video \(\mathcal{V}\), query crop \(v\) for object \(o\), and query frame \(q\), the output is a response track
\[
r = \{r_s, r_{s+1}, \cdots, r_e\},
\]
where each \(r_i\) is a bounding box \((x, y, w, h)\), and if multiple occurrences exist, the benchmark asks for the occurrence that minimizes
\[
q - r_e \quad \text{with } q > r_e.
\]
This makes VQ a last-seen retrieval problem over long egocentric video [2110.07058].

In **Natural Language Queries**, the input is a video \(\mathcal{V}\) and a natural-language question \(\mathcal{Q}\), and the output is a temporal response window \(\mathcal{R} = [t_s, t_e]\). The task includes only episodic queries answerable from the video itself and excludes external-knowledge or intention-reasoning questions. Ego4D describes **13 templates** spanning objects, places, and people, including forms such as “Where is object \(X\) before/after event \(Y\)?”, “What did I put in \(X\)?”, and “Who did I interact with when I did activity \(X\)?” [2110.07058].

In **Moment Queries**, the input is a video and a category label for a high-level activity or moment, and the output is the set
\[
\Phi_c = \left\{ \phi_n = (t_{n,s}, t_{n,e}, s_n) \right\}_{n=1}^N
\]
of all temporal segments instantiating that category, with start time, end time, and confidence. Ego4D organizes MQ around **110 activity categories** derived semi-automatically from narration summaries [2110.07058].

Later challenge reports introduce **GoalStep** as another temporal grounding track over long, untrimmed egocentric videos. In that setting, the query describes a procedural step, and the system must localize the corresponding step grounding interval. Challenge reports from 2024 through 2026 treat NLQ and GoalStep as closely related temporal segment localization problems, and the 2025 report further unifies NLQ, GoalStep, and MQ as variants of precise interval localization in egocentric video [2406.15778][2506.03710].

## 3. Annotations, splits, metrics, and baseline difficulty

The three original episodic-memory tasks are annotated on different subsets of Ego4D. **VQ** covers **433 hours** with **22,602 visual queries** across **54 scenarios**. **NLQ** covers **227 hours** with **19.2k queries** across **34 scenarios**. **MQ** covers **326.4 hours** with **22.2k action instances**, **2,488 clips**, and **110 categories** [2110.07058]. Within each benchmark, train, validation, and test splits are disjoint in video clips; for the public challenge, **test annotations are withheld** and submission is done through a server [2110.07058].

For NLQ, Ego4D reports **11.3k / 3.9k / 4.0k** train/validation/test queries over **136 / 45 / 46 hours**, with **8.3 ± 2.1 words** as the average query length and **9.3 ± 21.5 seconds** as the average response window. NaQ later characterizes the same problem as a “needle-in-a-haystack” localization setting: average video length is **8.2 minutes**, average ground-truth response length is **10.5 seconds**, and the target segment is only about **2%** of the video on average [2110.07058][2301.00746].

The main official metrics are task-specific:

| Track | Query and output | Main metrics |
|---|---|---|
| VQ-2D | Visual crop + query time \(\rightarrow\) response track | tAP, stAP, Succ, rec%, sEff |
| VQ-3D | VQ-2D + 3D localization | RMSE, angular error, succ |
| NLQ | Natural-language query \(\rightarrow\) temporal window | R@1/R@5 at IoU \(0.3, 0.5\) |
| MQ | Moment category \(\rightarrow\) all temporal segments | mAP at tIoU \([0.1:0.1:0.5]\), Recall@kx |
| GoalStep | Procedural step \(\rightarrow\) grounding interval | R@1/R@5 at mIoU \(0.3, 0.5\); Mean R@1 |

The original baselines illustrate the challenge’s difficulty. For VQ-2D, the best reported baseline in the benchmark paper, **Siam-RCNN + KYS + Residual**, obtains **Val Succ 39.8**, **Val tAP 0.12**, **Val stAP 0.04**, **Val rec 32.2**, and **Test Succ 41.6**, **Test tAP 0.12**, **Test stAP 0.05**, **Test rec 34.0**. For NLQ, the benchmark reports **2D-TAN** at **5.80 / 13.90 / 2.34 / 5.96** on test for \(r@1\), \(r@5\), IoU \(0.3\), and IoU \(0.5\), and **VSLNet** at **5.47 / 11.21 / 2.80 / 6.57**. For MQ, the adapted **VSGN** baseline reports **5.68** test mAP [2110.07058].

## 4. Methodological evolution from baseline localization to object-aware and efficiency-aware systems

Challenge work after the initial benchmark diversified along at least four lines: reducing false positives, enlarging supervision, reweighting temporal evidence, and improving efficiency.

For **VQ2D**, “Negative Frames Matter in Egocentric Visual Query 2D Localization” argues that the official detector-tracker baseline is overly prone to false positives on background frames because training mostly exposes the model to clean positive frames while evaluation includes noisy, blurry, or unlabeled background frames. The paper introduces negative proposals from inside the annotated interval with **IoU < 0.5**, from frames outside the object-visibility interval, and from other videos, combined with hard-negative mining and a positive-negative ratio of **1:64**. It reports that the training loop is reduced from roughly **15 days to under 24 hours**, evaluation to around **12 hours**, and the test result reaches **0.17% spatial-temporal AP**, stated as **31% higher than the baseline** [2208.01949].

For **NLQ**, “NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory” repurposes Ego4D narrations as pseudo-query supervision. NaQ creates about **850k NaQ samples** from **4,851 Ego4D training video clips**, or roughly **80× more samples** than the native NLQ training set. The method improves **VSLNet**, **EgoVLP**, and **ReLER**, enables zero-shot and few-shot NLQ, and reports that **ReLER\(^*\) + NaQ** reaches **18.46** at \(R@1\), IoU \(=0.3\), **10.74** at \(R@1\), IoU \(=0.5\), and **14.59** mean \(R@1\), outperforming prior challenge winners from CVPR 2022 and ECCV 2022 [2301.00746].

Another line is **frame-sensitive optimization**. The ReLER/Zhejiang University report “Action Sensitivity Learning for the Ego4D Episodic Memory Challenge 2023” introduces **Action Sensitivity Learning (ASL)**, which assigns learnable sensitivity weights to frames and disentangles them into \(h^{cls}(\bar c)\) and \(h^{loc}(\bar c)\) for classification and localization. On MQ, the submission reports **29.34 mAP** and **48.50 R@1**, ranking **1st place**; on NLQ, it reports **19.79 mean R@1**, ranking **2nd place** [2306.09172].

Efficiency became a distinct research target in **SpotEM**, which observes that existing EM methods spend over **99.9%** of compute on dense clip feature extraction. SpotEM combines a query-conditioned clip selector, low-cost room/object/interaction indexing features, and distillation. On Ego4D NLQ, it reports that computing only **10%–25%** of the clip features preserves about **84%–97%** of the original model’s accuracy, and that many settings retain **95%+** accuracy at about **4× lower compute** [2306.15850].

Object-centric grounding then became more explicit. “ObjectNLQ @ Ego4D Episodic Memory Challenge 2024” augments a GroundNLQ-style temporal localizer with an object branch built from **Co-DETR** pretrained on **LVIS v1.0** and **CLIP ViT-L/14** text encoding of detected object classes. The model reports **mean R@1 = 23.15**, ranking **2nd** on the NLQ challenge, and **R@1 = 33.00 at IoU = 0.3**, ranking **3rd** on GoalStep [2406.15778].

## 5. Unified localization systems and the 2025–2026 winning solutions

A major consolidation occurs in the OSGNet line. “OSGNet @ Ego4D Episodic Memory Challenge 2025” argues that prior unified localization methods often rely on late fusion, which tends to be suboptimal for egocentric grounding. OSGNet instead adopts an early-fusion design with a main branch that fuses video, text, and object features and a shot branch trained with contrastive learning. Videos are divided into **non-overlapping 16-frame snippets**; video features are extracted using **EgoVideo** and **InternVideo**, and text features using **EgoVideo** and **CLIP**. The report states that the method achieved **first place** in the **Natural Language Queries**, **Goal Step**, and **Moment Queries** tracks [2506.03710].

The reported 2025 ensemble results are strong across all three tasks: on **NLQ**, **R@1@0.3 = 30.19** and **R@1@0.5 = 21.78**; on **Goal Step**, **R@1@0.3 = 42.02** and **R@1@0.5 = 32.83**; on **MQ**, **mAP = 36.78** and **R1@0.5 = 54.08** [2506.03710]. The report attributes these gains to early fusion, object-aware and shot-aware modeling, NaQ pretraining, and ensembling.

The 2026 winning report, “OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026,” retains OSGNet as the candidate generator but adds a reranking stage with **GPT-5.4**. For both NLQ and GoalStep, the system keeps the **top-5 candidate segments** from OSGNet and lets the MLLM choose the best match [2605.20818].

For **NLQ**, reranking is done in two stages because GPT-5.4 operates under an image/token limit. Each candidate segment is split into **20-second clips**; frames are sampled at **1 FPS** and converted by GPT-5.4 into textual narrations, which are accumulated into an **episodic memory** for that candidate. GPT-5.4 then compares the episodic memories of all candidates and selects the one that best matches the query [2605.20818].

For **GoalStep**, the same top-5 reranking is combined with a sequential prior over ordered steps \(\{q_i\}_{i=1}^{K}\). Let \(\mathrm{start}_i\) be the start timestamp of the selected segment for query \(q_i\), and \(r_i\) its rank in the localization model’s candidate list. The report defines
\[
\mathrm{penalty}^{\mathrm{start}_i} = \max(0,\, \mathrm{start}_i - \mathrm{start}_{i+1})
\]
and minimizes
\[
\mathrm{cost} = \sum_{i=1}^{K} r_i + \sum_{i=1}^{K-1} \mathrm{penalty}^{\mathrm{start}_i}.
\]
This post-processing prefers candidates ranked highly by OSGNet while encouraging monotonically increasing start-time order [2605.20818].

The 2026 paper reports that the reranking gain is modest in the validation-style tables. On the **NLQ test split**, **OSGNet** reports **R@1@0.3 = 21.63** and **R@1@0.5 = 15.52**, while **OSGNet + rerank** reports **21.78** and **15.44**. On the **GoalStep test split**, **OSGNet** reports **55.31** and **47.82**, while **OSGNet + rerank** reports **55.39** and **47.91**. The authors note that the measured improvement on NLQ may be limited by **false negatives** in the evaluation protocol, because a reranked prediction can be semantically correct yet not match the exact annotated interval, and they state that the sequential-prior post-processing substantially improves the final GoalStep submission. The official GoalStep leaderboard table reports the submission **yisen_feng** in **1st** place with **R@1@0.3 = 63.02**, **R@1@0.5 = 54.21**, **R@5@0.3 = 80.12**, **R@5@0.5 = 74.93**, and **Mean R@1 = 58.61** [2605.20818].

## 6. Related extensions, diagnostic reformulations, and current open directions

Challenge-related research increasingly extends beyond one-shot offline localization. **EMQA** formalizes episodic memory question answering as localization in egocentric pixel space or allocentric top-down map space, using a reusable episodic scene memory built from RGB-D observations and pose. The proposed top-down semantic memory plus temporal augmentation outperforms language-only, frame-by-frame, and compressed-buffer baselines, with **29.11 IoU**, **62.27 Recall**, and **33.39 Precision** in top-down map space for the temporal variant [2205.01652].

A second extension is **online** memory. “Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory” introduces **OEM-VQL**, in which the full video history is not available at query time; the model must instead answer from a compact memory built during streaming observation. The proposed **ESOM** framework stores object tracks and representations online. Under oracle object discovery and tracking, it reports **81.92% Success** versus **55.89% Success** for the best offline baseline, **VQLoC**, while also showing that practical performance is severely limited by egocentric object detection and tracking quality [2411.16934].

A third extension is **interactive refinement**. “Interactive Episodic Memory with User Feedback” introduces **EM-QnF**, where a user can clarify or correct an initial prediction with natural-language feedback. The method adds a plug-and-play **Feedback ALignment Module (FALM)** and reports gains up to about **+4.9 R1** and **+5.4 R5**, while also showing that it generalizes to **500 feedback instances** from **11 users** [2604.24893].

A fourth extension is **diagnostic streaming evaluation**. **EGOSTREAM** organizes **2,250 curated questions** along seven cognitive dimensions—**detail, spatial, temporal, event, social, causal, and prospective memory**—and expands them into **8,528 recall-conditioned evaluations** through the **Answer Validity Window (AVW)**. The benchmark shows that comparable aggregate accuracies can hide very different memory profiles, that token pruning preserves fine-grained details and temporal structure better than token merging, that quantized offloading helps ultra-long-term recall, and that top-performing methods still reach only about **45% accuracy** while operating below real time at **>1s per frame** [2605.31557].

These developments indicate that the Ego4D Episodic Memory Challenge has evolved from a benchmark of offline localization over long first-person video into a broader research program on memory construction, memory management, retrieval, temporal consistency, and interaction. A plausible implication is that future progress will depend not only on stronger localizers, but also on representations that preserve event-critical information under long temporal horizons, on evaluation protocols that better separate genuine failure from annotation mismatch or world-state change, and on systems that can operate under streaming and interactive constraints.

Source: https://www.emergentmind.com/topics/ego4d-episodic-memory-challenge