Ego4D Episodic Memory Challenge

Updated 4 July 2026

The Ego4D Episodic Memory Challenge is a benchmark focused on retrieving and localizing past events, objects, and moments from long egocentric videos.
It defines three main query types—Visual, Natural Language, and Moment Queries—each with tailored tasks and metrics for precise temporal localization.
Recent methods emphasize efficiency, object-awareness, and multi-modal fusion, advancing practical solutions for real-world episodic memory in video.

The Ego4D Episodic Memory Challenge is the past-oriented component of the Ego4D benchmark suite, centered on retrieving or localizing previously observed events, objects, and moments from long egocentric video. In the original Ego4D benchmark, episodic memory comprises Visual Queries (VQ), Natural Language Queries (NLQ), and Moment Queries (MQ); later challenge reports also focus on GoalStep grounding, in which a procedural step description is localized in a long, untrimmed first-person video (Grauman et al., 2021, Feng et al., 2024). Across these formulations, the shared problem is to identify a response track or temporal interval that matches a query under long-horizon, first-person, real-world conditions.

1. Benchmark setting and motivation

Ego4D frames egocentric video as a substrate for episodic memory in Tulving’s sense: memory for specific lived experiences rather than general facts. The full dataset contains 3,670 hours of egocentric video from 931 unique camera wearers, collected across 74 locations worldwide and 9 countries; participants wore cameras for 1–10 hours at a time. Portions of the dataset include audio, 3D scans / meshes, gaze, stereo, IMU, and synchronized multi-camera views. The corpus is also densely narrated, with 3.85 million sentences and 13.2 sentences/minute on average (Grauman et al., 2021).

Within that larger benchmark suite, episodic memory is explicitly about querying the past. The benchmark is designed around a wearable or augmented-reality assistant that can index prior egocentric experience and later answer questions such as where an object was last seen, when an event occurred, or which moments instantiate a queried activity (Grauman et al., 2021). This emphasis distinguishes episodic memory from present-focused tasks such as hand-object manipulation or social interaction analysis, and from future-focused forecasting tasks.

A recurrent misconception is to equate episodic memory in Ego4D with answer generation alone. The challenge is primarily a localization problem. VQ requires a spatio-temporal response track, NLQ and GoalStep require temporal grounding intervals, and MQ requires retrieval of all temporal segments corresponding to a queried moment category (Grauman et al., 2021, Feng et al., 20 May 2026).

2. Task taxonomy and formal problem definitions

The original benchmark defines three query types. In Visual Queries, the input is an egocentric video, a static image crop of an object, and a query time; the system must identify where the object was last seen in the past video. Formally, given video $\mathcal{V}$ , query crop $v$ for object $o$ , and query frame $q$ , the output is a response track

$r = \{r_s, r_{s+1}, \cdots, r_e\},$

where each $r_i$ is a bounding box $(x, y, w, h)$ , and if multiple occurrences exist, the benchmark asks for the occurrence that minimizes

$q - r_e \quad \text{with } q > r_e.$

This makes VQ a last-seen retrieval problem over long egocentric video (Grauman et al., 2021).

In Natural Language Queries, the input is a video $\mathcal{V}$ and a natural-language question $\mathcal{Q}$ , and the output is a temporal response window $v$ 0. The task includes only episodic queries answerable from the video itself and excludes external-knowledge or intention-reasoning questions. Ego4D describes 13 templates spanning objects, places, and people, including forms such as “Where is object $v$ 1 before/after event $v$ 2?”, “What did I put in $v$ 3?”, and “Who did I interact with when I did activity $v$ 4?” (Grauman et al., 2021).

In Moment Queries, the input is a video and a category label for a high-level activity or moment, and the output is the set

$v$ 5

of all temporal segments instantiating that category, with start time, end time, and confidence. Ego4D organizes MQ around 110 activity categories derived semi-automatically from narration summaries (Grauman et al., 2021).

Later challenge reports introduce GoalStep as another temporal grounding track over long, untrimmed egocentric videos. In that setting, the query describes a procedural step, and the system must localize the corresponding step grounding interval. Challenge reports from 2024 through 2026 treat NLQ and GoalStep as closely related temporal segment localization problems, and the 2025 report further unifies NLQ, GoalStep, and MQ as variants of precise interval localization in egocentric video (Feng et al., 2024, Feng et al., 4 Jun 2025).

3. Annotations, splits, metrics, and baseline difficulty

The three original episodic-memory tasks are annotated on different subsets of Ego4D. VQ covers 433 hours with 22,602 visual queries across 54 scenarios. NLQ covers 227 hours with 19.2k queries across 34 scenarios. MQ covers 326.4 hours with 22.2k action instances, 2,488 clips, and 110 categories (Grauman et al., 2021). Within each benchmark, train, validation, and test splits are disjoint in video clips; for the public challenge, test annotations are withheld and submission is done through a server (Grauman et al., 2021).

For NLQ, Ego4D reports 11.3k / 3.9k / 4.0k train/validation/test queries over 136 / 45 / 46 hours, with 8.3 ± 2.1 words as the average query length and 9.3 ± 21.5 seconds as the average response window. NaQ later characterizes the same problem as a “needle-in-a-haystack” localization setting: average video length is 8.2 minutes, average ground-truth response length is 10.5 seconds, and the target segment is only about 2% of the video on average (Grauman et al., 2021, Ramakrishnan et al., 2023).

The main official metrics are task-specific:

Track	Query and output	Main metrics
VQ-2D	Visual crop + query time $v$ 6 response track	tAP, stAP, Succ, rec%, sEff
VQ-3D	VQ-2D + 3D localization	RMSE, angular error, succ
NLQ	Natural-language query $v$ 7 temporal window	R@1/R@5 at IoU $v$ 8
MQ	Moment category $v$ 9 all temporal segments	mAP at tIoU $o$ 0, Recall@kx
GoalStep	Procedural step $o$ 1 grounding interval	R@1/R@5 at mIoU $o$ 2; Mean R@1

The original baselines illustrate the challenge’s difficulty. For VQ-2D, the best reported baseline in the benchmark paper, Siam-RCNN + KYS + Residual, obtains Val Succ 39.8, Val tAP 0.12, Val stAP 0.04, Val rec 32.2, and Test Succ 41.6, Test tAP 0.12, Test stAP 0.05, Test rec 34.0. For NLQ, the benchmark reports 2D-TAN at 5.80 / 13.90 / 2.34 / 5.96 on test for $o$ 3, $o$ 4, IoU $o$ 5, and IoU $o$ 6, and VSLNet at 5.47 / 11.21 / 2.80 / 6.57. For MQ, the adapted VSGN baseline reports 5.68 test mAP (Grauman et al., 2021).

4. Methodological evolution from baseline localization to object-aware and efficiency-aware systems

Challenge work after the initial benchmark diversified along at least four lines: reducing false positives, enlarging supervision, reweighting temporal evidence, and improving efficiency.

For VQ2D, “Negative Frames Matter in Egocentric Visual Query 2D Localization” argues that the official detector-tracker baseline is overly prone to false positives on background frames because training mostly exposes the model to clean positive frames while evaluation includes noisy, blurry, or unlabeled background frames. The paper introduces negative proposals from inside the annotated interval with IoU < 0.5, from frames outside the object-visibility interval, and from other videos, combined with hard-negative mining and a positive-negative ratio of 1:64. It reports that the training loop is reduced from roughly 15 days to under 24 hours, evaluation to around 12 hours, and the test result reaches 0.17% spatial-temporal AP, stated as 31% higher than the baseline (Xu et al., 2022).

For NLQ, “NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory” repurposes Ego4D narrations as pseudo-query supervision. NaQ creates about 850k NaQ samples from 4,851 Ego4D training video clips, or roughly 80× more samples than the native NLQ training set. The method improves VSLNet, EgoVLP, and ReLER, enables zero-shot and few-shot NLQ, and reports that ReLER $o$ 7 + NaQ reaches 18.46 at $o$ 8, IoU $o$ 9, 10.74 at $q$ 0, IoU $q$ 1, and 14.59 mean $q$ 2, outperforming prior challenge winners from CVPR 2022 and ECCV 2022 (Ramakrishnan et al., 2023).

Another line is frame-sensitive optimization. The ReLER/Zhejiang University report “Action Sensitivity Learning for the Ego4D Episodic Memory Challenge 2023” introduces Action Sensitivity Learning (ASL), which assigns learnable sensitivity weights to frames and disentangles them into $q$ 3 and $q$ 4 for classification and localization. On MQ, the submission reports 29.34 mAP and 48.50 R@1, ranking 1st place; on NLQ, it reports 19.79 mean R@1, ranking 2nd place (Shao et al., 2023).

Efficiency became a distinct research target in SpotEM, which observes that existing EM methods spend over 99.9% of compute on dense clip feature extraction. SpotEM combines a query-conditioned clip selector, low-cost room/object/interaction indexing features, and distillation. On Ego4D NLQ, it reports that computing only 10%–25% of the clip features preserves about 84%–97% of the original model’s accuracy, and that many settings retain 95%+ accuracy at about 4× lower compute (Ramakrishnan et al., 2023).

Object-centric grounding then became more explicit. “ObjectNLQ @ Ego4D Episodic Memory Challenge 2024” augments a GroundNLQ-style temporal localizer with an object branch built from Co-DETR pretrained on LVIS v1.0 and CLIP ViT-L/14 text encoding of detected object classes. The model reports mean R@1 = 23.15, ranking 2nd on the NLQ challenge, and R@1 = 33.00 at IoU = 0.3, ranking 3rd on GoalStep (Feng et al., 2024).

5. Unified localization systems and the 2025–2026 winning solutions

A major consolidation occurs in the OSGNet line. “OSGNet @ Ego4D Episodic Memory Challenge 2025” argues that prior unified localization methods often rely on late fusion, which tends to be suboptimal for egocentric grounding. OSGNet instead adopts an early-fusion design with a main branch that fuses video, text, and object features and a shot branch trained with contrastive learning. Videos are divided into non-overlapping 16-frame snippets; video features are extracted using EgoVideo and InternVideo, and text features using EgoVideo and CLIP. The report states that the method achieved first place in the Natural Language Queries, Goal Step, and Moment Queries tracks (Feng et al., 4 Jun 2025).

The reported 2025 ensemble results are strong across all three tasks: on NLQ, R@[email protected] = 30.19 and R@[email protected] = 21.78; on Goal Step, R@[email protected] = 42.02 and R@[email protected] = 32.83; on MQ, mAP = 36.78 and [email protected] = 54.08 (Feng et al., 4 Jun 2025). The report attributes these gains to early fusion, object-aware and shot-aware modeling, NaQ pretraining, and ensembling.

The 2026 winning report, “OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026,” retains OSGNet as the candidate generator but adds a reranking stage with GPT-5.4. For both NLQ and GoalStep, the system keeps the top-5 candidate segments from OSGNet and lets the MLLM choose the best match (Feng et al., 20 May 2026).

For NLQ, reranking is done in two stages because GPT-5.4 operates under an image/token limit. Each candidate segment is split into 20-second clips; frames are sampled at 1 FPS and converted by GPT-5.4 into textual narrations, which are accumulated into an episodic memory for that candidate. GPT-5.4 then compares the episodic memories of all candidates and selects the one that best matches the query (Feng et al., 20 May 2026).

For GoalStep, the same top-5 reranking is combined with a sequential prior over ordered steps $q$ 5. Let $q$ 6 be the start timestamp of the selected segment for query $q$ 7, and $q$ 8 its rank in the localization model’s candidate list. The report defines

$q$ 9

and minimizes

$r = \{r_s, r_{s+1}, \cdots, r_e\},$ 0

This post-processing prefers candidates ranked highly by OSGNet while encouraging monotonically increasing start-time order (Feng et al., 20 May 2026).

The 2026 paper reports that the reranking gain is modest in the validation-style tables. On the NLQ test split, OSGNet reports R@[email protected] = 21.63 and R@[email protected] = 15.52, while OSGNet + rerank reports 21.78 and 15.44. On the GoalStep test split, OSGNet reports 55.31 and 47.82, while OSGNet + rerank reports 55.39 and 47.91. The authors note that the measured improvement on NLQ may be limited by false negatives in the evaluation protocol, because a reranked prediction can be semantically correct yet not match the exact annotated interval, and they state that the sequential-prior post-processing substantially improves the final GoalStep submission. The official GoalStep leaderboard table reports the submission yisen_feng in 1st place with R@[email protected] = 63.02, R@[email protected] = 54.21, R@[email protected] = 80.12, R@[email protected] = 74.93, and Mean R@1 = 58.61 (Feng et al., 20 May 2026).

Challenge-related research increasingly extends beyond one-shot offline localization. EMQA formalizes episodic memory question answering as localization in egocentric pixel space or allocentric top-down map space, using a reusable episodic scene memory built from RGB-D observations and pose. The proposed top-down semantic memory plus temporal augmentation outperforms language-only, frame-by-frame, and compressed-buffer baselines, with 29.11 IoU, 62.27 Recall, and 33.39 Precision in top-down map space for the temporal variant (Datta et al., 2022).

A second extension is online memory. “Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory” introduces OEM-VQL, in which the full video history is not available at query time; the model must instead answer from a compact memory built during streaming observation. The proposed ESOM framework stores object tracks and representations online. Under oracle object discovery and tracking, it reports 81.92% Success versus 55.89% Success for the best offline baseline, VQLoC, while also showing that practical performance is severely limited by egocentric object detection and tracking quality (Manigrasso et al., 2024).

A third extension is interactive refinement. “Interactive Episodic Memory with User Feedback” introduces EM-QnF, where a user can clarify or correct an initial prediction with natural-language feedback. The method adds a plug-and-play Feedback ALignment Module (FALM) and reports gains up to about +4.9 R1 and +5.4 R5, while also showing that it generalizes to 500 feedback instances from 11 users (Subedi et al., 27 Apr 2026).

A fourth extension is diagnostic streaming evaluation. EGOSTREAM organizes 2,250 curated questions along seven cognitive dimensions—detail, spatial, temporal, event, social, causal, and prospective memory—and expands them into 8,528 recall-conditioned evaluations through the Answer Validity Window (AVW). The benchmark shows that comparable aggregate accuracies can hide very different memory profiles, that token pruning preserves fine-grained details and temporal structure better than token merging, that quantized offloading helps ultra-long-term recall, and that top-performing methods still reach only about 45% accuracy while operating below real time at >1s per frame (Forte et al., 29 May 2026).

These developments indicate that the Ego4D Episodic Memory Challenge has evolved from a benchmark of offline localization over long first-person video into a broader research program on memory construction, memory management, retrieval, temporal consistency, and interaction. A plausible implication is that future progress will depend not only on stronger localizers, but also on representations that preserve event-critical information under long temporal horizons, on evaluation protocols that better separate genuine failure from annotation mismatch or world-state change, and on systems that can operate under streaming and interactive constraints.