OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026

Abstract: In this report, we present our champion solutions for the Natural Language Queries and GoalStep tracks of the Ego4D Episodic Memory Challenge at CVPR 2026. Both tracks require accurately localizing temporal segments from long untrimmed egocentric videos. To address these tasks, we propose a reranking-based framework that effectively leverages the strong video-language reasoning capability of multimodal LLM (MLLM) while preserving the efficiency and candidate recall of conventional localization pipelines. Specifically, we first obtain a set of candidate segments from existing localization model OSGNet, and then employ MLLM to select the segment that best matches the given query, thereby refining the final prediction. Ultimately, our method achieved first place in both the Natural Language Queries and GoalStep tracks. Our code can be found at https://github.com/iLearn-Lab/CVPR25-OSGNet.

• The paper proposes a two-stage pipeline using OSGNet for candidate segment proposals and GPT-5.4 for cross-candidate reranking.
• It leverages per-segment episodic narrations and temporal ordering priors to refine retrieval accuracy in NLQ and GoalStep tracks.
• Empirical results demonstrate modest yet leading improvements in R@1 metrics, underscoring enhanced robustness in egocentric video understanding.

OSGNet with MLLM Reranking for Temporal Localization in Ego4D Episodic Memory

Introduction

Egocentric video moment localization remains a challenging task due to the inherent complexity and diversity of first-person visual data. The Ego4D Episodic Memory Challenge, specifically its Natural Language Queries (NLQ) and GoalStep tracks, benchmarks progress in retrieving precise temporal segments from long, untrimmed egocentric videos conditioned on textual prompts. The discussed work explores the integration of conventional segment proposal models with multimodal LLM (MLLM)-based reranking to address limitations in generalization, robustness, and semantic reasoning faced by current approaches (2605.20818).

Methodology

Reranking Pipeline

The method is based on a two-stage reranking pipeline. Firstly, OSGNet, a competitive spatio-temporal segment grounding network, generates the top-5 candidate segments per query. The subsequent reranking leverages GPT-5.4 as a vision-language MLLM. Given computational and contextual input constraints of MLLMs, each candidate segment is subdivided into 20-second clips, and frame sampling (1 FPS) is performed. The frames are sequentially fed into GPT-5.4, which produces textual narrations. Accumulated narrations act as episodic memories for each candidate, over which GPT-5.4 performs cross-candidate reasoning to rerank and select the best match for the original query.

Figure 1: The reranking pipeline, illustrating candidate generation by OSGNet, per-segment episodic narration, and GPT-5.4-based segment selection.

Natural Language Queries (NLQ) Track

For NLQ, evaluation focuses on segment retrieval conditioned on free-form questions. The reranking pipeline slightly improves top-1 recall (R@1, mIoU=0.3) from 21.63% (OSGNet baseline) to 21.78%, but negligible change is observed for higher mIoU thresholds and for R@5. The benefit is attributed to improved discrimination of visually or semantically ambiguous negatives, but the overall gain is attenuated by dataset annotation ambiguity.

Figure 2: Two NLQ validation cases—successful reranking on subtle visual-semantic differences, and an apparent failure due to plausible yet unmatched prediction.

GoalStep (Step Grounding) Track

For GoalStep, grounded on high-level procedural step queries, the reranking strategy produces consistent top-1 accuracy improvements (R@1, mIoU=0.3: 55.31% → 55.39%; mIoU=0.5: 47.82% → 47.91%). Further, the method incorporates sequential ordering priors in post-processing: for an ordered sequence of $K$ step queries, predictions are penalized if temporal monotonicity is violated, and candidate selection minimizes a combined cost of proposal rank and temporal penalties. This robust post-processing provides a notable boost—reflected in leaderboard metrics where the approach surpasses all competitors with a mean R@1 of 58.61%.

Figure 3: Two GoalStep validation cases: improved disambiguation of procedural semantics by reranking, and an illustrative latent-positive failure.

Analysis and Implications

The OSGNet+MLLM reranking framework demonstrates several technical benefits:

• Candidate Set Filtering: By confining MLLM processing to a manageable candidate set, the framework addresses the context length limitations of advanced MLLMs without sacrificing fine-grained video understanding.
• Visual-Linguistic Reasoning: The pipeline leverages state-of-the-art MLLMs for deeper multimodal semantic alignment, improving discrimination between difficult negatives, especially in visually similar, textually ambiguous conditions.
• Sequential Priors in Procedural Reasoning: Incorporation of dataset-specific priors (e.g., step-order monotonicity) in candidate selection enables further refinement, especially critical for procedural activity localization.

However, the empirical improvement—while leading in the challenge—remains modest in magnitude, highlighting unsolved challenges in annotation granularity, hard-negative mining, and metric sensitivity to plausible but ungrounded predictions. The method's benefit is more substantial in procedurally-structured tasks, likely due to the structured temporal constraints being more amenable to post-hoc reasoning.

Future Directions

Key future avenues suggested by this research include:

• Tighter Multimodal Integration: Developing video-LLMs capable of end-to-end joint reasoning without separate candidate generation and reranking stages, possibly via enhanced memory architectures or temporal inference mechanisms.
• Efficient Long-Context Processing: Addressing computational limits in MLLMs for direct dense video input processing while retaining episodic memory fidelity.
• Refined Ground Truth and Metrics: Handling the latent positive problem and annotation ambiguities observed in failure cases, which currently obscure reranking benefits in standard metrics.

Conclusion

This work validates that integrating MLLM-based reranking with strong proposal networks and dataset-specific post-processing constitutes a robust strategy for egocentric video localization, as evidenced by its first-place performance in both NLQ and GoalStep tracks of the Ego4D Episodic Memory Challenge. The pipeline strikes an effective balance between efficiency, recall, and interpretability, while uncovering continuing challenges in aligning quantitative evaluation and human-centric episodic memory retrieval in long-form egocentric media.