Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding

Abstract: Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a benchmark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at https://github.com/google-deepmind/neptune.

• The paper introduces Minerva-Ego, a benchmark providing dense spatiotemporal reasoning traces for egocentric video question answering.
• It uses HD-EPIC video data with multi-step queries to expose significant performance gaps between human raters and leading MLLMs.
• Spatiotemporal hinting via visual prompts, including ground-truth segmentations, notably improves model accuracy and highlights the need for explicit visual grounding.

Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding

Introduction and Motivation

Minerva-Ego (2605.15342) addresses a central deficiency in contemporary video question answering (VideoQA) benchmarks for egocentric (first-person) video: the lack of explicit, granular reasoning trace annotations that detail not only what the answer is, but how that answer can be derived from temporal and spatial cues. While previous datasets focus on final answer correctness, Minerva-Ego introduces a new standard—dense, manually annotated spatiotemporal reasoning traces—enabling fine-grained error analysis for state-of-the-art (SotA) multimodal LLMs (MLLMs) in the context of long, complex, and highly interactive egocentric video.

Figure 1: Illustration of complex spatio-temporal hinting and reasoning grounding in egocentric video enabled by Minerva-Ego.

The benchmark leverages HD-EPIC egocentric video data, featuring 1,160 challenging multi-step multimodal questions, each with exhaustive human-annotated reasoning traces. These traces are densely grounded in both time (with precise timestamps) and space (via segmentation masks linking cues to objects and frames), thus facilitating new forms of model diagnosis and explicit comparison between human and model reasoning.

Dataset Design and Annotation

The design of Minerva-Ego is characterized by exhaustive manual curation at every stage: video selection, annotation, verification, and post-processing to minimize modality bias. Each question requires multi-step inference, referencing an average of 2.8 distinct objects and 6.3 timestamps per trace.

Figure 2: Distributions in trace length, timestamp references, and object references per question in Minerva-Ego, highlighting the multistep and fine-grained nature of the reasoning required.

Reasoning traces adopt a stepwise structure, requiring references to both objects (with corresponding segmentation masks) and explicit temporal regions. The intent is not only to challenge models on final answer correctness but to make explicit where their reasoning pipelines diverge from human annotators at the level of perception, temporal grounding, and object identification. Extensive ablation and post-hoc filtering ensure that questions cannot be shortcut by text priors or modality biases, increasing the benchmark’s diagnostic value.

Benchmarking and Model Evaluation

Minerva-Ego provides rigorous empirical evidence that current MLLMs—including proprietary models such as Gemini 2.5 Pro, GPT-5, and Claude Sonnet 4, as well as open-source baselines like Qwen-3—exhibit a substantial gap with human raters in both answer accuracy and reasoning trace fidelity.

Overall:

• Human performance: 91.8% MCQ accuracy
• Best frontier model (Gemini 2.5 Pro): 40.1% MCQ accuracy
• All leading MLLMs underperform, with the best model more than 50 percentage points below human performance.

Error analysis using the MiRA rubric further shows that perceptual correctness and temporal grounding are substantial failure modes for current models, despite strong logical and completeness scores in their generated traces.

Figure 3: MiRA rubric breakdown, showing that logical consistency and completeness are less problematic for MLLMs than perceptual grounding and temporal localization.

Detailed qualitative analyses highlight model errors in tracking objects, resolving time spans, and associating actions with correct spatial entities, especially under challenging egocentric dynamic occlusions and multimodal ambiguity.

Spatiotemporal Highlighting: Oracle and Practical Regimes

To probe whether explicit, structured cues can mitigate reasoning failures, the authors introduce spatiotemporal hinting: overlaying object masks, class labels, and emphasizing relevant frames in the model’s input. When performed under oracle conditions—using ground-truth human segmentations—this visual prompting yields a significant absolute accuracy gain (up to +5.8% for combined class-labeled, circled objects and temporal selection).

Figure 4: Visualization comparing model predictions without (left) and with (right) spatiotemporal hints, demonstrating marked improvement in temporal focus and object identification.

Using predicted segmentation (OWLv2), the improvements remain, though more modest (up to +2.5%), indicating both the utility of spatiotemporal selection and the current limits of automated object detection in open-ended egocentric video. Notably, temporal selection of frames independently provides measurable benefit—consistent with the finding that single-frame inputs dramatically degrade model performance, confirming Minerva-Ego’s multi-timescale challenge.

Figure 5: Effectiveness of different object highlighting strategies (masks, boxes, circles, class labels) in guiding model attention to relevant spatial regions in frames.

Analysis by Video Length, Skill, and Ablation

Analysis of model accuracy relative to video length indicates that most SotA models degrade on videos exceeding one hour, with consistently robust performance on shorter clips. A skill-type breakdown reveals relative robustness on state change, counterfactual, and situational awareness questions, but marked weaknesses for event occurrence, numerical reasoning, and counting.

Implications and Future Directions

Minerva-Ego elevates the standard for egocentric video QA by facilitating diagnosis at every stage of the perception-to-reasoning pipeline. The finding that even the strongest MLLMs fail to reliably ground textual answers in visual and temporal specifics invites several lines of further research:

• Model architectural innovation targeting explicit visual grounding in dynamic first-person scenes, potentially leveraging joint spatiotemporal segmentation and language as intermediate representations.
• Training or fine-tuning with explicit reasoning trace supervision, to enable models to iteratively “point and reason” over candidate objects and time ranges.
• Integration of online object tracking and open-vocabulary detection as core modules, closing the performance gap between oracle and automatic hinting regimes.
• Hierarchical attention over video segments guided by human-annotated reasoning structures, particularly scalable to long-horizon, egocentric scenarios.
• Further exploration of visual prompting as a mechanism to bridge perception errors in fixed-parameter models.

Minerva-Ego can serve both as a diagnostic benchmark for model evaluation and a dataset for training models explicitly on multi-step, explicitly mapped spatiotemporal reasoning tasks in open-world, ego-centric situations.

Figure 6: Distribution of video lengths, illustrating the inclusion of both short and long-form egocentric activities.

Figure 7: Reasoning trace composition and object reference statistics stratified by question type, benchmarking the complexity and diversity of skills required.

Figure 8: MiRA scores when reasoning traces are judged using GPT-5 rather than Gemini, confirming the robustness of failure mode identification.

Figure 9: Examining the relationship between input video length and SotA model accuracy, noting performance degradation for extended contexts.

Figure 10: Accuracy of frontier models categorized by required skill, with consistently poor performance on difficult counting and event resolution tasks.

Conclusion

Minerva-Ego (2605.15342) represents a step-change in the evaluation of egocentric video understanding: it enforces multi-step, explicit reasoning trace grounding in both space and time and systematically exposes both the strengths and deep limitations of contemporary MLLMs. The experimental evidence highlights a critical role for structured spatiotemporal prompting and motivates research on integrating explicit grounding, better multi-modal fusion, and visual reasoning into egocentric embodied AI. This resource is positioned to become a key benchmark for advancing research in both egocentric video understanding and interpretable, diagnosable multi-modal reasoning systems.