Minerva-Ego Benchmark for Egocentric Video QA

• Minerva-Ego is a benchmark for egocentric video understanding that integrates long-form videos with dense, object-grounded reasoning annotations.
• It employs 1,160 hand-crafted multiple-choice questions over 156 HD-EPIC videos, using explicit timestamps and object segmentation for spatiotemporal evaluation.
• Experiments show spatiotemporal hints enhance model performance, yet a significant gap remains compared to human-level reasoning.

Minerva-Ego is an egocentric long-form video question-answering benchmark introduced in “Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding” (Nagrani et al., 14 May 2026). Built on HD-EPIC, it pairs 1,160 hand-crafted multiple-choice questions over 156 egocentric videos with dense human-authored reasoning traces grounded in timestamps and object segmentation masks. Its design target is not only answer correctness but also the evaluation of intermediate reasoning, perceptual grounding, and temporal localization. In that sense, Minerva-Ego formalizes egocentric video understanding as a multimodal reasoning problem in which the relevant evidence is distributed across long temporal contexts and object-centric interactions, and it provides a benchmark for testing whether models looked at the right objects at the right times (Nagrani et al., 14 May 2026).

1. Definition and scope

Minerva-Ego is a benchmark for evaluating complex egocentric visual reasoning. It is built on HD-EPIC, a kitchen-based egocentric dataset collected in-the-wild with annotations for recipes, fine-grained actions, audio events, object movements, and object masks lifted to 3D. The benchmark contributes 1,160 complex, hand-crafted questions, each with five answer choices and a single correct answer, together with dense reasoning traces that include timestamps and references to specific objects-of-interest (Nagrani et al., 14 May 2026).

The dataset is long-form rather than clip-centric. Video durations range from 10 seconds to 75 minutes, with a mean of approximately 20 minutes. Reasoning traces average 122 words, contain a mean of 6.3 timestamps per question, and reference a mean of 2.8 objects, with some traces referencing up to 20 objects. Object-ID linkage to HD-EPIC annotations is available for 82% of referenced objects when HD-EPIC annotations exist. The videos have no spoken content, so audio and ASR are not used in the benchmark protocol (Nagrani et al., 14 May 2026).

A concise summary of the benchmark’s core statistics is given below.

Item	Value	Note
Questions	1,160	Hand-crafted, 5 choices each
Videos	156	Egocentric, from HD-EPIC
Video duration	10 s to 75 min	Mean $\approx 20$ min
Reasoning trace length	122 words	Mean
Timestamps per trace	6.3	Mean
Objects referenced	2.8	Mean; up to 20
Object-ID linkage	82%	Of referenced objects

The benchmark’s scope is narrower in domain than general video QA datasets, because it is kitchen-centric, but denser in perceptual grounding. This suggests that its primary value lies in diagnosing egocentric and embodied reasoning failures that are easy to obscure in final-answer-only evaluation.

2. Data construction and annotation

Each Minerva-Ego item consists of a long-form egocentric video, a five-choice multiple-choice question, a single correct answer, and a human-authored reasoning trace. The reasoning traces are stepwise, explicitly timestamped, and tied to objects of interest. Rater guidelines require multi-step questions involving multiple time segments, and every reasoning trace must contain at least one timestamp and at least one object reference. The questions use first-person phrasing, with “I” used consistently for the annotator and “user” for the camera wearer (Nagrani et al., 14 May 2026).

Object grounding is a defining component of the annotation schema. Minerva-Ego reuses HD-EPIC’s ground-truth object segmentation masks. Because HD-EPIC segments each object “when it is moved” in the first and last frame of the movement, Minerva-Ego augments these tracks by interpolating masks across intermediate frames with a re-implementation of SAM2. The primary annotations are therefore links between reasoning objects, timestamps, and per-frame pixel segmentations, rather than a box-only representation (Nagrani et al., 14 May 2026).

Quality control is multi-stage and manual. A second rater pool attempts the questions, and disagreements or conflations are reconciled by a third pool that may revise questions, options, or reasoning. Post-processing also attempts to reduce modality bias by following prior best practices and using a text-only model ensemble to flag candidate biased items for correction or removal. No inter-annotator $\kappa$ or related agreement coefficient is reported (Nagrani et al., 14 May 2026).

The qualitative examples in the paper illustrate the annotation philosophy. In one ordering question, the ground-truth reasoning identifies “drinking water” at 03:06, “pouring hot water into the flask” at 04:39, and “putting the second knife into the dishwasher” at 05:50, with the flask and knife linked to object identifiers. In another example, a counting question about grated lemons is resolved by locating two distinct lemons at 02:57 and 04:17. These examples are representative of the benchmark’s emphasis on event sequencing, repeated object instances, and explicit temporal evidence (Nagrani et al., 14 May 2026).

3. Task formulation and evaluation protocol

The task is multiple-choice video question answering over long-form egocentric videos. Although the final output is a text choice, the benchmark treats reasoning as multimodal because correct justification must align with timestamps and object-level evidence. Human and model performance are therefore evaluated along multiple axes rather than by answer accuracy alone (Nagrani et al., 14 May 2026).

The primary answer metric is multiple-choice accuracy:

$$\mathrm{Acc}_\text{MCQ} = \frac{C}{N} \times 100\%.$$

Here, $N$ is the number of questions and $C$ is the number of correct predictions. Since each question has five choices, random performance is 20.0% (Nagrani et al., 14 May 2026).

Reasoning quality is evaluated with the MINERVA rubric, or MiRA. The four criteria are perceptual correctness, temporal grounding, logical reasoning, and completeness. Each criterion is scored on $\{0,1,2\}$, where 0 means the criterion is not fulfilled, 1 means partially fulfilled, and 2 means completely fulfilled. MiRA is run as a rubric-guided judge; the paper reports results with Gemini 2.5 Pro and notes that repeating the evaluation with GPT-5 preserves the relative ordering of models (Nagrani et al., 14 May 2026).

A further diagnostic metric is object recall in reasoning. This is reported at two granularities: object-level recall, which requires matching the full object description, and noun-level recall, which requires matching only the head noun. The reported trend is that models reference only 20–50% of relevant objects from the ground-truth reasoning. That finding matters because a model can produce superficially coherent rationales while failing to mention a large fraction of the objects needed to solve the problem (Nagrani et al., 14 May 2026).

4. Benchmark results and failure structure

Minerva-Ego reports a large gap between current models and humans. On the full 1,160-question benchmark, the best model in the reported 64-frame setting is Gemini 2.5 Pro with thinking enabled at 40.1% MCQ accuracy, whereas human performance is 91.8%. GPT-5 reaches 37.6%, Gemini 2.5 Flash with thinking 35.6%, GPT-4.1 30.8%, Claude Sonnet 4 30.5%, and Qwen-3 29.3% (Nagrani et al., 14 May 2026).

System	Input setting	MCQ accuracy
Qwen-3	all @ 1 fps; thinking	29.3%
Claude Sonnet 4	64 frames; thinking	30.5%
GPT-4.1	64 frames; no reasoning	30.8%
Gemini 2.5 Flash	64 frames; no thinking	31.7%
Gemini 2.5 Flash	64 frames; thinking	35.6%
GPT-5	64 frames; medium reasoning effort	37.6%
Gemini 2.5 Pro	64 frames; thinking	40.1%
Humans	full videos	91.8%

Frame ablations show that additional temporal coverage matters substantially. For Gemini 2.5 Pro, performance rises from 27.5% in the QA-only condition, to 29.3% with a single frame, to 40.1% at 64 frames, 44.5% at 256 frames, 49.1% at 512 frames, and 49.8% at 1024 frames. The single-frame condition remains near chance, indicating that egocentric reasoning in this benchmark is not reducible to static-image recognition (Nagrani et al., 14 May 2026).

MiRA and qualitative analyses converge on the same error structure. Models score higher on logical reasoning and completeness than on perceptual correctness, and the benchmark identifies two dominant failure classes: perceptual identification errors and temporal grounding errors. The qualitative examples include missing a second egg, miscounting lemons, and attributing mat movement to vacuum suction rather than to the user picking the mat up. The object-recall results reinforce this diagnosis, because models mention only 20–50% of relevant objects in their reasoning traces (Nagrani et al., 14 May 2026).

A plausible implication is that current frontier video-LLMs are often better at producing globally coherent textual explanations than at maintaining fine-grained egocentric evidence alignment. Minerva-Ego is designed precisely to expose that discrepancy.

5. Spatiotemporal hints and “where/when” prompting

A central experimental contribution of Minerva-Ego is its study of spatiotemporal hints. The paper tests whether explicitly indicating where relevant objects are and when relevant events occur can improve egocentric reasoning. In the oracle setting, relevant objects are taken from the ground-truth reasoning and visualized by overlays such as mask contours, boxes, circles, and class names; temporal selection chooses frames where those objects occur, with the remaining frame budget filled by uniform sampling (Nagrani et al., 14 May 2026).

For Gemini 2.5 Pro at 256 frames, the baseline is 44.5% without temporal selection and 47.4% with temporal selection. Among visualization variants, circles plus class names perform best, reaching 47.3% without temporal selection and 50.3% with temporal selection. The peak improvement shown in the table is therefore $44.5\% \rightarrow 50.3\%$, or +5.8 percentage points; the narrative text cites 5.6, but the table reports 5.8 (Nagrani et al., 14 May 2026).

The benchmark also studies a practical predicted-hints setting. Here, Gemini 2.5 Pro first lists useful object categories, filtered to remove classes such as “hand,” “person,” fixtures, storage spaces, and large appliances. OWLv2 is then used for open-vocabulary detection and segmentation at threshold 0.6. In this setting, detection yields objects in 26% of frames, with a mean of 1.26 objects per frame when any object is present. Temporal selection alone raises the baseline from 44.5% to 46.5%, and adding class names yields 47.0%; circles are not beneficial in the predicted setting (Nagrani et al., 14 May 2026).

The same general pattern transfers to EgoTempo. For Gemini 2.5 Pro at 32 frames and 1 fps, baseline accuracy is 39.9%, spatial hints increase it to 41.1%, and temporal selection to 42.1%. This suggests that the “where/when” effect is not restricted to a single benchmark configuration (Nagrani et al., 14 May 2026).

These experiments do not introduce a new model architecture. Instead, they show that explicit spatiotemporal grounding can improve reasoning with prompt-time interventions alone. That result is technically important because it isolates perceptual and localization bottlenecks without conflating them with large-scale retraining.

6. Relation to MINERVA, other egocentric benchmarks, and naming ambiguity

Minerva-Ego should be distinguished from the broader “MINERVA” video reasoning dataset introduced in 2025. That earlier dataset provides 1,515 manually created questions over 223 videos, with five answer options and detailed reasoning traces, and it evaluates reasoning with the same four axes—perceptual correctness, temporal localization, logical reasoning, and completeness. However, it does not define a named subset called “Minerva-Ego.” Egocentric content appears there mainly in the Lifestyle category, especially travel and vlogging videos, but not as a separate benchmark split (Nagrani et al., 1 May 2025).

Relative to other egocentric datasets, Minerva-Ego is distinguished by dense reasoning supervision rather than by scale alone. EgoSchema focuses on long-form QA from Ego4D; EgoTempo emphasizes temporal localization; HourVideo contains approximately 12,976 questions on 500 Ego4D videos and is template-generated; EgoVQA, EgoTaskQA, and EgoMemoria are prior egocentric datasets, many of which rely on machine-generated QA. Minerva-Ego is presented as the first egocentric benchmark to pair complex hand-crafted multiple-choice questions with dense spatiotemporally grounded reasoning traces (Nagrani et al., 14 May 2026).

There is also a separate naming ambiguity outside video reasoning. In the neutrino-interaction literature around the Fermilab MINERvA experiment, “Minerva-Ego” appears as a descriptive label for a comprehensive, self-consistent GeV-regime neutrino–nucleus study and for related discussions of MINERvA measurements and form-factor constraints. That usage concerns neutrino–nucleus physics rather than egocentric video understanding (Lu et al., 2021, Tomalak et al., 2023). The two usages are unrelated except for the shared string.

Within multimodal reasoning, the significance of Minerva-Ego is therefore specific: it operationalizes egocentric QA as a problem of answer selection plus grounded intermediate reasoning, and it shows that explicit object- and time-aware prompting can improve model performance while leaving a very large residual gap to human performance (Nagrani et al., 14 May 2026).