EgoIllusion: Egocentric Hallucination Benchmark

• EgoIllusion is a specialized benchmark that measures hallucination in first-person video by leveraging both visual and auditory cues in dynamic interactions.
• It evaluates tasks like episodic reasoning, temporal order, object identification, and audio event recognition through adversarial prompt injections and manipulations.
• With over 1,400 egocentric videos and 8,000 annotated QA pairs, EgoIllusion provides a rigorous test of multimodal model robustness against language priors and misaligned sensory evidence.

Searching arXiv for EgoIllusion and related egocentric benchmark papers. Searching arXiv for "EgoIllusion" and "egocentric hallucination benchmark". EgoIllusion is a benchmark for evaluating hallucinations in multimodal LLMs (MLLMs) on egocentric video understanding. It is introduced as the first benchmark specifically designed for this setting, with the explicit premise that first-person video poses difficulties not captured by static-image, exocentric-video, or generic multimodal QA benchmarks: dynamic hand-object interactions, frequent occlusion, changing object states, long temporal dependencies, and multisensory cues such as environmental sound. In the benchmark’s framing, hallucinations are coherent but inaccurate responses that diverge from the actual sensory evidence in the video, and the benchmark is constructed to expose failures in which models rely on language priors or scene plausibility rather than grounding in the visual and auditory stream (Seth et al., 18 Aug 2025).

1. Historical placement and problem formulation

EgoIllusion addresses a gap in egocentric evaluation rather than proposing a mitigation method. Earlier hallucination benchmarks cited alongside it—such as POPE, HallusionBench, MMHal-Bench, Bingo, EasyDetect, VHTest, and VALOR—are described as mostly image-based, while VideoHallucer evaluates videos in exocentric settings. EgoIllusion differs in three declared respects: it is egocentric, it explicitly includes both visual and auditory cues, and it is hallucination-oriented by construction through prompt injection, adversarial object/action substitution, and temporal-order manipulation (Seth et al., 18 Aug 2025).

The benchmark’s central claim is that hallucination in egocentric video is not reducible to object presence detection. It also includes failures to verify whether an object was present at all, whether a specific action really occurred, whether an event happened before or after another event, and whether a sound was truly heard rather than merely plausible in context. This places EgoIllusion at the intersection of egocentric video QA, multimodal grounding, and hallucination analysis.

EgoIllusion is also positioned as an external benchmark for transfer evaluation in later work. EgoAVU describes EgoIllusion as an existing egocentric benchmark that, together with EgoTempo, attempts to enhance contextual understanding through visual captioning, and reports improved transfer after fine-tuning Qwen2.5-Omni on EgoAVU-Instruct (Seth et al., 5 Feb 2026). This later usage suggests that EgoIllusion has rapidly become a reference point for robustness in first-person multimodal reasoning.

2. Benchmark composition and task structure

EgoIllusion comprises more than 1,400 egocentric videos and 8,000 human-annotated question-answer pairs. The videos range from 30 seconds to over 5 minutes. The benchmark includes open-ended and closed-ended questions, covers both perception and reasoning, and explicitly spans both vision and audio. At a high level it is balanced into 4,000 perception questions and 4,000 reasoning questions (Seth et al., 18 Aug 2025).

The main paper lists six selected egocentric video-language tasks: Episodic Information Reasoning (EIR), Temporal Reasoning (TR), Human-Object Interaction (HOI), Visual Object Identification (VOI), Object State Change Detection (OSCD), and Audio Event Recognition (AER). The appendix table introduces a naming inconsistency by listing “Episodic Information Extraction” rather than OSCD, while the quantitative results table clearly evaluates OSCD. The paper itself notes that the raw results should be trusted where prose and formatting do not align exactly.

Task name	Role in the benchmark	Appendix format/count
EIR	episodic reasoning over events and objects	1000, open-ended
TR	before/after reasoning	2000, closed-ended
HOI	distinguish real from plausible interactions	1000, closed-ended
VOI	identify objects actually involved	2000, closed-ended
OSCD / “Episodic Information Extraction”	state-change / episodic retrieval style evaluation	1000, closed-ended
AER	audio-event grounding	1000, closed-ended

The task design is explicitly adversarial. EIR asks open-ended “how,” “what,” “why,” or “where” questions over temporally extended object histories. TR asks closed-ended before/after questions, with annotators instructed to choose events ideally 4–5 events apart. HOI distinguishes real interactions from plausible but non-occurring ones. OSCD tests state change and action completeness. VOI targets semantically plausible but absent objects. AER distinguishes sounds actually heard from sounds that would fit the environment but were not heard.

A common misconception is that EgoIllusion is simply an egocentric QA set with harder questions. The benchmark is more specific than that: its construction targets hallucination induction, not only recognition difficulty. That distinction is central to its interpretation.

3. Data sources, narration enrichment, and annotation protocol

The benchmark is built from multiple open-source egocentric datasets. The main paper names Ego4D-HCap / VideoRecap, EgoSeg, EPIC-KITCHENS, and Trek-150, while the appendix states that the dataset was curated primarily from VideoRecap and Ego4D. Videos were manually filtered to retain rich object interactions and meaningful temporal dynamics, including scenarios such as meal preparation in a kitchen, painting a canvas, assembling furniture, and navigating urban environments. Low-diversity clips such as stirring a pot for several minutes or walking down an empty hallway with little interaction were excluded (Seth et al., 18 Aug 2025).

A substantial technical component is the narration enhancement pipeline. Given a video $V$ with narration captions

$C = \{c_1, \dots, c_n\},$

a global video description $D$, active objects

$O_I = \{o_1, \dots, o_M\},$

and visible objects identified from key frames

$O_V = \{o_1, \dots, o_P\},$

the set of non-active or background objects is defined as

$O_S \leftarrow O_V - O_I.$

Qwen2-Audio is used to detect relevant audio cues from each clip, and the resulting enriched narration set is

$C' = \{c'_1, \dots, c'_n\},$

where each $c'_i$ includes human actions, active objects, non-active/background objects, and environmental sounds. The paper is explicit that this is not a fully automatic pipeline in the final benchmark: manual filtering and correction are applied afterward to fix possible errors in object and sound descriptions.

The benchmark was human-annotated by five experts—3 male and 2 female, all MS/PhD students with strong computer vision background and prior video-annotation experience—over a 4-week period. Annotators used task-specific guidelines, examples, exercises, discussion sessions, and a custom annotation tool containing videos, enriched narrations, and detailed instructions. The paper states that annotation had IRB approval. Quality control included a back-and-forth review process between annotators and authors, weekly meetings, and cross-verification of 1,000 randomly selected QA pairs. Inter-annotator agreement was measured with Krippendorff’s Alpha, with $\alpha = 0.78$, which the paper interprets as substantial agreement.

4. Hallucination-triggering design and evaluation protocol

EgoIllusion constructs hallucination-triggering questions through three strategies: Prompt Injection (PI), Adversarial Sampling (AS), and Manipulating Temporal Order (MTO). PI uses misleading or adversarial instructions to exploit a model’s tendency to trust language prompts over visual evidence; one example transforms “Where did the person leave their keys?” into “Why did the person leave their hat?” AS replaces the true active object with a non-active object that is visible in the scene, making the false alternative visually and semantically plausible. MTO changes event order or mismatches action and sound sequences to test whether a model understands chronology rather than plausibility (Seth et al., 18 Aug 2025).

The evaluation protocol uses accuracy throughout. For closed-ended questions, outputs are normalized to “Yes” or “No” by string matching and accuracy is computed. For open-ended questions, the benchmark uses an LLM-as-judge procedure: GPT-4o is the main judge, Gemini-Pro is also used to reduce judge bias, and the procedure first determines whether the response implicitly assumes the presence of an object and then independently judges factual correctness. The paper states that lower accuracy means more hallucination.

The benchmark evaluates 10 MLLMs: 8 open-source/open-weight and 2 proprietary/closed-source. The table also includes human evaluation, which creates a minor count ambiguity if human results are counted as a row. The evaluated machine models include Qwen2.5VL, VideoLlama3, InternVideo / InternVideo2.5, LLaVA-NEXT, LLaVA-OV 0.5B, LLaVA-OV 7B, ImageBind-LLM, MiniCPM / MiniCPM-o 2.6, VideoLlama2, Gemini-Pro / Gemini-1.5 Pro, and GPT-4o, with some wording inconsistencies across main text and appendix. The benchmark itself is evaluation-only: the appendix states that no training is required, experiments were run on 10 NVIDIA A6000 GPUs, and one benchmark inference run takes about 1–2 hours.

An important technical distinction is modality support. Some models are evaluated only visually, while audio-capable models are evaluated on AER. The paper later reports that under modality-confound experiments on 200 clips, performance drops below 50% under both audio and visual perturbations, with the drop worse for audio: 32% for Gemini and 28% for MiniCPM. This supports the paper’s claim that audio grounding remains particularly weak.

5. Empirical results, failure modes, and benchmark interpretation

The headline quantitative result is that even the strongest evaluated model remains well below human performance. Human performance is reported as $86.1 \pm 0.3$. The best machine score is Gemini-Pro at $C = \{c_1, \dots, c_n\},$0. Other overall results include MiniCPM at $C = \{c_1, \dots, c_n\},$1, GPT-4o at $C = \{c_1, \dots, c_n\},$2, InternVideo at $C = \{c_1, \dots, c_n\},$3, VideoLlama3 at $C = \{c_1, \dots, c_n\},$4, LLaVA-NEXT at $C = \{c_1, \dots, c_n\},$5, LLaVA-OV 0.5B at $C = \{c_1, \dots, c_n\},$6, LLaVA-OV 7B at $C = \{c_1, \dots, c_n\},$7, ImageBind-LLM at $C = \{c_1, \dots, c_n\},$8, Qwen2.5VL at $C = \{c_1, \dots, c_n\},$9, and VideoLlama2 at $D$0 (Seth et al., 18 Aug 2025).

Task-wise performance reveals substantial variation. EIR is among the hardest tasks: the best model there is MiniCPM at 57.3, while humans score 80.1. TR is highly unstable across models: LLaVA-OV 7B reaches 67.5, Qwen2.5VL 67.3, Gemini-Pro 60.8, and GPT-4o 47.5, while humans score 86.5. VOI is among the easiest for current models, with GPT-4o at 73.9 and humans at 88.4. AER is notably weak across all audio-capable models: VideoLlama2 reaches 52.6, Gemini-Pro 52.5, and humans 86.3. The paper’s broad conclusion is that perception is easier than reasoning, while audio is especially difficult.

Hallucination-inducing strategies also separate model behaviors. For PI, VideoLlama3 reaches 60.1, while Gemini-Pro and GPT-4o score 53.9 and 54.2. For AS, VideoLlama3 reaches 66.0, Gemini-Pro 64.9, and GPT-4o 62.1. For MTO, LLaVA-OV 7B reaches 67.5, Qwen2.5VL 67.3, Gemini-Pro 60.8, GPT-4o 59.7, MiniCPM 47.3, and VideoLlama2 38.9. The paper interprets MTO as exposing severe difficulty in chronological understanding for egocentric events.

Manual analysis of 1,000 incorrect responses yields six fine-grained error categories: perception errors, logical errors, procedural errors, spatial errors, factual errors, and no answer. The dominant category is perception error, accounting for 48.6% of Gemini 1.5 Pro’s mistakes and 43.7% of MiniCPM’s mistakes. The paper’s interpretation is that many apparent reasoning failures are rooted in bad grounding before reasoning begins. On closed-ended hallucinated questions, Gemini-1.5 Pro and MiniCPM also show a strong yes-bias, disproportionately answering “Yes” to plausible-but-false events or objects.

The paper’s representative cases are designed to show confident unsupported elaboration rather than ambiguous failure. In an EIR / PI example, the question “How did the person collect the nail?” has the correct answer “The person did not perform this action,” but the model responds with a fabricated description involving a fuel injector and a magnet. Similar failures appear in HOI and VOI through confident but unsupported affirmative answers. These cases reinforce the benchmark’s stated aim: to test whether models verify evidence before answering.

6. Subsequent use, limitations, and research significance

EgoIllusion’s most immediate downstream role is as a transfer benchmark for egocentric multimodal learning. EgoAVU evaluates Qwen2.5-Omni and two EgoAVU-Instruct-fine-tuned variants on EgoIllusion and reports accuracy improving from 56.32 for the base model to 60.36 for LoRA fine-tuning and 60.24 for full fine-tuning, corresponding to absolute gains of 4.04 and 3.92 points and relative gains of 7.2% and 7.0%. EgoAVU characterizes this as evidence that improved audio-visual grounding transfers beyond its in-house benchmark to prior egocentric benchmarks such as EgoTempo and EgoIllusion (Seth et al., 5 Feb 2026).

EgoIllusion also has clear limitations, many of which the benchmark paper states explicitly. It is an evaluation benchmark rather than a correction method, so it does not by itself mitigate hallucinations. It does not yet include speech modality, focusing instead on visual and non-speech auditory cues such as background sounds. Open-ended evaluation depends on LLM-as-judge. There are minor naming and formatting inconsistencies between the main text and appendix. Audio evaluation is possible only for models that support audio in the tested setup (Seth et al., 18 Aug 2025).

A common misconception is that EgoIllusion is principally an audio benchmark because it includes AER and environmental sounds. The paper does not support that reading. Audio is treated as a first-class hallucination source, but the benchmark is broader: it jointly measures object grounding, hand-object interaction, state change, episodic memory, temporal reasoning, and audio-event grounding under egocentric conditions. Another misconception is that the benchmark measures generic QA accuracy. Its own construction criteria and error analyses emphasize hallucination susceptibility under adversarial prompting and scene-plausible substitutions.

A plausible implication is that EgoIllusion has become important not only because of its scale—more than 1,400 videos and 8,000 QA pairs—but because it operationalizes a specific failure mode of MLLMs in first-person video: fluent responses that remain insufficiently grounded in multimodal evidence. In that sense, its role within egocentric research is analogous to a stress test for grounded reasoning. It provides a standardized way to separate performance on conventional egocentric understanding from robustness against hallucination-triggering prompts, object substitutions, and temporal distortions.