EgoExoBench: Cross-View Video Benchmark

Updated 7 July 2026

EgoExoBench is a benchmark that evaluates first- and third-person video understanding by integrating semantic alignment, viewpoint association, and temporal reasoning.
It combines six diverse ego-exo paired datasets into 7,330 multiple-choice questions across eleven subtasks, targeting cross-view inference challenges.
Empirical findings reveal that current multimodal large language models underperform compared to humans, underscoring difficulties in cross-view visual grounding and reasoning.

EgoExoBench is a benchmark for first- and third-person view video understanding in multimodal LLMs. It is introduced as the first benchmark specifically built to test whether models can reason across egocentric and exocentric video views in a human-like way, rather than operating on a single viewpoint in isolation. Built from six publicly available ego-exo paired datasets, it contains 7,330 four-way multiple-choice question-answer pairs spanning eleven subtasks organized into three core challenges: semantic alignment, viewpoint association, and temporal reasoning (He et al., 24 Jul 2025).

1. Scope and conceptual framing

The benchmark is motivated by a capability that the paper treats as intrinsic to human intelligence: transferring and integrating knowledge across first-person and third-person viewpoints. The central claim is not merely that models should recognize actions in multiple camera setups, but that they should align what is seen from an embodied perspective with what is seen by an external observer, and vice versa. This includes learning from others’ demonstrations, identifying oneself or one’s actions from outside views, and preserving event structure across perspective changes (He et al., 24 Jul 2025).

EgoExoBench is therefore positioned against prior video benchmarks that advance single-view understanding—either egocentric or exocentric—but do not directly measure cross-view transfer, alignment, and joint reasoning. The benchmark’s three challenge families formalize this gap. Semantic alignment probes whether two clips from different viewpoints depict the same task, action, person, or object. Viewpoint association probes whether a model can translate spatial information between views. Temporal reasoning probes whether a model can infer ordering, progression, prediction, and relative skill across paired ego/exo streams.

This organization suggests a deliberately layered view of cross-view intelligence. The benchmark begins with semantic correspondence, moves to spatial translation between embodied and observer frames, and then escalates to sequence-level reasoning. A plausible implication is that EgoExoBench is intended not simply as a recognition benchmark, but as an evaluation of whether MLLMs can interleave visual grounding, linguistic interpretation, and cross-perspective inference.

2. Challenge structure and subtask taxonomy

EgoExoBench contains three major categories and eleven subtasks. Each subtask is instantiated as a four-choice multiple-choice question, but the underlying reasoning demand varies substantially across categories (He et al., 24 Jul 2025).

Challenge	Subtasks	Core focus
Ego–Exo Relation	Task Relation, Action Relation, Object Relation, Person Relation	Semantic alignment across viewpoints
Ego–Exo View Transition	Egocentric Wearer Identification, Direction Prediction, Body Part Action Understanding	Spatial and identity mapping between views
Ego–Exo Temporal Reasoning	Action Prediction, Action Order, Sequence Alignment, Skill Evaluation	Temporal inference across paired streams

Within Ego–Exo Relation, Task Relation asks whether clips from different views correspond to the same high-level task; Action Relation tests action-level correspondence; Object Relation focuses on the interacted object; and Person Relation asks for cross-view identity matching. These subtasks are designed so that distractors remain semantically close, preventing trivial background or scene matching.

Within Ego–Exo View Transition, Egocentric Wearer Identification asks the model to identify the camera wearer in a third-person scene, Direction Prediction requires translation of directional motion across viewpoints, and Body Part Action Understanding tests whether models can preserve body-part-specific semantics such as left-hand versus right-hand actions. This family operationalizes viewpoint transition as a spatial reasoning problem rather than a pure recognition problem.

Within Ego–Exo Temporal Reasoning, Action Prediction asks for next-action inference from partial video, Action Order asks for temporal ordering between clips, Sequence Alignment asks whether multi-step sequences match or differ, and Skill Evaluation asks for better-versus-worse performance relative to a reference. The paper emphasizes that these tasks can involve asynchronous or partially overlapping recordings, making them more demanding than synchronized clip comparison.

3. Construction pipeline and source datasets

EgoExoBench is assembled from six publicly available ego-exo paired datasets: Ego-Exo4D, EgoExoLearn, LEMMA, EgoMe, TF2023, and CVMHAT. The mixture is explicitly important because some of these datasets provide synchronized multi-view videos, whereas others provide asynchronous recordings. In the paper’s description, Ego-Exo4D, LEMMA, TF2023, and CVMHAT are synchronized, while EgoExoLearn and EgoMe are asynchronous; this allows the benchmark to test both aligned and non-aligned ego-exo reasoning (He et al., 24 Jul 2025).

The construction pipeline has three stages. First, clips are collected from the six datasets to cover diverse environments and activities, including kitchens, laboratories, sports fields, cooking, repair, and sports. Second, question-answer pairs are generated using three strategies. Annotation-derived QA is used when deterministic structured annotations are available. LLM-generated QA is used when richer or more natural question formulation is needed. Human-annotated QA is used especially for tasks requiring fine-grained spatial understanding, where the paper reports that current MLLMs are not reliable enough to automatically generate valid questions. Third, the resulting questions undergo quality assurance through consistency verification and vision-grounded filtering.

Two filters are structurally important. Consistency verification discards ambiguous or multiply valid questions by checking alignment with the original annotations. Vision-grounded filtering removes questions that can be answered from text alone. This filtering logic is central to the benchmark’s design, because the stated target is cross-view video reasoning rather than linguistic shortcut exploitation.

The benchmark also specifies subtask-level data provenance. Task Relation uses Ego-Exo4D, EgoExoLearn, and LEMMA; Action Relation uses LEMMA and EgoExoLearn; Object Relation uses LEMMA; Person Relation uses CVMHAT; Egocentric Wearer Identification uses TF2023; Direction Prediction uses Ego-Exo4D; Body Part Action Understanding uses Ego-Exo4D and EgoExoLearn; Action Prediction uses LEMMA and EgoMe; Action Order uses LEMMA; Sequence Alignment uses Ego-Exo4D; and Skill Evaluation uses EgoExoLearn and Ego-Exo4D. This mapping makes clear that EgoExoBench is a composite benchmark rather than a monolithic dataset.

4. Evaluation protocol and benchmark organization

EgoExoBench is framed entirely as a four-way multiple-choice question answering benchmark. Each item provides a question stem and four options, and the model must output the letter of the correct answer. The paper adopts this format because free-form video-language evaluation is described as difficult and noisy, whereas four-choice multiple-choice questions enable scalable and reliable scoring. Accuracy is the primary metric for both model and human evaluation (He et al., 24 Jul 2025).

The benchmark contains 7,330 MCQs in total, spanning three major categories, eleven subtasks, six source datasets, and both synchronized and asynchronous ego-exo video pairs. The appendix reports some subtask-level counts, including 212 Task Relation items from LEMMA, 108 from EgoExoLearn, and 237 from Ego-Exo4D; 612 Action Relation items from LEMMA and 216 from EgoExoLearn; and 444 Skill Evaluation items from EgoExoLearn plus 379 from Ego-Exo4D. The paper does not summarize every subtask count in the prose, but the overall total is explicitly stated.

The evaluation is zero-shot, with no fine-tuning. The paper reports results for 13 state-of-the-art MLLMs and standardizes the prompting format to include the question stem and labeled answer options, with an instruction to return only the answer letter. A rule-based extractor reads the final response. Human performance is evaluated on a subset of 30 questions per subtask, for a total of 330 questions, under both one-minute and three-minute settings. The main human baseline is the deliberate setting, where humans achieve 90.1% average accuracy.

The appendix also reports a preliminary sanity check on whether models can distinguish multiple video inputs as separate streams. On a 100-question identical-pair identification task, all tested models perform well above random chance. The paper uses this result to argue that the benchmark’s main difficulty is not basic multi-video input parsing, but the cross-view reasoning demanded by the tasks themselves.

5. Empirical findings, ablations, and failure modes

The central empirical result is that contemporary MLLMs perform substantially worse on EgoExoBench than humans, and that performance deteriorates most sharply on genuine cross-view reasoning rather than simpler single-view-style recognition. The paper reports that, among open-source models, Qwen2.5-VL-72B is strongest overall with 47.0% average accuracy in one reported table variant and 44.7% in the appendix table. Among closed-source models, GPT-o4-mini reaches 48.0% overall in the main table, while the appendix reports Gemini 2.5 Pro as the best overall model at 51.7% average accuracy. Against these values, the human baseline of 90.1% leaves a gap of roughly 42 points to the strongest model reported in the main text (He et al., 24 Jul 2025).

The paper’s qualitative interpretation is that current MLLMs remain much better at local semantic recognition than at perspective transfer. Semantic alignment itself remains only moderate. Viewpoint transition is especially difficult, with Egocentric Wearer Identification exposing pronounced weakness in many open-source systems. Temporal reasoning is weak overall, particularly for Action Order, Sequence Alignment, and Skill Evaluation. The benchmark therefore supports the claim that current systems are not yet robust cross-view reasoners even when they appear strong on conventional video QA.

A notable ablation concerns chain-of-thought prompting. The paper samples 100 questions per subtask and compares baseline zero-shot prompting with prompts that append “Let’s think step by step.” The reported result is that chain-of-thought usually hurts rather than helps. For Qwen2.5-VL-72B, the paper notes about a 20% drop on Person Relation and about a 19% drop on Action Relation. The interpretation given is that these tasks require tightly interleaved visual interpretation, linguistic reasoning, and cross-view mapping; a text-only reasoning scaffold can induce drift rather than improve grounding.

A second ablation studies cross-perspective reference video input on selected tasks. For Action Prediction, an additional reference clip from the other viewpoint helps consistently, with reported gains of +8.2 for Qwen2.5-VL-72B, +9.1 for InternVL3-78B, and +1.1 for GPT-4o. For Skill Evaluation, the effect is inconsistent or slightly negative, with reported changes of -1.5, -0.9, and -0.2 respectively. The paper argues that skill assessment depends on subtle, domain-specific judgments that additional visual context alone does not reliably resolve.

The paper’s failure analysis identifies four recurring error modes: semantic misalignment under viewpoint change, viewpoint confusion between the wearer and nearby individuals, temporal reasoning failures on asynchronous or partially overlapping clips, and body-part localization errors such as left/right confusion. These failure modes are used to support the larger conclusion that plain scale or generic prompting is insufficient for reliable ego-exo reasoning.

6. Relation to adjacent benchmarks and common points of confusion

EgoExoBench sits within a broader line of work on ego-exo video understanding, but it differs in both task scope and evaluation target. EgoExoLearn is a dataset built to bridge asynchronous procedural activity understanding between egocentric and exocentric viewpoints. It provides 747 total video sequences, 120 hours of footage, calibrated eye gaze for all egocentric videos, and benchmarks including cross-view association, cross-view action planning, cross-view referenced skill assessment, and cross-view referenced video captioning. EgoExoBench directly incorporates EgoExoLearn as one of its six source datasets, particularly for tasks involving asynchronous demonstration-follower relations and reference-based skill evaluation (Huang et al., 2024).

A separate point of confusion arises from EgoExoMem. That benchmark is described as the first benchmark for cross-view memory reasoning over synchronized egocentric and exocentric videos, and the paper explicitly notes that it is “also referred to in the query as ‘EgoExoBench,’ but the paper’s benchmark name is EgoExoMem.” EgoExoMem differs from EgoExoBench in both data organization and target task: it is a synchronized, multiple-choice video question answering benchmark centered on memory over time and reasoning across two complementary views, whereas EgoExoBench is a broader MLLM evaluation suite spanning semantic alignment, viewpoint association, and temporal reasoning across synchronized and asynchronous paired data (Liu et al., 18 May 2026).

Other adjacent resources emphasize still different problem formulations. EgoExo-Fitness focuses on synchronized egocentric and exocentric full-body action understanding in fitness scenarios, with benchmarks for action classification, action localization, cross-view sequence verification, cross-view skill determination, and guidance-based execution verification. Its distinctive contribution is interpretable action judgment through technical keypoint verification, natural-language execution comments, and action quality scores, rather than the cross-dataset MLLM reasoning emphasis of EgoExoBench (Li et al., 2024).

Taken together, these neighboring benchmarks clarify the niche that EgoExoBench occupies. It is not a single-domain dataset, not a purely synchronized memory benchmark, and not a fitness-specific assessment suite. Its defining role is as a composite evaluation resource for MLLMs that asks whether models can align semantics across perspectives, translate between ego and exo spatial frames, and reason over time across paired streams. This suggests that its primary significance lies in exposing a missing capability boundary in embodied and assistive AI: connecting “what I see” with “what others see” across space, time, and viewpoint.