EgoExoAsk Benchmark

• The paper introduces a novel evaluation paradigm where generated questions are designed to elicit segment-specific, hidden expert insights rather than just visible facts.
• EgoExoAsk is a benchmark comprising 27,666 QA pairs derived from formatted expert commentary in Ego-Exo4D, focusing on expert knowledge elicitation.
• A retrieval-based protocol using cosine similarity between question and expert comment embeddings validates question quality and guides retriever training.

Searching arXiv for the primary paper and closely related ego/exo benchmarks to ground the article. EgoExoAsk is a benchmark and dataset for evaluating video question generation in terms of expert knowledge elicitation rather than answerability. Its central premise is that, for a video segment $s$, a good generated question $q$ is one that makes a human expert reveal segment-specific, otherwise invisible knowledge, rather than merely restating visible facts. Built from Ego-Exo4D expert commentary, EgoExoAsk operationalizes this objective with a retrieval-based protocol in which generated questions are scored by whether they retrieve the correct expert comment from a hard negative pool (Zhang et al., 17 Dec 2025).

1. Conceptual scope and motivation

EgoExoAsk is proposed against what the paper identifies as a misalignment in standard VQG and VideoQA evaluation. In conventional setups, questions are typically judged by whether they can be answered from the video or by a model. EgoExoAsk instead targets the interviewing function of questions: a generated question should uncover invisible or implicit expert knowledge behind a segment, elicit expert commentary specific to the observed action, and support quantitative comparison without requiring repeated consultation with human experts (Zhang et al., 17 Dec 2025).

The formal reframing is explicit. Traditional VQG is written as

$q = \mathcal{M}(a, s), \quad a \in \mathcal{S},$

where questions are generated for given answers. EgoExoAsk instead treats expert-knowledge elicitation as

$q = \mathcal{M}(s),$

where the desired answer lies in $\mathcal{S}'$, the hidden expert knowledge (Zhang et al., 17 Dec 2025). This shifts evaluation from visible-content QA toward prompting latent domain expertise.

A common misconception is to treat EgoExoAsk as a VideoQA benchmark in another form. The benchmark is structurally different: the question is the model output, the answer is expert commentary rather than a short ground-truth label, and scoring is retrieval-based rather than answer-generation-based. The paper therefore positions EgoExoAsk not as a new answerability test, but as an instrument for measuring whether a model can ask questions that would make an expert reveal useful unseen knowledge (Zhang et al., 17 Dec 2025).

2. Dataset origin and construction pipeline

EgoExoAsk is constructed from the expert commentary annotations of Ego-Exo4D, a large-scale multimodal ego-exo dataset centered on skilled human activity. Ego-Exo4D provides synchronized egocentric and exocentric video together with expert commentary across domains such as cooking, music, soccer, health, basketball, dance, bike repair, and rock climbing; EgoExoAsk uses that commentary as its source of expert knowledge (Grauman et al., 2023).

The construction pipeline has three stages. First, original expert comments are reformatted. Because the raw comments are often transcribed speech that may be informal, redundant, or shallow, Qwen3-32B is used to make them more formal, consolidated, grounded, and informative. Comments that are too short or merely evaluative are filtered out, with GPT-4o used to double-check exclusions. Second, a question $q_{t,i}$ is generated for each formatted comment $c_{t,i}$. This generation uses atomic descriptions from a temporal window $[t_{t,i}-w, t_{t,i}]$ with $w=7$ seconds, and the prompt is conditioned on the TYPE label, either [Good Execution] or [Tips for Improvement]. Third, the generated question is verified and, if necessary, regenerated according to explicit rules: it must be based on observable actions, not too general, not too overlapping with the expert commentary, and restricted to one question focus with no multiple question words (Zhang et al., 17 Dec 2025).

The resulting dataset contains 27,666 QA pairs. Its split is also task-specific: 21,240 QA pairs are used for retriever training, 3,177 QA pairs are designated as seen expert knowledge for some VQG settings, and 3,099 QA pairs form the validation benchmark. Those 3,099 QA pairs are tied to 1,237 video segments and serve as the final benchmark for VQG performance measurement (Zhang et al., 17 Dec 2025).

This construction makes the commentary—not the visible action label—the target of elicitation. The benchmark examples illustrate that the generated questions are meant to probe reasoning, technique, or improvement points implied by expert commentary, rather than visible object identity. The result is a dataset oriented toward coaching-relevant, performance-relevant, and explanation-relevant questioning (Zhang et al., 17 Dec 2025).

3. Retrieval-based simulation of expert communication

The benchmark replaces repeated expert interaction with question-to-answer retrieval. For each segment $s_{t,j}$, a generated question is treated as successful if, when encoded and matched against a segment-specific retrieval pool, it ranks the correct expert commentary above hard negatives. The retrieval pool $q$0 contains positive comments $q$1 attached to that segment and negative comments $q$2 sampled from other segments in the same video, other videos in the same scenario, and random other comments if needed (Zhang et al., 17 Dec 2025).

This protocol is intended to simulate interviewer–expert communication. The interviewer asks a question $q$3, the expert recalls an answer/comment $q$4, and the evaluator checks whether the retrieved comment matches the segment-specific expert knowledge. The hard-negative design is essential because it prevents trivial success based on broad topic overlap alone; the question must distinguish the correct expert commentary from nearby comments within the same activity context (Zhang et al., 17 Dec 2025).

The retrieval criterion is stated directly: $q$5 with cosine similarity. A question is counted as “good” at cutoff $q$6 if any positive comment appears in the top-$q$7 ranked results. Recall is then defined as

$q$8

where $q$9 is the number of good questions at cutoff $q = \mathcal{M}(a, s), \quad a \in \mathcal{S},$0 and $q = \mathcal{M}(a, s), \quad a \in \mathcal{S},$1 is the total number of queries. The paper also reports mean rank and median rank, where rank is the best rank among all positive comments for a query (Zhang et al., 17 Dec 2025).

This design makes the benchmark a proxy for elicitation quality rather than linguistic plausibility. A plausible implication is that EgoExoAsk measures whether a question is discriminative with respect to hidden expert knowledge, not merely whether it is fluent or generic.

4. Retriever training and formal evaluation machinery

The retriever is an embedding-based QA matcher $q = \mathcal{M}(a, s), \quad a \in \mathcal{S},$2 trained on EgoExoAsk question-comment pairs. The base encoder is all-MiniLM-L6-v2, trained on the 21,240 QA pairs in the training split with batch size 512, 10 epochs, AdamW, a linear scheduler, and learning rate 5e-5 (Zhang et al., 17 Dec 2025).

Its training objective is the contrastive loss

$q = \mathcal{M}(a, s), \quad a \in \mathcal{S},$3

with $q = \mathcal{M}(a, s), \quad a \in \mathcal{S},$4 the batch size, $q = \mathcal{M}(a, s), \quad a \in \mathcal{S},$5, and $q = \mathcal{M}(a, s), \quad a \in \mathcal{S},$6 cosine similarity (Zhang et al., 17 Dec 2025).

The paper justifies the retriever as a proxy for expert response behavior. If a question retrieves the correct expert comment, it is treated as evidence that the question is good at eliciting that knowledge. This proxy is empirically supported by held-out retrieval results: the zero-shot retriever obtains R@1 0.1347, R@5 0.2600, R@10 0.3286, and MeanR 401.22, whereas the trained retriever reaches R@1 0.3300, R@5 0.5512, R@10 0.6433, and MeanR 70.8 (Zhang et al., 17 Dec 2025).

These numbers matter methodologically. They support two claims made in the paper: first, the QA mapping is learnable; second, the questions are not simply leaking answer text. In other words, the benchmark is not reduced to lexical overlap. It evaluates whether question form and segment-specific expert knowledge can be aligned through retrieval.

5. Experimental behavior of the metric

The paper evaluates multiple VLLMs—Qwen2.5-VL, InternVideo2.5, and Video-LLaVA—under three question-generation conditions: naive (video only), w/ caption (video plus atomic descriptions), and w/ RAG (video plus retrieved seen expert knowledge via atomic descriptions). Both exo and ego views are tested (Zhang et al., 17 Dec 2025).

The empirical pattern is presented as evidence that the metric behaves sensibly. Gold questions drawn directly from EgoExoAsk achieve the best retrieval scores and serve as an upper bound, while Random questions perform worst and serve as a lower bound. More importantly, richer input context is rewarded: adding atomic descriptions improves performance over naive generation, with w/ caption improving R@1 and MeanR by about 0.03 and 1.5 on average over naive. For naive generation, a higher frame rate often helps, suggesting that more visual detail leads to better questioning. By contrast, RAG is not always better than naive generation, which the authors attribute to retrieved context not always being domain-appropriate (Zhang et al., 17 Dec 2025).

The qualitative examples are aligned with this interpretation. A specific question about adding salt at a certain stage retrieves the matching expert comment at rank 1; more specific movement-related questions retrieve relevant climbing commentary higher in the ranking; generic questions are judged worse. The overall conclusion is that the metric aligns with VQG settings and rewards richer contextual access, which the paper takes as evidence that the benchmark is measuring the intended capability (Zhang et al., 17 Dec 2025).

A further implication is that EgoExoAsk is sensitive not just to linguistic quality, but to context selection and question targeting. This suggests that future VQG systems for expert elicitation may need to optimize specificity and procedural relevance rather than general informativeness alone.

6. Position within ego/exo benchmark research

EgoExoAsk occupies a distinct position within the broader ego/exo literature. EgoBlind evaluates open-ended, timestamp-restricted video QA for blind users in first-person daily-life assistance, emphasizing user-intent reasoning, temporal context tracking, and relative spatial reasoning from the wearer’s perspective (Xiao et al., 11 Mar 2025). Ego-EXTRA benchmarks expert-to-trainee assistance in real procedural tasks through natural dual-agent dialogue grounded in egocentric execution, using multiple-choice VQA derived from expert-trainee conversations (Ragusa et al., 15 Dec 2025). EgoExoAsk differs from both: it does not evaluate answering assistance questions, nor selecting a correct option, but generating a question that would induce expert commentary not explicitly visible in the segment (Zhang et al., 17 Dec 2025).

Its closest substrate is Ego-Exo4D. That dataset’s expert commentary is explicitly designed to capture what is effective or ineffective, why a movement matters, and where improvement is needed, with 117,812 timestamped commentary segments and 2–5 expert perspectives per take in many cases (Grauman et al., 2023). EgoExoAsk can therefore be understood as converting one of Ego-Exo4D’s most distinctive annotation layers into a benchmark for question generation. Rather than consuming expert commentary as supervision for answering, it treats commentary as latent knowledge to be elicited.

This shift has broader significance for evaluation methodology. Standard VideoQA emphasizes answer correctness, factual visual grounding, and visible content. EgoExoAsk instead scores whether a generated question can unlock expert insight. A plausible implication is that it provides a benchmark for systems intended to support coaching, interview assistance, or structured knowledge capture from domain experts, especially in settings where expert time is limited and the value of a question lies in what it reveals rather than in whether it can be answered directly from pixels (Zhang et al., 17 Dec 2025).