Ego-EXTRA: Egocentric Assistance Dataset

• Ego-EXTRA is an egocentric video-language dataset capturing real-time expert-trainee dialogues during procedural tasks.
• It employs a Wizard of Oz protocol with wearable Project Aria devices and generates over 15k high-quality visual QA sets.
• Empirical results show current multimodal models achieve ~33% accuracy versus human performance at ~90% in assistive scenarios.

Ego-EXTRA is a video-language egocentric dataset for EXpert-TRAinee assistance built to study wearable assistants that provide expert-level natural-language guidance during procedural activity execution. It contains 50 hours of unscripted egocentric video in which trainees perform real tasks while remote experts observe only the trainee’s first-person view and intervene through dialogue, either reactively or proactively. The dataset is paired with a benchmark of more than 15k high-quality Visual Question Answer sets derived from those interactions, and its central empirical finding is that current multimodal LLMs remain far from human-level performance in this assistance setting (Ragusa et al., 15 Dec 2025).

1. Definition and research scope

Ego-EXTRA is designed around a specific assistive regime: a trainee executes a procedural task while a remote expert, acting as a wearable intelligent assistant, observes the activity exclusively through the trainee’s egocentric stream and communicates through natural language. This setup targets questions and interventions of the kind that arise in situated assistance, such as identifying objects, selecting tools, deciding the next step, checking whether an action is correct, or warning about an incipient mistake. The intended application space includes tasks such as repairing a bike, assembling furniture, cooking, and baking, where visual grounding and procedural state tracking are inseparable from language understanding (Ragusa et al., 15 Dec 2025).

The dataset is not organized as post-hoc narration, third-party description, or synthetic instruction overlay. Its defining property is temporally aligned, two-way expert-trainee interaction during task execution. The paper positions this as the key missing ingredient in prior egocentric resources: existing datasets may contain procedures, actions, narrations, or language annotations, but they do not capture realistic dialogue in which expert feedback is grounded in the user’s current first-person visual context. A plausible implication is that Ego-EXTRA is less a general-purpose egocentric corpus than a specialized benchmark for assistive egocentric video-language systems.

2. Data collection protocol and corpus composition

The collection protocol follows a Wizard of OZ design. The trainee physically performs the task while wearing a custom rig with Project Aria glasses, a smartphone aligned to a similar viewpoint, and earbuds. The expert sits in a separate room, watches the trainee’s egocentric feed on a laptop, and communicates through the earbuds. This separation is structural rather than incidental: it forces the expert to rely only on the trainee’s first-person visual stream, mirroring the sensory constraints of a wearable assistant (Ragusa et al., 15 Dec 2025).

Two interaction protocols are used. In the Pro-Active protocol, the expert is instructed to intervene freely, suggest steps, correct errors, and provide any useful information. In the On-Demand protocol, the trainee acts more autonomously and the expert answers when asked, except when a dangerous or clearly erroneous action is about to occur. The dataset intentionally minimizes proactive sessions, with an approximate 1:3 ratio between Pro-Active and On-Demand videos. The two protocols differ substantially in dialogue density: Pro-Active videos average 29.64 minutes, with experts speaking 82.4% of the words and an average of 264 turns per video, whereas On-Demand videos average 23.41 minutes, with experts speaking 61.39% of the words and an average of 142 turns per video.

The procedural domain spans 4 scenarios and 10 activities. The scenarios are bike workshop, kitchen, bakery, and assembly. Examples explicitly mentioned include replacing bike brake pads, cooking a tart, and assembling a chair.

Property	Value
Total videos	123
Total duration	50 hours
Activities	10
Scenarios	4
Resolution	1408 × 1408
Frame rate	15 fps
Average video length	25.24 minutes
Mean conversation turns per video	177.76
Participants	33 trainees, 4 experts
Age range	18 to 52 years

These corpus statistics matter because Ego-EXTRA is neither a short-clip dataset nor a narrowly scripted lab collection. Its videos are long enough to sustain genuine procedural development, and the dialogue density indicates that the language stream is not sparse metadata but a primary signal.

3. Multimodal instrumentation, synchronization, and dialogue structure

Ego-EXTRA is multimodal at both the trainee and expert sides. The trainee’s Project Aria setup records RGB, SLAM cameras, eye-tracking cameras, IMUs, magnetometer, barometer, GPS, WiFi, and Bluetooth; the custom acquisition profile includes an RGB camera at 1408 × 1408 and 15 FPS, SLAM cameras at 30 FPS, and eye-tracking cameras at 30 FPS. The expert station is equipped with a Tobii Pro Fusion Bar, which records the expert’s gaze while viewing the live egocentric feed (Ragusa et al., 15 Dec 2025).

Post-processing includes sending the Aria data to Machine Perception Services (MPS), synchronizing Aria streams with expert video and bidirectional audio, projecting expert gaze into the Aria reference frame, transcribing the conversations, translating them into English using Llama 3.1 405B, and manually verifying translation quality. Each conversational turn is timestamped. Synchronization is supported by a QR-code-based calibration step before acquisition and a countdown at session start. The expert-to-trainee gaze projection is described with a fixed $3 \times 3$ homography matrix $H$.

The dialogue structure is central to the dataset’s semantics. The transcripts record alternating T: and E: turns, and the linguistic content is heavily grounded in the current visual scene. The paper identifies 13 main categories of trainee questions. Prominent categories include instructional/procedural questions such as “What do I do now,” clarifications such as “What is the color of the inside?,” and comparisons such as “Clockwise or counterclockwise?” Other frequent categories include location, removal, insertion, and troubleshooting. Less frequent categories, including confirmation, tool selection, purpose, alignment, suitability, and measurement, still exceed 800 instances each. The language is rich in pronouns and ambiguous references—“it,” “this,” “another,” “more,” “less”—which makes visual grounding obligatory rather than optional.

4. Benchmark construction and task formulation

The benchmark is built from the recorded conversations as a multiple-choice visual question answering task. The paper distinguishes a QA pair, consisting of a question and its correct answer, from a QA set, which contains the question, the correct answer, and four distractors, yielding a five-way multiple-choice instance with exactly one correct option (Ragusa et al., 15 Dec 2025).

Benchmark construction proceeds in three stages. First, Llama 3.1 405B is prompted to extract trainee questions and expert answers from the transcripts, correct transcription and grammar issues, and generate four plausible wrong answers for each pair. Second, human annotators validate the resulting QA sets using labels such as Acceptable, To be Discarded, Transcription Error, and Requires Manual Revision. Third, annotators inspect the corresponding video clips and assess whether the question is visually grounded. Workers are filtered by an above-90% historical approval rate and a perfect qualification score, and each QA item is reviewed by 5 independent workers. Approximately 25% of initial QA candidates are discarded during textual validation, and 28% are discarded during video-grounding validation.

At inference time, the main multimodal setup provides the model with the question, five answer options, and a video clip immediately preceding the question timestamp. The explicit temporal context is

$\delta = 5s$

and 8 frames are sampled uniformly from that 5-second clip. The expected model output is the index of the correct answer in $\{1,2,3,4,5\}$. Formal notation is deliberately minimal: beyond $H$ and $\delta=5s$, the paper does not introduce a more elaborate mathematical task definition. This suggests that Ego-EXTRA is benchmark-driven rather than method-driven.

5. Empirical results and difficulty profile

The benchmark is evaluated with both language-only LLMs and video-language MLLMs. The strongest average result among the explicitly reported MLLMs is LLaVa-OneVision at 33.06% average accuracy, followed by Qwen 2.5-VL at 31.11%, while the best language-only model reported is Llama 3.1 Instruct 70B at 26.65%. The human baseline, computed on 217 questions, reaches 89.65% average accuracy. The gap between the best reported MLLM and human performance is therefore more than fifty percentage points (Ragusa et al., 15 Dec 2025).

System	Average accuracy
Llama 3.1 Instruct 70B	26.65
DeepSeek-R1 Turbo	23.51
LLaVa Video	28.55
Qwen 2.5-VL	31.11
LLaVa-OneVision	33.06
Sample Human Baseline	89.65

The scenario-wise pattern is also uneven. Qwen 2.5-VL is strongest in Kitchen with 35.87%, while LLaVa-OneVision leads on average and attains 35.77% in Kitchen as well. These numbers indicate that even the best models remain only modestly above the five-way random baseline of 20% in many settings.

Several ablations clarify what the benchmark demands. Uniformly sampling 8 frames over the 5-second clip is consistently stronger than using only the last 8 frames before the question timestamp or a single frame at the question timestamp. Increasing the temporal window from 5 s to 15 s or 30 s degrades LLaVa-OneVision from 33.06% to 28.69% and 29.78%, respectively. This suggests that longer context is not automatically beneficial for current models, and that temporally localized procedural evidence is often more useful than broader but noisier context.

Transcript context helps, but it does not replace vision. For LLaVa-OneVision 7B, the QA-only setup reaches 16.62%, QA + Transcript reaches 29.26%, QA + Video reaches 33.06%, and QA + Video + Transcript reaches 36.17%. This suggests that dialogue memory and visual evidence are complementary, with video remaining the stronger single modality. The paper’s broader conclusion is that current models struggle with precisely the capabilities that expert-level assistance requires: visual grounding of ambiguous references, fine-grained procedural state tracking, comparison questions, tool and part identification, and temporally localized reasoning.

6. Relation to prior resources, limitations, and significance

The paper positions Ego-EXTRA against existing egocentric datasets such as EPIC-Kitchens-100, Ego4D, Ego-Exo4D, CaptainCook4D, LEMMA, MECCANO, Assembly-101, and HoloAssist. Its main distinction is not simply that it contains procedures or language, but that it records real-time expert-trainee dialogue during task execution while the expert observes only the trainee’s egocentric stream. The paper explicitly describes it as the first dataset of egocentric videos centered around natural vision-language dialogue interactions between experts and trainees aimed at evaluating user assistants in procedural tasks (Ragusa et al., 15 Dec 2025).

This positioning implies several important negatives. Ego-EXTRA is not primarily a post-hoc narration dataset, not a pure action-recognition resource, and not a general conversational corpus detached from embodiment. Its benchmark is also deliberately narrower than full assistant deployment: it evaluates multiple-choice answer selection, not full open-ended interaction management, long-horizon planning, or autonomous intervention policy generation. Audio conversations are not publicly shared for privacy reasons; the released language signal is the transcript and its English translation. The benchmark construction also filters out irrelevant proactive turns, reflecting the fact that natural dialogue contains substantial non-benchmarkable material.

The dataset’s significance lies in making expert assistance a first-class evaluation target for egocentric video-language research. A plausible implication is that it shifts emphasis from passive understanding toward interactive procedural competence. In that respect, Ego-EXTRA functions simultaneously as a dataset, a benchmark, and a diagnostic instrument: it exposes the mismatch between current video-LLM performance and the much higher standard implied by expert human guidance in real procedural environments.