EgoExoLearn: Asynchronous Ego-Exo Procedural Dataset

• EgoExoLearn is a dataset bridging asynchronous egocentric and exocentric procedural actions, characterized by differing time, place, and viewpoint between demonstrations and enactments.
• It comprises 747 video sequences—432 egocentric and 315 demonstration videos—with calibrated gaze data, detailed multimodal annotations, and skill assessments across daily-life and laboratory settings.
• The benchmark suite evaluates tasks like cross-view association, action anticipation, planning, and skill assessment, highlighting challenges in procedural alignment and view invariance.

EgoExoLearn is a large-scale real-world dataset for studying how procedural activities can be bridged between egocentric and exocentric viewpoints when the two recordings are asynchronous rather than synchronized. It is designed to emulate demonstration following: a participant watches an exocentric demonstration video and later performs the same task from a first-person viewpoint in a different environment, producing a setting in which time, place, and camera perspective all differ. The dataset spans 120 hours of daily-life and laboratory activity video, includes calibrated gaze and detailed multimodal annotations, and defines benchmarks for cross-view association, action understanding, skill assessment, and captioning. In this formulation, EgoExoLearn targets the broader problem of mapping an observed procedure into one’s own viewpoint for later execution, rather than merely matching simultaneous multi-camera observations (Huang et al., 2024).

1. Problem formulation and scope

EgoExoLearn is centered on a specific capability: bridging asynchronous procedural actions across an egocentric view and an exocentric demonstration view. The intended scenario is not synchronized multi-view capture of the same moment, but a more realistic instructional setting in which a person studies a demonstration and then reproduces the procedure later, from a first-person perspective, in another environment. The dataset motivates this as a foundational ingredient for embodied systems that should learn by observing humans in the real world, especially in daily assistance and professional support settings such as laboratory procedures (Huang et al., 2024).

A common misconception is to treat EgoExoLearn as a conventional multiview dataset. It is not. Prior ego-exo datasets are described as typically recording ego and exo video synchronously in the same environment, focusing on single actions, or lacking the multimodal annotations needed for instructional following. EgoExoLearn instead defines a harder setting in which the ego and exo streams differ not only by viewpoint but also by temporal alignment, environment, and execution details. This makes the dataset particularly relevant for procedural reasoning, transfer across views, and human-like observational learning (Huang et al., 2024).

The dataset also has a strong application orientation. The daily-life component targets activities such as cooking and household procedures, while the laboratory component targets specialized workflows in which errors are costly or dangerous. This domain composition makes the benchmark suitable for studying both familiar household procedures and technically constrained professional actions within the same ego-exo learning framework (Huang et al., 2024).

2. Dataset composition and data collection

EgoExoLearn contains 747 video sequences and 120 total hours of footage. The video inventory is divided into 432 egocentric videos totaling 96.5 hours and 315 demonstration videos totaling 23.5 hours. The data cover two domains: 5 daily-life task types and 3 specialized laboratory task types. Egocentric recordings were collected in 4 kitchens and 3 laboratories (Huang et al., 2024).

Component	Quantity
Video sequences	747
Total footage	120 hours
Egocentric videos	432 videos, 96.5 hours
Demonstration videos	315 videos, 23.5 hours

The collection protocol is explicitly designed to mirror human demonstration following. Before recording, participants complete a questionnaire about demographics and self-rated expertise. They are then shown one or more exocentric demonstration videos and study them until they feel ready. During task execution, participants wear Pupil Invisible Glasses so that the task is recorded from the egocentric viewpoint, and gaze calibration is performed both before and after the session. Participants are allowed, though not encouraged, to revisit the demonstration video during recording. This protocol produces ego videos that are instruction-conditioned rather than synchronized with the exocentric recording, which is the defining property of the dataset (Huang et al., 2024).

This collection design distinguishes EgoExoLearn from paired-camera datasets in a substantive way. The exocentric video is a reference demonstration, not merely an alternate camera angle of the same episode. A plausible implication is that cross-view transfer on EgoExoLearn necessarily couples viewpoint invariance with procedural abstraction, because the model must absorb the structure of an observed demonstration and relate it to a later first-person enactment.

3. Modalities and annotation schema

EgoExoLearn is multimodal in a strong sense. It provides egocentric video, exocentric demonstration video, calibrated gaze data for all egocentric videos, coarse-level language annotations, fine-level language annotations, verb and noun labels aligned to hand involvement, and skill-level annotations derived from pairwise human judgments (Huang et al., 2024).

The annotation pipeline has four stages. First, annotators mark coarse-level procedural steps such as “Prepare the pork” or “Suction filtration,” each with a start timestamp, end timestamp, action label, and language description explaining what is done, how it is done, and the purpose of the step. The dataset contains 39 coarse action categories, with 41.2 coarse annotations per video on average and an average coarse segment length of 21.5 seconds (Huang et al., 2024).

Second, annotators add fine-level language annotations for atomic actions such as “take knife” or “pull syringe plunger.” These descriptions emphasize which hand is used, what object is used, and why it is used. After manual checking, the dataset contains 95 verb classes and 254 noun classes. The verb and noun labels are extracted from language annotations using NLTK and spaCy, mostly constrained to the Ego4D taxonomy, with additional manually introduced labels when needed. Because hand involvement is annotated, verbs and nouns can be associated separately with the left or right hand (Huang et al., 2024).

Third, descriptions written in native languages are translated into English using ChatGPT and Google Translation API, after which the verbs and nouns are parsed and manually validated. Fourth, skill is annotated by pairwise ranking rather than absolute scoring. Annotators compare two videos showing the same action and decide which is performed with higher skill. The authors prepared 40,191 video pairs, each labeled by 4 annotators; after filtering out pairs with fewer than 3 consistent votes, 34,239 valid pairs remain. This relative-skill design is used later for the cross-view referenced skill-assessment benchmark (Huang et al., 2024).

The dataset is also described as the first egocentric dataset to provide temporal bounded language captions, annotated cross-view associations, multi-label video segments, and calibrated gaze. This combination is important because it enables research that jointly uses action labels, free-form language, hand-specific semantics, gaze, and procedural structure within the same asynchronous ego-exo setting (Huang et al., 2024).

4. Benchmark suite and evaluation protocols

EgoExoLearn introduces a benchmark suite for testing whether models can bridge procedural activities across views. The paper highlights cross-view association, cross-view action understanding, cross-view referenced skill assessment, and cross-view referenced captioning, with action understanding subdivided into action anticipation, action planning, and action segmentation (Huang et al., 2024).

The cross-view association task measures whether an egocentric and an exocentric clip depict the same procedural action. It is formulated as a multiple-choice problem with two directions: ego2exo and exo2ego. Each query is paired with 20 samples, and performance is measured by Top-1 Accuracy. Ground-truth ego-exo pairs are constructed through a five-stage process: scenario matching, noun and verb matching, sentence matching with an LLM, negative sampling, and two-round manual verification. Importantly, the training split does not provide these pairs, preserving the challenge of learning association from unpaired data (Huang et al., 2024).

The cross-view action anticipation task asks the model to predict the next fine-level action at horizon $\tau = 1$ second. It uses fine-level verb and noun labels intersected across ego and exo views to define a shared closed set, with rare categories filtered out. The evaluation metric is class-mean Top-5 recall, following EPIC-KITCHENS-style evaluation. The companion cross-view action planning task asks the model to generate the next $K = 8$ coarse-level action steps, evaluated by ED@K, where lower edit distance is better (Huang et al., 2024).

The cross-view referenced skill assessment benchmark extends pairwise skill ranking by adding a reference exocentric demonstration clip. Given two ego clips, $C_{ego1}$ and $C_{ego2}$, and one exo reference clip, $C_{exo}$, the model predicts which ego clip is more skilled. The paper focuses this benchmark on Egg cracking, Peeling, Stir-fry, and Cutting. Baselines incorporate the exocentric reference either with a triplet loss, which encourages the more skilled ego clip to be closer to the demonstration, or with a relation network that operates on concatenated ego/exo features (Huang et al., 2024).

The paper notes that cross-view referenced captioning and some details of action segmentation are covered in supplementary material. Even so, the benchmark definition makes clear that EgoExoLearn is not restricted to classification or retrieval; it is intended as a general testbed for cross-view procedural understanding under asynchronous observation.

5. Baseline behavior and empirical lessons

The benchmark results emphasize that asynchronous ego-exo bridging is difficult. In cross-view association, the authors evaluate zero-shot models such as EgoVLP, LaViLa, and InternVideo, alongside fine-tuned variants trained with ego-only, exo-only, or joint ego-exo data. Zero-shot models exhibit only limited bridging ability, although EgoVLP performs relatively well among the pretrained baselines. Fine-tuning on both ego and exo data performs best, and a gaze-cropped ego-input variant further improves results. In the reported test setting, the ego-exo model with gaze reaches 55.3 / 51.1 for ego2exo / exo2ego association, outperforming weaker single-view alternatives (Huang et al., 2024).

The action-understanding results show a similar pattern. The paper compares Unsupervised Domain Adaptation (UDA), Knowledge Distillation (KD), and Co-training (CT) for cross-view anticipation and planning. There are substantial performance gaps when a model trained on one view is evaluated on the other. Gaze consistently narrows this gap even without complex adaptation, UDA and KD improve over naive transfer, and co-training performs best overall because it directly exploits both views (Huang et al., 2024).

For skill assessment, the empirical message is more restrained. Using gaze improves conventional pairwise skill ranking substantially. Adding the exocentric reference via the relation-network or triplet-loss formulations improves results further, but only modestly. The best results combine reference video + gaze, yet the limited incremental gain from the exocentric reference indicates that current models do not fully exploit cross-view demonstration information for procedural quality judgment (Huang et al., 2024).

These findings support several dataset-level conclusions. First, the difficulty is not merely appearance mismatch; it includes asynchronous mismatch and execution variability. Second, gaze is a consistently useful auxiliary signal across association, anticipation, planning, and skill assessment. Third, simple joint ego-exo training remains a strong baseline. Fourth, current adaptation methods are not sufficient to solve fine-grained procedural alignment across viewpoints (Huang et al., 2024).

6. Role in subsequent research and continuing challenges

Subsequent work has treated EgoExoLearn as a benchmark for increasingly specialized cross-view learning problems. In "Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency" (Shi et al., 10 Mar 2026), EgoExoLearn is described as an existing benchmark for bridging asynchronous Ego and Exo procedural activities, containing 120 hours of videos in daily and professional scenes with fine and coarse-level noun/verb classes, timestamps, and descriptions. That work uses the dataset’s official cross-view action anticipation benchmark and follows the standard Exo2Ego and Ego2Exo settings, while reframing the problem as Test-time Ego-Exo Adaptation for Action Anticipation. This places EgoExoLearn not only as a dataset but also as a reference benchmark for online adaptation under substantial temporal-spatial view gaps (Shi et al., 10 Mar 2026).

EgoExoLearn has also been used to study generalization beyond seen viewpoints. "Towards Generalizing Temporal Action Segmentation to Unseen Views" (Bahrami et al., 3 Apr 2025) evaluates an unseen-view temporal action segmentation protocol on EgoExoLearn using the dataset’s zero-shot exo-only protocol. Because only one training view per sequence is available there, that work applies only its action loss, not its sequence loss, and reports improvements over the prior EgoExoLearn method in both validation and test settings, including stronger test ego and test exo $F1@Avg$ with both I3D and DinoV2 features (Bahrami et al., 3 Apr 2025).

This pattern suggests that EgoExoLearn has become a substrate for multiple problem formulations: association, anticipation, planning, segmentation, and reference-conditioned assessment. At the same time, the dataset itself exposes unresolved challenges. The original study identifies asynchronous mismatch, different environments, and edited demonstrations as fundamental difficulties; later work explicitly treats EgoExoLearn as a natural stress test for temporal-spatial view-gap adaptation (Huang et al., 2024). Taken together, these results position EgoExoLearn as a benchmark for learning procedural correspondence across viewpoint, time, and environment, rather than a narrowly defined multiview video corpus.