Exo2Ego: Exocentric to Egocentric Transfer

• Exo2Ego is a family of research problems focused on translating third-person (exocentric) observations into detailed first-person (egocentric) representations across various tasks.
• It leverages techniques like multi-view synthesis, latent space alignment, and structured intermediate representations (e.g., HOI masks, 3D hand poses) for accurate cross-view translation.
• Practical implementations address both generative and predictive challenges, enhancing applications in video prediction, procedural learning, and multimodal understanding.

Searching arXiv for recent Exo2Ego-related papers to ground the article in current literature. Exo2Ego denotes exocentric-to-egocentric transfer: given third-person observations, learn or infer the corresponding first-person representation, image, video, object mask, caption, memory, or action plan. In the recent literature, the term is used for exocentric-to-egocentric cross-view translation (Luo et al., 2024), cross-view video prediction in which an exocentric video, the first frame of the corresponding ego-centric video, and textual instructions are used to generate future ego-centric frames (Xu et al., 16 Apr 2025), and exocentric knowledge guided multimodal LLMs for egocentric video understanding (Zhang et al., 12 Mar 2025). This breadth is characteristic: Exo2Ego is less a single benchmark than a family of cross-view problems unified by the attempt to map what an external camera observes to what the actor sees or does.

1. Semantic scope and task formulations

The literature uses Exo2Ego across several technically distinct formulations. In representation transfer, the objective is to pre-train on large third-person corpora and transfer to first-person downstream tasks; in object correspondence, the objective is to map an exocentric query object mask to its egocentric counterpart; in image and video generation, the objective is to synthesize egocentric imagery from exocentric observations; in procedural learning, the objective is to transfer captions, plans, or skill assessments across views; and in world modeling or memory reasoning, the objective is to use exocentric structure to improve egocentric prediction or recall (Li et al., 2021).

Task family	Representative formulation	Representative papers
Representation transfer	Third-person pre-training for first-person tasks	(Li et al., 2021, Zhang et al., 12 Mar 2025)
Object correspondence	Predict $\hat m^{ego}$ from $(I^{ego}, I^{exo}, m^{exo})$	(Fu et al., 2024)
Image/video generation	Generate ego view from exo image or video	(Luo et al., 2024, Xu et al., 16 Apr 2025, Park et al., 22 Jun 2025, Mahdi et al., 25 Nov 2025)
Procedural understanding	Cross-view association, planning, captioning, imitation assessment	(Huang et al., 2024, Qiu et al., 31 Jan 2025, Ohkawa et al., 2023, Shi et al., 7 Apr 2025)
Memory/world modeling	Use exo evidence for ego memory or ego world models	(Tran et al., 14 May 2026, Liu et al., 18 May 2026)

Several papers provide explicit problem statements. In EgoExo-Gen, the formulation is: given $\mathcal{V}_{\text{exo}}$, $g_1$, and $\mathcal{T}$, predict the ego-centric video $\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$ without any camera pose or depth information, with future ego-view hand-object interaction masks as an intermediate representation (Xu et al., 16 Apr 2025). In ObjectRelator, the Exo2Ego direction is the symmetric version of ego–exo object correspondence: query $(I^{exo}, m^{exo})$, target $(I^{ego}, m^{ego})$, and predict $\hat m^{ego}$ given $(I^{ego}, I^{exo}, m^{exo})$ (Fu et al., 2024). In EgoExoLearn and EgoMe, the setting is asynchronous rather than synchronized: a learner watches exocentric demonstrations and later performs the same procedure egocentrically, so the problem becomes semantic and procedural alignment rather than frame correspondence (Huang et al., 2024).

A common misconception is to equate Exo2Ego exclusively with novel-view synthesis. The literature does not support that reduction. Exo2Ego also includes exocentric pre-training for egocentric activity recognition, retrieval-augmented captioning with third-person instructional videos, exocentric-to-egocentric dense procedural captioning, and exocentric knowledge transfer into egocentric MLLMs (Xu et al., 2024).

2. Data regimes and benchmarks

Exo2Ego research is organized around two main data regimes: synchronized multi-view capture and asynchronous demonstration-following. The synchronized regime supports direct cross-view prediction and correspondence. Ego-Exo4D is the most frequently used resource in this category: one line of work describes it as a large multiview dataset of 1286h with synchronized ego and at least 4 exo cameras, and uses its cooking scenario with 564h across 60 locations, 33,448 training clips, and 1,000 validation clips for Exo2Ego video prediction (Xu et al., 16 Apr 2025). Another line emphasizes its object-level annotations, reporting about $(I^{ego}, I^{exo}, m^{exo})$0M annotated instance masks at 1 fps across 1335 takes, with domains including Cooking, BikeRepair, Health, Music, Basketball, and Soccer (Fu et al., 2024).

H2O, Assembly101, Aria Pilot, TACO, HOMAGE, LEMMA, and Nymeria appear as complementary benchmarks. H2O is used both for zero-shot Exo2Ego video prediction and for single-image or object-centric exocentric-to-egocentric translation (Xu et al., 16 Apr 2025). Assembly101 and Aria Pilot form part of the early cross-view translation benchmark introduced by “Put Myself in Your Shoes,” which curated synchronized tabletop activity video pairs from H2O, Aria Pilot, and Assembly101 (Luo et al., 2024). Nymeria contributes egocentric video, 3D body motion from Xsens IMU mocap, 6DoF wrist poses from miniAria wristbands, and Project Aria MPS camera trajectories and semi-dense point clouds, making it central to controllable egocentric generation and world modeling (Gu et al., 25 May 2026).

The asynchronous regime is structurally different. EgoExoLearn contains 120 hours, with 432 egocentric execution videos totaling 96.5 hours and 315 exocentric demonstration videos totaling 23.5 hours, along with calibrated gaze and detailed multimodal annotations for cross-view association, cross-view action planning, and cross-view referenced skill assessment (Huang et al., 2024). EgoMe contains 7902 paired exo-ego videos, totaling 15804 videos and 82 hours 46 minutes, spanning 184 activity categories in 41 scenarios, and adds exo-ego eye gaze, IMU data, coarse and fine language annotations, and correct/false following labels (Qiu et al., 31 Jan 2025). These datasets recast Exo2Ego as imitation learning rather than synchronized view conversion.

Dense-captioning and instruction-following benchmarks formalize Exo2Ego at the language level. Exo2EgoDVC introduces EgoYC2, a 43-hour egocentric cooking dataset whose captions follow the definition of YouCook2 captions, enabling cross-view dense video captioning from web instructional videos to egocentric cooking (Ohkawa et al., 2023). UE$(I^{ego}, I^{exo}, m^{exo})$1DPAC extends this direction with EgoMe-UE$(I^{ego}, I^{exo}, m^{exo})$2DPAC, where the source view is labeled and the target view is unlabeled, and adaptation is driven by gaze consensus rather than target captions (Shi et al., 7 Apr 2025). At the MLLM scale, Ego-ExoClip contributes 1.1M synchronized ego-exo clip-text pairs derived from Ego-Exo4D, while EgoBench aggregates eight egocentric evaluation tasks spanning reasoning, episodic memory, action recognition, retrieval, navigation, planning, and alignment (Zhang et al., 12 Mar 2025).

3. Core modeling paradigms

A recurring design decision is whether Exo2Ego should be solved by direct end-to-end mapping or by explicit intermediate structure. Early cross-view synthesis work argues for decoupling the problem into high-level structure transformation and pixel-level hallucination: exocentric frames and exocentric hand layouts are first mapped to ego hand layouts, then a latent diffusion model synthesizes the corresponding ego image (Luo et al., 2024). EgoExo-Gen generalizes the same principle to video prediction by predicting future ego-view HOI masks $(I^{ego}, I^{exo}, m^{exo})$3 and then conditioning a video diffusion model on $(I^{ego}, I^{exo}, m^{exo})$4, $(I^{ego}, I^{exo}, m^{exo})$5, and the predicted masks (Xu et al., 16 Apr 2025). EgoWorld adopts a geometric variant of this template: reconstruct a point cloud from exocentric depth, transform it into ego coordinates using 3D hand poses, project it into a sparse egocentric RGB map $(I^{ego}, I^{exo}, m^{exo})$6, and let a latent diffusion model inpaint the dense egocentric image from $(I^{ego}, I^{exo}, m^{exo})$7, $(I^{ego}, I^{exo}, m^{exo})$8, and $(I^{ego}, I^{exo}, m^{exo})$9 (Park et al., 22 Jun 2025).

A second paradigm is explicit alignment in latent space. Ego-Exo transfers visual representations from third-person to first-person video by augmenting exocentric pre-training with pseudo-labels for Ego-Score, Object-Score, and Interaction-Map, and optimizing the joint loss

$\mathcal{V}_{\text{exo}}$0

so that exocentric features become predictive of egocentric signals without paired exo/ego video during pre-training (Li et al., 2021). Exo2Ego for MLLMs similarly learns explicit mapping functions between egocentric and exocentric feature spaces, with cycle consistency and KL guidance, so that an egocentric clip can be mapped into an exocentric-like representation and exploited by a teacher MLLM already rich in third-person knowledge (Zhang et al., 12 Mar 2025).

A third paradigm is cross-view correspondence via attention and alignment modules. EgoExo-Gen uses Ego–Exo Memory Attention, where the current exo visual feature acts as a query and past exo and ego features act as keys and values, with an annealed residual memory path that shifts from ego-query-based supervision to exo-query-based prediction (Xu et al., 16 Apr 2025). ObjectRelator introduces Multimodal Condition Fusion, which fuses exo text and exo visual object embeddings through cross-attention with a learnable residual, and XObjAlign, which minimizes $\mathcal{V}_{\text{exo}}$1 so that ego and exo embeddings of the same object become close (Fu et al., 2024). Exo2EgoSyn adapts WAN 2.2 by combining EgoExo-Align, MultiExoCon, and PoseInj: first-frame latent alignment reorients the model’s same-view bias toward the ego view, multi-view exocentric video tokens replace text tokens in cross-attention, and temporally compressed Plücker pose embeddings are injected into the latent state to preserve geometry-aware synthesis (Mahdi et al., 25 Nov 2025).

These variations share a common pattern. Exo2Ego methods rarely rely on raw RGB alone. They usually introduce an intermediate structure that is more stable across views than pixels themselves: hand layouts, HOI masks, object masks, 3D hand pose, point clouds, body pose, gaze, or exocentric feature embeddings.

4. Generative and predictive Exo2Ego

Image and video synthesis constitute the most visible branch of Exo2Ego. “Put Myself in Your Shoes” established a two-stage Exo2Ego generator for synchronized tabletop activity videos and reported consistent gains over pix2pixHD, P-GAN, vid2vid, and pixelNeRF on H2O, Aria Pilot, and Assembly101, with particularly large improvements in FID, LPIPS, and hand feasibility (Luo et al., 2024). Its central claim was not geometric exactness but the usefulness of a hand-layout prior for cross-view correspondence.

EgoExo-Gen extends the task from frame translation to future prediction. On Ego-Exo4D, it reports SSIM $\mathcal{V}_{\text{exo}}$2, PSNR $\mathcal{V}_{\text{exo}}$3, LPIPS $\mathcal{V}_{\text{exo}}$4, and FVD $\mathcal{V}_{\text{exo}}$5, outperforming SVD, Seer, DynamiCrafter, SparseCtrl, SEINE, and ConsistI2V (Xu et al., 16 Apr 2025). Its ablations are especially informative. Using oracle future HOI masks, the baseline without masks reaches SSIM $\mathcal{V}_{\text{exo}}$6, PSNR $\mathcal{V}_{\text{exo}}$7, and FVD $\mathcal{V}_{\text{exo}}$8, whereas hand+object masks reach SSIM $\mathcal{V}_{\text{exo}}$9, PSNR $g_1$0, and FVD $g_1$1, indicating that structural HOI information is more critical than text for plausible egocentric motion (Xu et al., 16 Apr 2025). The paper also shows that simply adding exo RGB or exo HOI masks directly to the diffusion model is sub-optimal relative to a dedicated Exo→Ego HOI prediction stage.

EgoWorld focuses on single-image exocentric-to-egocentric translation with richer geometry. On H2O unseen objects, it reports FID $g_1$2, PSNR $g_1$3, SSIM $g_1$4, and LPIPS $g_1$5, improving over CFLD’s FID $g_1$6, PSNR $g_1$7, SSIM $g_1$8, and LPIPS $g_1$9 (Park et al., 22 Jun 2025). The model’s conditioning ablations show that text alone already improves realism and semantics, pose alone improves hand alignment, and the combination of sparse ego reprojection, 3D-informed pose map, and text is best. This supports a broader Exo2Ego theme: geometry and semantics are complementary rather than interchangeable.

Foundation-model adaptation is a separate trend. Exo2EgoSyn modifies WAN 2.2 rather than training a bespoke generator from scratch. On Ego-Exo4D Bike, Basketball, CPR, and Covid categories, it improves over the VAWAN baseline in PSNR, SSIM, and LPIPS; for example, on Basketball the baseline scores $\mathcal{T}$0 and the adapted model scores $\mathcal{T}$1 (Mahdi et al., 25 Nov 2025). The critical insight is that same-view bias in large video models can be redirected rather than removed: if the model is conditioned on a predicted ego first-frame latent rather than an exo latent, it continues generation in the ego view.

Exo2Ego also now reaches predictive world modeling. EgoExo-WM converts exocentric video into action-conditioned egocentric training trajectories by extracting 3D whole-body pose from exocentric video, synthesizing an aligned egocentric view, and training a world model in DINOv3 latent space (Tran et al., 14 May 2026). On HOMAGE, its average latent $\mathcal{T}$2 error is $\mathcal{T}$3, compared with $\mathcal{T}$4 for Ego-WM and $\mathcal{T}$5 for Naive EgoExo-WM, and its planning experiments reduce HOMAGE wrist MPJPE from $\mathcal{T}$6 with UniEgoMotion + Ego-WM to $\mathcal{T}$7 with UniEgoMotion + EgoExo-WM (Tran et al., 14 May 2026). E$\mathcal{T}$8C, by contrast, is not a full Exo2Ego translator but a controllable egocentric generator with 3D environmental memory and separate ego and exo human controls; it is best interpreted as a rendering engine that could serve as the second stage of an Exo2Ego pipeline once ego camera trajectory and ego body motion have been inferred (Gu et al., 25 May 2026).

5. Retrieval, captioning, understanding, and reasoning

Language-centric Exo2Ego methods exploit the fact that procedural semantics often transfer more easily across views than raw appearance. EgoInstructor retrieves semantically relevant third-person instructional videos to improve egocentric captioning, training a cross-view retrieval module with EgoExoNCE so that egocentric and exocentric video features align through shared text features (Xu et al., 2024). On Ego4D cooking, MPT-1B captioning improves from CIDEr $\mathcal{T}$9 in the 0-shot setting to $\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$0 with one retrieved exocentric reference and to $\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$1 with eight retrieved references, while random exocentric videos do not help on EgoLearner (Xu et al., 2024). This result is important because it shows that Exo2Ego transfer can be mediated entirely by retrieval and language without explicit synthesis.

Dense procedural captioning translates the same idea into temporal segmentation. Exo2EgoDVC creates a benchmark for transferring dense video captioning from YouCook2 to EgoYC2, using view-invariant adversarial learning in pre-training and fine-tuning stages so that the captioning model operates on features that are hard to classify by view (Ohkawa et al., 2023). UE$\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$2DPAC removes target-view labels and instead uses Gaze Consensus-guided Ego-Exo Adaptation Network. In the Exo2Ego setting with PDVC, GCEAN reaches BLEU4 $\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$3, METEOR $\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$4, CIDEr $\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$5, SODA$\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$6 $\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$7, SODA$\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$8 $\mathcal{V}_{\text{ego}}=\{g_1,\dots,g_N\}$9, and SODA tIoU $(I^{exo}, m^{exo})$0, improving on the Sync baseline across all reported metrics (Shi et al., 7 Apr 2025). This establishes gaze as a viable cross-view alignment signal for unsupervised dense caption transfer.

At the MLLM level, Exo2Ego uses Ego-ExoClip, a 1.1M synchronized ego-exo clip-text dataset, and a three-stage teacher–student pipeline to transfer exocentric knowledge into egocentric video understanding (Zhang et al., 12 Mar 2025). On EgoBench, it reports $(I^{exo}, m^{exo})$1 accuracy on EgoSchema, $(I^{exo}, m^{exo})$2 on QAEgo4D closed QA, $(I^{exo}, m^{exo})$3 mAP on Charades-Ego, $(I^{exo}, m^{exo})$4 mAP and $(I^{exo}, m^{exo})$5 nDCG on EPIC-KITCHENS-100 MIR, $(I^{exo}, m^{exo})$6 accuracy on EgoPlan, and $(I^{exo}, m^{exo})$7 on VLN-QA (Zhang et al., 12 Mar 2025). The significance is methodological as much as empirical: rather than retrieving exocentric evidence at inference, the model internalizes an ego↔exo mapping in feature space.

Reasoning benchmarks push Exo2Ego beyond understanding into memory. EgoExoMem introduces cross-view memory reasoning over synchronized egocentric and exocentric videos, with 2.6K human-verified MCQs across eight temporal, spatial, and cross-view QA types (Liu et al., 18 May 2026). Existing MLLMs remain limited: the best model reaches only $(I^{exo}, m^{exo})$8, while E$(I^{exo}, m^{exo})$9-Select with Gemini 2.5 Flash reaches $(I^{ego}, m^{ego})$0 by allocating memory budget across ego and exo views and using per-view k-DPP sampling (Liu et al., 18 May 2026). A plausible implication is that future Exo2Ego systems will need not only cross-view invariance but also explicit view routing: some questions are framed from one viewpoint while their evidence is grounded in the other.

6. Limitations, misconceptions, and open directions

The main limitation across Exo2Ego is that the cross-view map is underdetermined. Generative methods therefore depend heavily on intermediate priors. In EgoExo-Gen, complex hand motion, rapid changes, or severe occlusion degrade mask prediction, and errors in predicted HOI masks translate directly into incorrect or jittery hand positions in the generated ego video (Xu et al., 16 Apr 2025). ObjectRelator is frame-level only, with no tracking or memory, and still fails on weak boundaries, many similar objects, and very small target objects (Fu et al., 2024). EgoWorld depends on accurate depth, 3D hand pose, and VLM text; subtle finger configurations and semantically wrong object descriptions remain failure modes (Park et al., 22 Jun 2025). Exo2EgoSyn depends on a good first-frame latent and struggles when fast motion or textureless initial ego content makes the view-alignment module unreliable (Mahdi et al., 25 Nov 2025).

A second limitation is that strong results often rely on structured assumptions that do not hold uniformly across the field. Some methods require synchronized multi-view capture, some require the first ego frame, some use relative camera poses, and some are explicitly designed to avoid both camera pose and depth at test time (Xu et al., 16 Apr 2025). EgoExo-WM currently converts only about 10 hours of exocentric video and evaluates mostly 2-second planning horizons, while E$(I^{ego}, m^{ego})$1C assumes mostly static environments and does not itself learn exocentric-to-egocentric translation from an external camera stream (Tran et al., 14 May 2026). This suggests that “Exo2Ego” should not be treated as a single difficulty level; the problem changes substantially depending on whether synchronization, calibration, initial ego context, or egocentric supervision are available.

A third misconception is that domain adaptation alone solves Exo2Ego. Ego-Exo already showed that domain-adversarial training yields almost no gain over Third-only pre-training and sometimes worse, whereas distilling ego-relevant signals such as hands, objects, and interactions into exocentric pre-training improves egocentric downstream performance (Li et al., 2021). The broader literature reinforces that point: successful Exo2Ego systems almost always inject structure, whether as HOI masks, object masks, gaze consensus, 3D hand pose, point clouds, body kinematics, or first-frame latent alignment.

Open directions are consistent across papers. Better long-horizon HOI prediction, stronger temporal modeling, and more explicit 3D reasoning are recurring needs in EgoExo-Gen and EgoExo-WM (Xu et al., 16 Apr 2025). Asynchronous procedural learning points toward richer cross-view planning, skill assessment, and multimodal alignment with gaze and IMU in EgoExoLearn and EgoMe (Huang et al., 2024). Cross-view memory reasoning points toward joint query–answer view routing and multi-camera memory rather than simple view-invariant embeddings (Liu et al., 18 May 2026). More generally, the field appears to be moving from narrow view translation toward a unified account in which exocentric video supplies geometric coverage, procedural priors, and scalable data, while egocentric modeling supplies the action-centric perspective required for manipulation, assistance, and embodied decision-making.