From Synchrony to Sequence: Exo-to-Ego Generation via Interpolation

Abstract: Exo-to-Ego video generation aims to synthesize a first-person video from a synchronized third-person view and corresponding camera poses. While paired supervision is available, synchronized exo-ego data inherently introduces substantial spatio-temporal and geometric discontinuities, violating the smooth-motion assumptions of standard video generation benchmarks. We identify this synchronization-induced jump as the central challenge and propose Syn2Seq-Forcing, a sequential formulation that interpolates between the source and target videos to form a single continuous signal. By reframing Exo2Ego as sequential signal modeling rather than a conventional condition-output task, our approach enables diffusion-based sequence models, e.g. Diffusion Forcing Transformers (DFoT), to capture coherent transitions across frames more effectively. Empirically, we show that interpolating only the videos, without performing pose interpolation already produces significant improvements, emphasizing that the dominant difficulty arises from spatio-temporal discontinuities. Beyond immediate performance gains, this formulation establishes a general and flexible framework capable of unifying both Exo2Ego and Ego2Exo generation within a single continuous sequence model, providing a principled foundation for future research in cross-view video synthesis.

• The paper introduces Syn2Seq-Forcing which uses pseudo-interpolation of frames and SLERP-based pose blending to achieve continuous exo-to-ego transitions.
• It employs a Diffusion Forcing Transformer for flexible sequential signal modeling across 356k pretraining videos and 40k fine-tuned category videos.
• Empirical evaluations show significant gains in PSNR, SSIM, and LPIPS over baselines, confirming enhanced synchronization and visual fidelity.

Sequential Interpolation Framework for Exo-to-Ego Video Generation

Problem Definition and Motivation

The Exo-to-Ego video generation problem entails synthesizing a temporally synchronized egocentric (first-person) video sequence based on a paired exocentric (third-person) video and corresponding camera pose data. The formulation is practically relevant in AR/VR, robotics, and human-computer interaction, underlying tasks requiring the transformation of external observations to immersive, first-person perspectives. Conventional paired supervised approaches suffer from synchronization-induced discontinuities: sharp geometric and appearance jumps at the exo-to-ego boundary. Such spatio-temporal and pose discontinuities are not addressed by existing video generators, which typically assume smooth, continuous camera trajectories. This causes substantial degradation in transition coherence and visual fidelity at exo-to-ego junctions.

Figure 1: Syn2Seq-Forcing interpolates between views and camera poses to enable smooth exo2ego transitions, in contrast to standard approaches that fail at discontinuous camera pose boundaries.

Methodology

Sequential Signal Modeling via Syn2Seq-Forcing

The paper introduces Syn2Seq-Forcing, reframing the Exo2Ego task from condition–output prediction to sequential signal modeling. The approach concatenates exocentric video frames, pseudo-interpolated transition frames, and egocentric video frames into a unified temporal sequence. Corresponding camera poses are interpolated using SLERP for rotation and linear blending for translation, forming a continuous pose signal that bridges boundary discontinuities.

Figure 2: Overview of the Syn2Seq-Forcing training and inference pipeline: pseudo-labels for transitions are generated with WFLF, and DFoT is trained on randomly sampled transitions conditioned on poses.

Transition segments are generated using the WAN2.2 First/Last Frame (WFLF) interpolator, queried with the boundary exocentric and egocentric frames, along with category-specific text prompts. Pose interpolation employs SLERP between boundary quaternions, ensuring geometric continuity.

The core generative engine is a Diffusion Forcing Transformer (DFoT), which supports flexible conditioning over arbitrary subsets of temporal history—each frame is independently noised, allowing dynamic context aggregation and classifier-free guidance-style history modulation during sampling. Training proceeds in a two-stage paradigm: large-scale pretraining on direct Exo–Ego pairs (356k videos), followed by fine-tuning on interpolated triplets per category (40k videos/category).

During inference, exocentric frames are concatenated with a noise-only suffix, which is progressively denoised into a transition segment and the target egocentric frames, all within the same model and process.

Empirical Evaluation

Quantitative and Qualitative Analysis

Evaluation is conducted on three Ego-Exo4D benchmark categories (Bike, Health, Cooking), using PSNR, SSIM, and LPIPS as metrics. Syn2Seq-Forcing outperforms all baselines, with statistically significant gains, e.g., Health: PSNR = 16.71, SSIM = 0.573, LPIPS = 0.483; Bike: PSNR = 15.63, SSIM = 0.472, LPIPS = 0.501. These results are robust across all categories, highlighting the model's capacity to bridge exo–ego discontinuities.

Figure 3: Visual comparison across methods; Syn2Seq-Forcing achieves higher resemblance to ground-truth and improved transition smoothness.

Ablation studies isolate the contribution of interpolation. Frame-only interpolation produces substantial gains over direct mapping; the addition of pose interpolation yields further improvements, confirming that spatio-temporal discontinuity is the principal bottleneck.

Figure 4: Interpolation effects on generated outputs—remediation of abrupt jumps and enhanced continuity.

The WFLF pseudo-interpolator greatly surpasses DFoT's native inference-time interpolation, indicating that pretrained boundary-conditioned interpolators are essential for realistic transitions (Bike-INT: PSNR = 15.69 vs. 13.11). Pose embedding variants are compared, with Plücker embeddings delivering optimal results due to their compatibility with geometric signal interpolation.

Cross-View Synthesis Capability

Syn2Seq-Forcing is inherently flexible: its sequence-centric formulation enables both Exo2Ego and Ego2Exo generation without architectural modification. The model jointly synthesizes the transition and the target domain segment, maintaining temporal coherence and visual consistency.

Figure 5: Generation of intermediate frames bridging exocentric and egocentric boundary.

Figure 6: Model's performance on cross-view exocentric-to-egocentric video generation.

Practical and Theoretical Implications

The interpolation-based sequential modeling paradigm decomposes synchronization-induced discontinuities, opening avenues for unified, bidirectional view translation in cross-view video synthesis. It enables controllability, extensibility to longer temporal horizons, and scalable adaptation to other cross-domain sequence modeling tasks. The strong empirical evidence emphasizes the necessity of explicit spatio-temporal interpolation rather than reliance on standard condition–output architectures. The framework does not require multi-view exocentric evidence, textual descriptions, or 3D priors, thus reducing annotation and modeling complexity.

From a research standpoint, this formulation facilitates hierarchical attention and compositional temporal conditioning, crucial for modeling physically plausible transitions and grounded view synthesis. Future work includes integrated pose extraction from interpolated frames, increased synthetic fine-tuning data, and exploration of longer-term sequential cross-view modeling.

Conclusion

Syn2Seq-Forcing demonstrates that interpolation-centric sequential modeling is superior for Exo2Ego video generation under synchronization-induced discontinuity, yielding improved coherence, fidelity, and geometric plausibility. The sequence-centric framework supports flexible cross-view translation and offers a principled baseline for future research in multi-view and cross-modal video synthesis (2604.13793).