LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models

Abstract: Recovering high-quality video from sparse event streams is a challenging task. Regression methods often blur textures, while existing generative models struggle with long-term stability. We propose LongE2V, a novel approach that leverages pre-trained video diffusion priors to jointly handle event-based video reconstruction, prediction, and frame interpolation. By fine-tuning a foundational video model, our approach achieves high data efficiency and superior perceptual quality. We introduce Autoregressive Unrolling and Adaptive Context Switching to mitigate temporal drift in extremely long sequences. We also propose Reencoding Alignment with Cross Residual Correction to ensure precise bidirectional consistency during frame interpolation. Furthermore, Event Voxel Density Augmentation ensures robustness across varying sensor resolutions. Extensive experiments on real-world benchmarks demonstrate that LongE2V outperforms state-of-the-art methods across all three tasks, exhibiting exceptional temporal coherence and zero-shot generalization. Project page: https://cdfan0627.github.io/LongE2V-page/

• The paper introduces a unified diffusion-based framework for event video reconstruction, prediction, and interpolation using autoregressive unrolling and adaptive context switching.
• It demonstrates superior performance in perceptual quality and temporal coherence, outperforming existing regression and generative models across key benchmarks.
• It integrates novel mechanisms like reencoding alignment and cross residual correction to mitigate error accumulation and ensure high-fidelity, zero-shot frame interpolation.

LongE2V: A Unified Diffusion-Based Framework for Event Video Reconstruction, Prediction, and Interpolation

Introduction

LongE2V presents a unified event-based video generation paradigm leveraging pre-trained video diffusion priors. The architecture is designed to address the long-standing challenges of event-to-video translation: reconstruction from sparse neuromorphic event streams, long-horizon prediction from partial observations, and frame interpolation for intermediate content synthesis. The system specifically adapts foundational video diffusion models (such as CogVideoX DiT backbones) and introduces several novel mechanisms—Autoregressive Unrolling, Adaptive Context Switching, Reencoding Alignment, and Cross Residual Correction—to ensure temporal stability and cross-domain consistency. The empirical evaluation demonstrates the superiority of LongE2V in terms of both perceptual quality (LPIPS, SSIM, PSNR), temporal coherence, and zero-shot generalization, outperforming state-of-the-art regression-based and generative E2V baselines across real-world datasets.

Figure 1: Challenges in event-based video generation. Regression-based reconstruction blurs details, naive diffusion prediction exhibits color drift, and prior interpolation models show ghosting under dynamics.

Model Architecture and Methodology

Event Representation and Diffusion Conditioning

The input to LongE2V consists of asynchronous event streams discretized as event voxel grids, spatially and temporally binned to synchronize with video frames. The system aggregates event polarities into spatio-temporal bins, yielding contextually rich input volumes. These voxels, paired with optionally supplied context frames, are encoded via a frozen 3D VAE, producing latents suitable for DiT-based video diffusion modeling.

Autoregressive Unrolling for Stable Long-Term Generation

Standard video generation pipelines suffer from error accumulation when recursively predicting long sequences, especially due to the mismatch between training (ground truth context) and inference (model-generated context). LongE2V introduces an Autoregressive Unrolling strategy: after initial convergence on ground-truth contexts, iterative fine-tuning is conducted wherein model predictions are re-injected as context for subsequent training steps. This explicitly closes the train-inference distribution gap, forcing the model to adapt to its own prediction drift.

Figure 2: Autoregressive Unrolling bridges the train-inference gap, using self-predicted context frames for fine-tuning and robustifying temporal stability.

Adaptive Context Switching via Attention Diagnostics

Drift mitigation is further enhanced by an Adaptive Context Switching mechanism. By analyzing average attention weights from DiT’s context to current tokens, the framework dynamically determines when to update history context. If attention drops below threshold $\tau$, the context is updated with regenerated chunks, preventing propagation of irrelevant or erroneous context representations, as formalized with token-to-token attention mean metrics.

Event Voxel Density Augmentation

Event camera outputs exhibit variance in event density due to sensor models and depth. To generalize across these, LongE2V employs a random resizing and cropping data augmentation regime for event voxels, maintaining statistical consistency and robust downstream frame alignment across variable inputs.

Figure 3: The training pipeline incorporates Event Voxel Density Augmentation and geometric sync, encoding all event/first/context frames into aligned latents. The first projection layer is extended for multi-modal conditioning.

Bidirectional Interpolation: Reencoding Alignment and Cross Residual Correction

Interpolation between distant temporal boundaries is nontrivial due to temporal misalignment in latent space—direct time flipping of 3D VAE latents does not ensure pixelwise or semantic correspondence. LongE2V addresses this via Reencoding Alignment: predicted latents are decoded, flipped in pixel space, then re-encoded, synchronizing the representations. However, this introduces lossy degradation.

To address detail loss, Cross Residual Correction computes the residual between original and re-encoded latents, and injects these crosswise into opposite branches (forward residual into backward-aligned latents, etc.), enforcing temporal consensus and detail preservation. This mechanism ensures precise bidirectional frame synthesis with minimal ghosting or synthetic artifacts.

Figure 4: Reencoding Alignment and Cross Residual Correction: pixel-space flipping and re-encoding ensure precise alignment, while cross-branch residual injection recovers lost details.

Empirical Evaluation

Supervised and Zero-Shot Performance

LongE2V was trained solely on the BS-ERGB dataset (7,636 frames) and evaluated on EVREAL, ECD, MVSEC, and HQF benchmarks. The model outperforms all SOTA regression and generative models—E2VID, HyperE2VID, VDM-EVFI—in both reconstruction and prediction metrics. Notably, LongE2V achieves superior LPIPS (perceptual), SSIM, and PSNR scores, especially on long sequences, demonstrating robust temporal stability and subject consistency.

Figure 5: Qualitative comparison on ECD, MVSEC, and HQF. Regression baselines blur details; VDM-EVFI accumulates artifacts; LongE2V maintains coherent, faithful reconstructions and predictions.

For frame interpolation, despite receiving no explicit fine-tuning for this task, LongE2V yields best-in-class LPIPS scores and high SSIM/PSNR in zero-shot transfer. Competing models (CBMNet-Large, TLXNet+) fail on dynamic or fine-text scenes, suffering from blur, ghosting, or structural collapse, whereas LongE2V produces sharp, physically plausible results.

Figure 6: Zero-shot interpolation on BS-ERGB and HQF. Competing models exhibit blur or collapse under large motion, while LongE2V preserves structure and legibility, eliminating ghosting.

Ablation Analysis

Component ablation confirms that: (1) pre-trained video diffusion priors are essential for convergence, (2) context-free and naive autoregressive baselines fail on long-term tasks, (3) removal of Reencoding Alignment or Cross Residual Correction degrades interpolation with temporal ghosting and detail loss, and (4) omitting Event Voxel Density Augmentation results in sensor-specific artifacts.

Figure 7: Qualitative ablation—removal of pretrained prior, context, or unrolling causes noise or structural artifacts; the full pipeline achieves stable, high-fidelity results.

Figure 8: Interpolation ablation—omitting Reencoding Alignment, Cross Residual Correction or augmentation results in ghosting, blur, and instability. The full method matches GT-like sharpness.

Text-Guided Event Video Colorization

LongE2V generalizes to multi-modal settings with mixed text and event conditions. Introducing text prompts at training, the model can perform text-driven colorization and style synthesis from event streams, decoupling event-inferred geometry from prompt-driven appearance.

Figure 9: Text-Guided Event Video Colorization: The model imbues event-driven reconstructions with prompt-provided color and style, preserving event-induced motion coherence.

Limitations

Despite state-of-the-art performance, failure modes exist when input event density is insufficient or noisy—severely sparse or corrupt event streams degrade reconstructions, and the model remains susceptible to “hot pixel” artifacts, amplifying event noise in output frames.

Figure 10: (a) Sparse event streams degrade reconstructions; (b) Noisy events ("hot pixel") propagate into output frames, exposing noise sensitivity.

Practical and Theoretical Implications

LongE2V demonstrates that large pre-trained conditional video diffusion models, when augmented with appropriate fine-tuning protocols and domain-specific augmentations, can serve as powerful “all-in-one” event-based video generators. The adaptability to reconstruction, prediction, and interpolation, including zero-shot transfer, positions this approach as a versatile platform for neuromorphic vision and sensor fusion. The proposed architectural and training innovations—autoreg unrolling, adaptive context diagnostics, and bidirectional alignment—point toward principled solutions in mitigating train-inference gaps, error accumulation, and temporal misalignment, which are critical issues in both autoregressive and bidirectional generative modeling.

Future Directions

Potential research directions involve accelerating inference (currently several orders of magnitude slower than regression networks), improving noise-robust event representations, and scaling context-memorization for even longer-horizon consistency. Extending the paradigm to directly fuse additional modalities (e.g., IMU, Lidar, multi-sensor fusion) or real-time applications in high-speed robotics are natural extensions.

Conclusion

LongE2V establishes a new regime for event-based video reconstruction, prediction, and interpolation by unifying conditional video diffusion with autoregressive and bidirectional stabilization mechanisms. Extensive analysis validates its empirical and theoretical contributions, confirming the value of strong video priors and error-adaptive training in neuromorphic video generation. The approach advocates for generalizable, robust, and perceptually faithful event-to-video translation and provides a clear direction for future research on sensor-conditioned temporal generative models.

---

Reference: "LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models" (2607.08770)