LongE2V: Event-Based Video Diffusion

• LongE2V is an event-conditioned video diffusion framework that unifies video reconstruction, prediction, and interpolation using a single pre-trained diffusion model.
• It leverages event voxel grids and task-specific strategies like Autoregressive Unrolling and Adaptive Context Switching to address sparse event data and long-horizon temporal drift.
• The framework employs mechanisms such as Reencoding Alignment and Cross Residual Correction to ensure bidirectional consistency and enhance high-frequency texture preservation.

LongE2V is an event-conditioned video diffusion framework for three inverse problems in event vision: event-based video reconstruction, event-based video prediction, and event-based frame interpolation. It is defined around sparse event streams from event cameras, where each event is $e_i=(x_i,y_i,t_i,p_i)$ with spatial location $(x_i,y_i)$, timestamp $t_i$, and polarity $p_i\in\{-1,+1\}$. The method uses pre-trained video diffusion priors, specifically a fine-tuned foundational image-to-video model, to recover dense RGB video with high perceptual quality while addressing long-horizon temporal drift and bidirectional interpolation consistency. The system introduces Autoregressive Unrolling, Adaptive Context Switching, Reencoding Alignment, Cross Residual Correction, and Event Voxel Density Augmentation as task-specific mechanisms within a unified architecture (Fan et al., 9 Jul 2026).

1. Problem formulation and task unification

LongE2V addresses three distinct inverse problems under a single model family (Fan et al., 9 Jul 2026). In event-based video reconstruction, the input is only the event stream, and the goal is to recover dense RGB frames. The paper identifies the core difficulties as the ill-posedness of inferring absolute intensity from relative changes, the extreme sparsity of the event signal, and the preservation of texture fidelity.

In event-based video prediction, the model is given a start frame and subsequent events, and it must generate long sequences with minimal temporal drift. The emphasis is explicitly on horizons of hundreds to thousands of frames, where error accumulation and drift become dominant failure modes. In event-based frame interpolation, the start and end frames are available together with events in between, and the task is to synthesize intermediate frames while preserving bidirectional consistency and avoiding ghosting under complex motion.

The central claim of LongE2V is that these tasks can be unified by conditioning a single pre-trained video diffusion backbone on event-derived voxel grids and on task-specific context initialization (Fan et al., 9 Jul 2026). This suggests a shift away from task-isolated pipelines toward a common generative latent space in which reconstruction, prediction, and interpolation differ mainly in context construction, temporal directionality, and update policy rather than in backbone architecture.

2. Event representation and conditioning interface

LongE2V converts asynchronous events into a temporal voxel grid $V\in\mathbb{R}^{B\times H\times W}$ by accumulating event polarities over $B$ temporal bins with linear interpolation:

$$V(t,y,x)=\sum_i p_i \cdot \max(0,1-|t-t_i^*|)\cdot \delta(x-x_i,y-y_i),$$

where $\delta$ is the Kronecker delta and $t_i^*\in[0,B-1]$ is the normalized timestamp within a window of duration $\Delta T$. In all experiments, $(x_i,y_i)$0, matching the 3-channel input of the backbone’s 3D VAE (Fan et al., 9 Jul 2026).

Temporal synchronization is performed by partitioning the event stream into windows $(x_i,y_i)$1 and constructing voxels $(x_i,y_i)$2 aligned to frame $(x_i,y_i)$3, yielding the synchronized event sequence $(x_i,y_i)$4. During fine-tuning, current event voxels at the initial timestamp are zero-padded to align with frames $(x_i,y_i)$5.

The conditioning pipeline is built on a frozen 3D VAE encoder with $(x_i,y_i)$6 temporal and $(x_i,y_i)$7 spatial compression. Three temporally aligned latent sequences are concatenated along the channel dimension: context latents plus zero-padded first-frame latent $(x_i,y_i)$8, context latents plus train-time noise latents $(x_i,y_i)$9, and context event latents plus current event latents. To ingest the additional event channels, the first projection layer of the patchify module is expanded from

$t_i$0

This layer is fully fine-tuned, whereas the Diffusion Transformer blocks are adapted with LoRA (Fan et al., 9 Jul 2026).

Two auxiliary conditioning choices are also explicit. A $t_i$1 dropout is applied to $t_i$2 during training to improve robustness when the start frame is missing, as in reconstruction. Text prompts are used with $t_i$3 probability during training to preserve multi-modal conditioning capability, specifically text-guided colorization. A possible misconception is that the model is purely event-conditioned; the training protocol instead preserves a broader conditioning interface even though the core evaluation focuses on event-based tasks.

3. Backbone, objective, and long-horizon stabilization

The foundation model is CogVideoX I2V, used as a latent diffusion backbone with a frozen 3D VAE encoder/decoder and a Diffusion Transformer denoiser $t_i$4 (Fan et al., 9 Jul 2026). The DiT is fine-tuned via LoRA with rank $t_i$5, while the expanded patchify input projection is fully fine-tuned. The training objective is the standard $t_i$6-prediction loss:

$t_i$7

where $t_i$8 are clean VAE latents, $t_i$9 are noisy latents from the forward diffusion process $p_i\in\{-1,+1\}$0, and $p_i\in\{-1,+1\}$1 denotes the conditioning latents. Sampling uses standard DDPM-style denoising in latent space. The paper explicitly states that no auxiliary losses such as $p_i\in\{-1,+1\}$2 or LPIPS are used; reconstruction quality and temporal consistency are attributed to the diffusion prior together with LongE2V’s conditioning and context strategies (Fan et al., 9 Jul 2026).

Long-horizon generation is stabilized by Autoregressive Unrolling and Adaptive Context Switching. AU is a training strategy designed to close the train–inference gap caused by the use of self-generated rather than ground-truth context at test time. Long sequences are processed in 49-frame chunks, each preceded by a fixed 20-frame context. The training schedule is: initial training with ground-truth context until convergence, then repeated unrolling cycles in which the model first generates predicted context frames over the training set and then fine-tunes on these self-generated contexts. The reported configuration uses 3000 initial steps, followed by 3 AU cycles of 3000 steps each (Fan et al., 9 Jul 2026).

ACS is an inference-time mechanism for deciding when context should be refreshed. Rather than updating context after every chunk, the model measures context relevance through DiT attention. For denoised latents $p_i\in\{-1,+1\}$3, multi-head attention maps $p_i\in\{-1,+1\}$4 are collected, and the mean attention from current tokens to context tokens is computed as

$p_i\in\{-1,+1\}$5

If $p_i\in\{-1,+1\}$6 with $p_i\in\{-1,+1\}$7, the existing context is retained. If $p_i\in\{-1,+1\}$8, the current generation is discarded, the context is updated to the immediate predecessor, and the chunk is regenerated once. According to the reported ablations, removing AU produces train–test mismatch and drift artifacts, while removing ACS causes periodic accumulation and grid-like artifacts (Fan et al., 9 Jul 2026).

4. Bidirectional interpolation, latent alignment, and density robustness

Frame interpolation is handled through separate forward and backward generations between the start and end frames, both guided by aligned event streams (Fan et al., 9 Jul 2026). The paper argues that temporal flipping in the compressed 3D VAE latent space is not commutative with decode–flip–encode in pixel space:

$p_i\in\{-1,+1\}$9

This motivates Reencoding Alignment. Given denoised clean latents from the forward and backward branches, RA constructs aligned latents by decode–flip–encode:

$V\in\mathbb{R}^{B\times H\times W}$0

RA is intended to remove temporal misalignment and the ghosting associated with latent-space inconsistency.

Because re-encoding causes information loss, LongE2V adds Cross Residual Correction. Residuals are computed relative to pure decode–encode reconstructions:

$V\in\mathbb{R}^{B\times H\times W}$1

$V\in\mathbb{R}^{B\times H\times W}$2

These residuals are then cross-injected symmetrically:

$V\in\mathbb{R}^{B\times H\times W}$3

The corrected latents are fused by linear alpha blending from forward-dominant near the start frame to backward-dominant near the end frame, followed by re-noising for subsequent diffusion steps. The paper reports that RA eliminates ghosting caused by latent misalignment, while CRC restores high-frequency textures and enforces bidirectional temporal consensus (Fan et al., 9 Jul 2026).

Robustness to sensor heterogeneity is provided by Event Voxel Density Augmentation. EVDA randomly resizes event voxels within $V\in\mathbb{R}^{B\times H\times W}$4, with $V\in\mathbb{R}^{B\times H\times W}$5 the original resolution and $V\in\mathbb{R}^{B\times H\times W}$6 slightly above network input size to permit random cropping. It also uses FireNet-style normalization based on non-zero voxel statistics and applies identical geometric transforms to event voxels, first frame, context frames, and current video frames. This suggests that LongE2V treats density variation as a conditioning-domain problem rather than only as an image-resolution problem.

5. Training protocol, inference recipes, and implementation details

The reported training configuration uses AdamW with learning rate $V\in\mathbb{R}^{B\times H\times W}$7, cosine scheduler, weight decay $V\in\mathbb{R}^{B\times H\times W}$8, batch size $V\in\mathbb{R}^{B\times H\times W}$9, and gradient accumulation of 4 steps (Fan et al., 9 Jul 2026). The operational clip size is 49 frames at $B$0 resolution with a 20-frame context. Inputs below the backbone resolution are upsampled to match it, whereas larger inputs are upsampled by about $B$1 and processed using Per-tile Denoising and Fusion, as in VDM-EVFI.

Task-specific inference initialization differs across the three problems. For reconstruction, the first chunk uses zero tensors for the start-frame and context latents, and context events are zero-padded. For prediction, the first chunk replicates the start frame 20 times to populate context, with context events again zero-padded. For interpolation, the first chunk replicates the start and end frames 10 times each to form the 20-frame context, and RA+CRC is applied before alpha blending and re-noising (Fan et al., 9 Jul 2026).

The practical reproducibility guidance is explicit. Data preprocessing filters missing sequences in the BS-ERGB training set; evaluation uses EVREAL subsets from ECD, MVSEC, and HQF for reconstruction and prediction, and BS-ERGB plus HQF for interpolation. Global brightness of generated outputs is aligned to ground truth for fair quantitative comparison because events lack absolute intensity. For adaptation to new sensors or datasets, the paper recommends calibrating EVDA ranges to match target sensor resolution, adjusting the ACS threshold $B$2 for motion complexity, and, under extremely sparse events, considering a larger context window or text conditioning (Fan et al., 9 Jul 2026).

6. Benchmarks, ablations, relation to prior work, and limitations

LongE2V is trained on the BS-ERGB training set comprising 7,636 frames. Evaluation for reconstruction and prediction uses EVREAL benchmark subsets from ECD, MVSEC, and HQF, with total frame counts of 1,855 for ECD, 11,321 for MVSEC, and 15,499 for HQF. Interpolation is evaluated on the BS-ERGB test set with 4,546 frames and on HQF with 15,513 frames, at 31 skips. Metrics are PSNR, SSIM, LPIPS, and VBench Subject Consistency for long-term identity and structure coherence (Fan et al., 9 Jul 2026).

The reported quantitative results are as follows. For reconstruction, LongE2V achieves 22.28 PSNR, 0.708 SSIM, and 0.139 LPIPS on ECD; 16.83 PSNR, 0.440 SSIM, and 0.405 LPIPS on MVSEC; and 16.45 PSNR, 0.603 SSIM, and 0.240 LPIPS on HQF. For prediction with a start frame, it reports 24.40 PSNR, 0.771 SSIM, and 0.110 LPIPS on ECD; 18.18 PSNR, 0.502 SSIM, and 0.359 LPIPS on MVSEC; and 16.67 PSNR, 0.619 SSIM, and 0.229 LPIPS on HQF. For interpolation at 31 skips, it reports 24.40 PSNR, 0.744 SSIM, and 0.124 LPIPS on BS-ERGB in a zero-shot setting, and 25.39 PSNR, 0.800 SSIM, and 0.105 LPIPS on HQF (Fan et al., 9 Jul 2026).

For long-term temporal consistency on HQF, VBench Subject Consistency is reported as 0.5413 for E2VID+ reconstruction, 0.4953 for HyperE2VID reconstruction, 0.6279 for VDM-EVFI prediction, 0.7204 for LongE2V reconstruction, and 0.7187 for LongE2V prediction. The paper also reports inference speed on NVIDIA RTX A6000 for HQF bike_bay_hdr in steady state: E2VID at 0.0024 s/frame, VDM-EVFI at 5.4483 s/frame, and LongE2V at 3.5964 s/frame (Fan et al., 9 Jul 2026).

The ablations are central to the paper’s interpretation of the method. On HQF reconstruction, removing the pre-trained prior causes failure to converge and pure noise; removing context causes severe artifact accumulation; removing AU produces drift artifacts; removing ACS yields grid artifacts from periodic accumulation. On BS-ERGB interpolation, removing RA causes temporal misalignment and blur, with LPIPS degrading to 0.202 in one reported comparison; removing CRC leads to ghosting or transparent artifacts and worsens LPIPS by 0.037 versus the full model; removing EVDA causes artifacts and color deviations from density mismatch (Fan et al., 9 Jul 2026).

In relation to prior work, LongE2V is positioned against regression methods such as E2VID, FireNet, SPADE-E2VID, ET-Net, and HyperE2VID, against interpolation-focused methods such as TimeLens variants, CBMNet-Large, TLXNet+, and EPA, and against prior video-diffusion adaptation in VDM-EVFI. The paper’s stated novelty lies in a single diffusion-based framework jointly addressing reconstruction, prediction, and interpolation; AU plus ACS for long-term temporal coherence; RA plus CRC for bidirectional consistency in 3D VAE latent space; and high data efficiency, with training exclusively on BS-ERGB yet zero-shot interpolation performance (Fan et al., 9 Jul 2026).

The reported limitations are equally specific. Reconstruction may fail under extremely sparse or noisy events. Hot pixels in event streams can be preserved or amplified in outputs. Diffusion-based generation is slower than purely regression-based models. Future directions named in the paper are accelerating inference, exploring memory mechanisms for even longer-term consistency, and improving event denoising and robustness under extreme sparsity. A plausible implication is that the model’s main bottleneck has shifted from representational adequacy to computational and sensing robustness rather than to lack of generative capacity.