ER-VAE: Event-to-Residual Alignment VAE

• The paper introduces ER-VAE as a deterministic latent mapping module that bypasses traditional flow estimation by learning to encode event data into a residual alignment for video frame prediction.
• ER-VAE tightly aligns event-derived latent representations with true residuals using L1 and L2 loss functions, resulting in improved cosine similarity (0.21 to 0.45) and reduced L1 distance.
• The model leverages a pre-trained Stable Diffusion VAE where only the encoder and quantization layers are fine-tuned, ensuring high-fidelity synthesis in event-conditioned single-frame prediction.

The Event-to-Residual Alignment Variational Autoencoder (ER-VAE) is a deterministic latent mapping module designed for event-driven video frame prediction. Employed as Stage 1 within the DESSERT framework, ER-VAE transforms temporally accumulated event data into a latent residual approximating the true inter-frame latent difference, derived from a pre-trained Stable Diffusion VAE. By tightly aligning event-derived latent representations with ground-truth residuals in latent space via an L1 alignment loss and in image space with an L2 reconstruction loss, ER-VAE provides an effective event-conditioned prior for diffusion-based single-frame synthesis, facilitating superior temporal consistency and fidelity in synthesized video frames (Kong et al., 19 Dec 2025).

1. Position within the DESSERT Framework

ER-VAE serves as the first stage of the DESSERT (Diffusion-based Event-driven Single-frame Synthesis via Residual Training) pipeline, a two-stage approach targeted at overcoming common limitations in event-based video frame prediction. Traditional approaches reliant on event-based optical flow are prone to artifacts such as blurring and holes due to imprecise pixel warping. ER-VAE sidesteps direct flow estimation by directly learning to encode asynchronous event streams—rasterized as event frames—into a latent residual that, when combined with the anchor frame’s latent representation, enables the reconstruction of the target frame. This residual, defined as

$$\mathbf{z}_{\mathrm{res}} = \mathbf{z}_{t+1} - \mathbf{z}_{t}$$

where $\mathbf{z}_t$ and $\mathbf{z}_{t+1}$ are the latents of the anchor and target frames, is aligned with a latent derived from the event frame, producing an event-conditioned prior for the subsequent diffusion denoising stage.

2. Encoder and Decoder Architecture

The ER-VAE encoder, initialized from the Stable Diffusion v2.1 VAE encoder, is engineered to process event frames $e \in \mathbb{R}^{H \times W \times 2}$, where the two channels correspond to stacked positive and negative event counts between anchor and target frames. The processing comprises:

• Four down-sampling residual convolutional blocks (each halving the resolution), yielding an activation map of dimension $\tfrac{H}{8} \times \tfrac{W}{8} \times 4$.
• A vector-quantization module, identical to Stable Diffusion’s codebook mechanism, maps the continuous activations to discrete latent codes, $$\mathbf{z}_{\mathrm{event}}\in \mathbb{R}^{4\times H/8\times W/8}$$.

Only the encoder and quantization layers are fine-tuned; the Stable Diffusion VAE decoder—consisting of four up-sampling residual blocks—is kept fixed. During reconstruction, the sum $\mathbf{z}_t + \mathbf{z}_{\mathrm{event}}$ is decoded to form the predicted frame $\hat I_{t+1}$.

3. Variational Formulation and Loss Objective

Despite the “VAE” nomenclature, ER-VAE does not employ either a KL divergence regularization or stochastic latent sampling. Instead, it is cast as a deterministic latent mapping, where both the variational posterior and the conditional prior are degenerate Dirac delta distributions:

$$q_{\phi}(z \mid e, r) = \delta(z - f_\phi(e)), \quad p_\theta(z \mid e) = \delta(z - f_\phi(e))$$

Consequently, the traditional ELBO collapses to a pure reconstruction-alignment loss. The training objective is:

$$\mathcal{L}_{\mathrm{E2R}} = \|\hat{\mathbf{z}}_{\mathrm{res}} - \mathbf{z}_{\mathrm{res}}\|_{1} + \gamma \|\hat I_{t+1} - I_{t+1}\|_{2}^{2}$$

where $\mathbf{z}_t$0 is the event-derived residual prediction, $\mathbf{z}_t$1 is an image loss weight (cosine-annealed from 0.3 to 0.5; final value approximately $\mathbf{z}_t$2), and $\mathbf{z}_t$3 is the reconstructed image. No cross-attention or additional alignment modules are used; alignment is strictly enforced via these L1 and L2 losses in latent and pixel spaces.

4. Training Procedure and Implementation

ER-VAE utilizes the following training protocol:

• Base VAE: Stable Diffusion 2.1, with the decoder frozen and encoder+quantization modules fine-tuned.
• Latent shape: $\mathbf{z}_t$4 (channels = 4).
• Optimizer: AdamW with learning rate $\mathbf{z}_t$5 and weight decay $\mathbf{z}_t$6.
• Batch size: 64 across 4 NVIDIA A6000 GPUs with gradient accumulation.
• Mixed precision (FP16) is used for efficiency.
• Number of iterations: 70k, following a curriculum DLT schedule for robustness.
• VAE scaling factor: identical to Stable Diffusion 2.1, which empirically improves stability.
• KL regularization ($\mathbf{z}_t$7): set to 0, omitted entirely.

The event frame is derived by rasterizing asynchronous events between times $\mathbf{z}_t$8 and $\mathbf{z}_t$9, and the ground-truth residual is computed by subtracting the anchor frame’s latent from the target frame’s latent.

5. Empirical Results and Ablation Analysis

Empirical findings from ablation studies demonstrate the impact of various training and architectural choices:

Metric/Setting	Base Value	After ER-VAE
Cosine similarity (event vs. residual)	0.21	0.45
L1 distance (event vs. residual)	0.63	0.30
One-frame PSNR (with VAE scaling)	—	+1.2 dB
Image MSE (with $\mathbf{z}_{t+1}$0)	—	$\mathbf{z}_{t+1}$10.2 dB higher PSNR/SSIM

• Omission of the KL term consistently yields improved alignment and convergence.
• DLT augmentation, which involves a curriculum over temporal intervals (1→2→3 frames), improves multi-frame prediction fidelity. For instance, on the HS-ERGB dataset (7-frame prediction), PSNR improves from 24.87 to 26.10; on GoPro (15-frame prediction), from 14.09 to 14.55.
• During inference, the ER-VAE’s output latent is blended with Gaussian noise to initialize diffusion, with a 0.3 weighting on $\mathbf{z}_{t+1}$2 providing optimal trade-offs for PSNR, SSIM, and LPIPS.
• The MSE image loss ($\mathbf{z}_{t+1}$3) plays a stabilizing role, particularly in final PSNR/SSIM statistics.

A plausible implication is that the deterministic latent alignment paradigm, as instantiated in ER-VAE, is preferable for event-to-residual mapping over conventional stochastic VAE formulations in this regime (Kong et al., 19 Dec 2025).

6. Significance and Distinctiveness

ER-VAE is distinguished from conventional VAEs in both its deterministic formulation and its explicit leveraging of event data alignment in latent space. It avoids flow estimation and direct pixel-warping, mitigating spatial artifacts endemic to prior event-based synthesis methods. Its explicit residual learning, anchored by pre-trained diffusion model latents and a curriculum over temporal intervals, produces temporally consistent, high-fidelity frame predictions. The tight alignment between event-encoded and true residual latents establishes a strong event-conditioned prior, enabling Stage 2’s diffusion U-Net to denoise to final reconstructions with significantly improved sharpness and consistency (Kong et al., 19 Dec 2025).

Future research directions may focus on extending ER-VAE-like approaches to unsupervised regimes, integrating cross-attention for richer event-to-latent mapping, or scaling to long-range frame prediction scenarios via hierarchical or recurrent incorporation of event information. However, the strict elimination of stochastic latent regularization in favor of deterministic alignment emerges as a central, empirically validated design principle in DESSERT’s ER-VAE module.