---
title: 'ER-VAE: Event-to-Residual Alignment VAE'
url: https://www.emergentmind.com/topics/event-to-residual-alignment-variational-autoencoder-er-vae
type: topic
---

# ER-VAE: Event-to-Residual Alignment VAE

The Event-to-Residual Alignment Variational Autoencoder (ER-VAE) is a deterministic latent mapping module designed for event-driven video frame prediction. Employed as Stage 1 within the DESSERT framework, ER-VAE transforms temporally accumulated event data into a latent residual approximating the true inter-frame latent difference, derived from a pre-trained Stable Diffusion VAE. By tightly aligning event-derived latent representations with ground-truth residuals in latent space via an L1 alignment loss and in image space with an L2 reconstruction loss, ER-VAE provides an effective event-conditioned prior for diffusion-based single-frame synthesis, facilitating superior temporal consistency and fidelity in synthesized video frames [2512.17323].

## 1. Position within the DESSERT Framework

ER-VAE serves as the first stage of the DESSERT (Diffusion-based Event-driven Single-frame Synthesis via Residual Training) pipeline, a two-stage approach targeted at overcoming common limitations in event-based video frame prediction. Traditional approaches reliant on event-based optical flow are prone to artifacts such as blurring and holes due to imprecise pixel warping. ER-VAE sidesteps direct flow estimation by directly learning to encode asynchronous event streams—rasterized as event frames—into a latent residual that, when combined with the anchor frame’s latent representation, enables the reconstruction of the target frame. This residual, defined as
$$
\mathbf{z}_{\mathrm{res}} = \mathbf{z}_{t+1} - \mathbf{z}_{t}
$$
where $\mathbf{z}_t$ and $\mathbf{z}_{t+1}$ are the latents of the anchor and target frames, is aligned with a latent derived from the event frame, producing an event-conditioned prior for the subsequent diffusion denoising stage.

## 2. Encoder and Decoder Architecture

The ER-VAE encoder, initialized from the Stable Diffusion v2.1 VAE encoder, is engineered to process event frames $e \in \mathbb{R}^{H \times W \times 2}$, where the two channels correspond to stacked positive and negative event counts between anchor and target frames. The processing comprises:

- Four down-sampling residual convolutional blocks (each halving the resolution), yielding an activation map of dimension $\tfrac{H}{8} \times \tfrac{W}{8} \times 4$.
- A vector-quantization module, identical to Stable Diffusion’s codebook mechanism, maps the continuous activations to discrete latent codes, $\mathbf{z}_{\mathrm{event}}\in \mathbb{R}^{4\times H/8\times W/8}$.

Only the encoder and quantization layers are fine-tuned; the Stable Diffusion VAE decoder—consisting of four up-sampling residual blocks—is kept fixed. During reconstruction, the sum $\mathbf{z}_t + \mathbf{z}_{\mathrm{event}}$ is decoded to form the predicted frame $\hat I_{t+1}$.

## 3. Variational Formulation and Loss Objective

Despite the “VAE” nomenclature, ER-VAE does not employ either a KL divergence regularization or stochastic latent sampling. Instead, it is cast as a deterministic latent mapping, where both the variational posterior and the conditional prior are degenerate Dirac delta distributions:
$$
q_{\phi}(z \mid e, r) = \delta(z - f_\phi(e)), \quad p_\theta(z \mid e) = \delta(z - f_\phi(e))
$$
Consequently, the traditional ELBO collapses to a pure reconstruction-alignment loss. The training objective is:
$$
\mathcal{L}_{\mathrm{E2R}} =
\|\hat{\mathbf{z}}_{\mathrm{res}} - \mathbf{z}_{\mathrm{res}}\|_{1}
+ \gamma \|\hat I_{t+1} - I_{t+1}\|_{2}^{2}
$$
where $\hat{\mathbf{z}}_{\mathrm{res}} = f_{\phi}(e)$ is the event-derived residual prediction, $\gamma$ is an image loss weight (cosine-annealed from 0.3 to 0.5; final value approximately $0.4$), and $\hat I_{t+1}$ is the reconstructed image. No cross-attention or additional alignment modules are used; alignment is strictly enforced via these L1 and L2 losses in latent and pixel spaces.

## 4. Training Procedure and Implementation

ER-VAE utilizes the following training protocol:

- Base VAE: Stable Diffusion 2.1, with the decoder frozen and encoder+quantization modules fine-tuned.
- Latent shape: $4 \times \tfrac{H}{8} \times \tfrac{W}{8}$ (channels = 4).
- Optimizer: AdamW with learning rate $5 \times 10^{-4}$ and weight decay $1 \times 10^{-2}$.
- Batch size: 64 across 4 NVIDIA A6000 GPUs with gradient accumulation.
- Mixed precision (FP16) is used for efficiency.
- Number of iterations: 70k, following a curriculum DLT schedule for robustness.
- VAE scaling factor: identical to Stable Diffusion 2.1, which empirically improves stability.
- KL regularization ($\beta$): set to 0, omitted entirely.

The event frame is derived by rasterizing asynchronous events between times $t$ and $t+1$, and the ground-truth residual is computed by subtracting the anchor frame’s latent from the target frame’s latent.

## 5. Empirical Results and Ablation Analysis

Empirical findings from ablation studies demonstrate the impact of various training and architectural choices:

| Metric/Setting                           | Base Value | After ER-VAE |
|-------------------------------------------|------------|--------------|
| Cosine similarity (event vs. residual)    |   0.21     |    0.45      |
| L1 distance (event vs. residual)          |   0.63     |    0.30      |
| One-frame PSNR (with VAE scaling)         |   —        |   +1.2 dB    |
| Image MSE (with $\gamma > 0$)             |   —        |   $\sim$0.2 dB higher PSNR/SSIM |

- Omission of the KL term consistently yields improved alignment and convergence.
- DLT augmentation, which involves a curriculum over temporal intervals (1→2→3 frames), improves multi-frame prediction fidelity. For instance, on the HS-ERGB dataset (7-frame prediction), PSNR improves from 24.87 to 26.10; on GoPro (15-frame prediction), from 14.09 to 14.55.
- During inference, the ER-VAE’s output latent is blended with Gaussian noise to initialize diffusion, with a 0.3 weighting on $\mathbf{z}_{\mathrm{event}}$ providing optimal trade-offs for PSNR, SSIM, and LPIPS.
- The MSE image loss ($\gamma > 0$) plays a stabilizing role, particularly in final PSNR/SSIM statistics.

A plausible implication is that the deterministic latent alignment paradigm, as instantiated in ER-VAE, is preferable for event-to-residual mapping over conventional stochastic VAE formulations in this regime [2512.17323].

## 6. Significance and Distinctiveness

ER-VAE is distinguished from conventional VAEs in both its deterministic formulation and its explicit leveraging of event data alignment in latent space. It avoids flow estimation and direct pixel-warping, mitigating spatial artifacts endemic to prior event-based synthesis methods. Its explicit residual learning, anchored by pre-trained diffusion model latents and a curriculum over temporal intervals, produces temporally consistent, high-fidelity frame predictions. The tight alignment between event-encoded and true residual latents establishes a strong event-conditioned prior, enabling Stage 2’s diffusion U-Net to denoise to final reconstructions with significantly improved sharpness and consistency [2512.17323].

Future research directions may focus on extending ER-VAE-like approaches to unsupervised regimes, integrating cross-attention for richer event-to-latent mapping, or scaling to long-range frame prediction scenarios via hierarchical or recurrent incorporation of event information. However, the strict elimination of stochastic latent regularization in favor of deterministic alignment emerges as a central, empirically validated design principle in DESSERT’s ER-VAE module.

Source: https://www.emergentmind.com/topics/event-to-residual-alignment-variational-autoencoder-er-vae