RAEv2: Advanced Representation Autoencoder

• RAEv2 is an advanced representation autoencoder architecture that improves generative modeling by aggregating multi-layer encoder outputs.
• It employs a combined reconstruction and representation alignment (REPA) loss to preserve both global semantics and local spatial details.
• Its internal latent-space guidance technique accelerates convergence and boosts sample quality across image, text-to-image, and world-model tasks.

RAEv2 is an advanced representation autoencoder architecture that systematically improves and extends the Representation Autoencoder (RAE) paradigm. Designed to leverage pretrained vision encoders (such as DINOv2 and DINOv3) in place of traditional variational autoencoding modules, RAEv2 introduces several architectural and training innovations. Core developments include a multi-layer representation aggregation from the encoder, unified training with representation alignment, and direct internal guidance by latent-space reparameterization. RAEv2 demonstrates state-of-the-art convergence speed and generative quality across diverse settings, such as image generation, text-to-image synthesis, and world models (Singh et al., 18 May 2026).

1. Generalized Encoder Architecture

The original RAE framework replaces conventional VAE modules with a frozen, pretrained vision encoder $E$ and a compact trainable decoder $D$. Stage 1 minimizes reconstruction loss: $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$ Stage 2 freezes $D$ and trains a diffusion transformer (DiT), $\Phi$, to generate in the latent space $z=E(x)$.

RAEv2 generalizes the latent representation by aggregating the outputs of the final $K$ layers of $E$: $$r = E_{L-K+1}(x) + \cdots + E_{L}(x) \in \mathbb{R}^{N \times d}$$ where $E_\ell(x)$ is the patch-token output at layer $D$0. This multi-layer sum exploits the property that addition in high-dimensional space preserves critical geometric structure. Increasing $D$1 improves Stage 1 reconstruction monotonically—for example, rFID decreases from 0.60 at $D$2 to 0.18 at $D$3. Optimal $D$4 for generative performance is empirically found near $D$5 (guided FID, gFID, reaches 1.06 at 80 epochs).

Earlier encoder layers retain local spatial detail, and deeper layers encode global semantic information. The sum yields latents that are simultaneously rich in semantic and spatial content, enabling effective non-fine-tuned utilization of pretrained representations.

2. Combined RAE and Representation Alignment (REPA) Loss

REPA (Representation Alignment) addresses limitations in spatial structure by aligning intermediate DiT features to the pretrained encoder representation. At a given diffusion step $D$6, with DiT features $D$7, a projection head $D$8 computes: $D$9 REPA distills encoder representations into intermediate model layers, improving preservation of spatial and structural properties.

In Stage 2, the total loss combines flow-matching (velocity or $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$0-prediction) and alignment: $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$1 Empirical analysis indicates RAE components correlate with global representation quality (as measured by linear probing), whereas REPA correlates with spatial structure (self-similarity metrics). Their combination delivers complementary benefits.

Stage-2 Algorithmic Sketch (plain-text pseudocode): $\Phi$2 Here, $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$2 is the guidance scale and $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$3 is the true velocity.

3. Latent-Space Guidance without Auxiliary Models

Traditional classifier-free guidance (CFG) is ineffective in RAE, with no improvement over baseline (gFID = 3.86 vs 3.75). Prior variants require training a secondary, weaker diffusion model for AutoGuidance (AG), achieving gFID = 3.31. RAEv2 employs a "free" internal guidance technique by reparameterizing DiT outputs in the latent space:

• DiT predicts velocity $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$4.
• Compute $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$5 and $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$6.
• Internal guidance in latent space:

$$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$7

• Reconvert for sampling: $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$8.
• Requires only one model and one pass.

This yields gFID = 1.06 at 80 epochs, outperforming AG (gFID = 1.14). The mechanism provides efficient and effective guidance with no auxiliary model or computational overhead.

4. Evaluation Metrics and Comparative Benchmarks

RAEv2 establishes multiple benchmarks for sample quality and convergence efficiency:

• gFID (Guided FID): Frechet Inception Distance between 50K generated and real images, with $$L_{\rm recon} = \mathbb{E}_{x\sim \text{data}} \bigl[\|D(E(x)) - x\|^2\bigr]$$9 indicating guidance strength.
• FD$D$0: Representation Fréchet Distance, geometric mean across six feature spaces (Inception, ConvNeXt, DINOv2, MAE, SigLIP, CLIP).
• EP_FID@$D$1: Minimum epochs to reach unguided gFID $D$2; quantifies convergence speed.

ImageNet-256 results (DiT^DH-XL, DINOv3-L encoder):

$D$3

RAEv2 achieves more than one order of magnitude speedup in convergence (EP_FID@2 from 177 to 35 epochs vs RAE-XL), and sets final FD$D$4 (state-of-the-art).

5. Empirical Performance across Domains

ImageNet-256: RAEv2 reaches unguided gFID = 2.0 in 35 epochs (vs RAE’s 177). At 80 epochs, achieves gFID = 1.06; FD$D$5.

Text-to-Image Generation (DiT^DH-XL, SiGLIP2-B encoder):

$D$6

Finetuning further increases GenEval and DPG scores (+3-4 and +1-2 points above RAE).

Navigation World Models (RECON dataset):

$D$7

RAEv2-NWM halves FVD compared to RAE and outperforms previous methods.

Key hyperparameters (ImageNet): DiT^DH-XL backbone (28 encoder/2 decoder blocks, hidden 1152/2048, 16 heads), batch size 1024, learning rate annealed from $D$8 to $D$9, 25-epoch warmup, 50-epoch decay; flow matching with $\Phi$0-prediction and ODE (Euler) sampling over 50 steps; REPA head at depth 8 with $\Phi$1.

6. Conclusion and Significance

RAEv2 introduces a simple but highly effective series of improvements to representation autoencoding for generative modeling. The multi-layer sum encoder formulation increases latent expressivity without finetuning. The combined RAE+REPA loss realizes complementary tradeoffs between global semantics and spatial precision. The direct internal guidance technique eliminates the need for auxiliary guidance models, accelerating training and inference. Across image, language-conditional, and world-modeling domains, RAEv2 demonstrates superior convergence speeds (10× over RAE baselines), state-of-the-art sample quality, and robust transferability.

RAEv2 and comprehensive codebase are available at https://raev2.github.io (Singh et al., 18 May 2026).