How best to learn latent representations for diffusion models

Determine the optimal approach for learning latent representations that will be modeled by diffusion generative models, specifying training and regularization strategies that yield high-quality generation while maintaining interpretable control over latent information content.

Background

Latent representations enable diffusion models to scale efficiently to high-resolution image, video, and audio generation. The original Latent Diffusion Model employs a VAE-style KL penalty with a manually chosen weight, which complicates reasoning about the latent information content and bitrate. Recent alternatives use semantically focused or heavily regularized autoencoders that simplify training but often lose high-frequency details and reconstruction fidelity.

The paper highlights a fundamental trade-off between latent information density and decoder reconstruction quality, motivating the need for principled latent learning methods. Unified Latents (UL) is proposed as a partial answer, co-training a diffusion prior and diffusion decoder and linking encoder noise to the prior’s minimum noise level to provide an interpretable bitrate bound. The explicit statement in the introduction acknowledges that, despite such efforts, the broader question of how best to learn latents remains unresolved.

References

However, it remains unclear how best to learn such latents.

— Unified Latents (UL): How to train your latents  (2602.17270 - Heek et al., 19 Feb 2026) in Section 1 (Introduction)

Lastly, in the case of Gaussian diffusion, we have only considered the most basic form of embedding, leaving open the exploration of richer encoding maps.

— Parallelism, critical windows, and separations among diffusion language models  (2609.20539 - Chen et al., 17 Sep 2026) in Outlook section

Learning latent representations that can effectively encode long passages and work for diffusion is still an open challenge that we address in this paper.

— Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders  (2609.27248 - Pathak et al., 23 Sep 2026) in Section 2, Related Work, subsection “Text Diffusion”

In addition, while we observe that semantic latents can accelerate diffusion training and that input-level pre-merge compression outperforms post-hoc token merging, we do not yet have a complete understanding of which properties of semantic encoders are most important for generation.

— DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence  (2609.39222 - Huang et al., 30 Sep 2026) in Appendix, subsection “Limitations and Future Work”