Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing (2512.17909v1)

Abstract: Modern Latent Diffusion Models (LDMs) typically operate in low-level Variational Autoencoder (VAE) latent spaces that are primarily optimized for pixel-level reconstruction. To unify vision generation and understanding, a burgeoning trend is to adopt high-dimensional features from representation encoders as generative latents. However, we empirically identify two fundamental obstacles in this paradigm: (1) the discriminative feature space lacks compact regularization, making diffusion models prone to off-manifold latents that lead to inaccurate object structures; and (2) the encoder's inherently weak pixel-level reconstruction hinders the generator from learning accurate fine-grained geometry and texture. In this paper, we propose a systematic framework to adapt understanding-oriented encoder features for generative tasks. We introduce a semantic-pixel reconstruction objective to regularize the latent space, enabling the compression of both semantic information and fine-grained details into a highly compact representation (96 channels with 16x16 spatial downsampling). This design ensures that the latent space remains semantically rich and achieves state-of-the-art image reconstruction, while remaining compact enough for accurate generation. Leveraging this representation, we design a unified Text-to-Image (T2I) and image editing model. Benchmarking against various feature spaces, we demonstrate that our approach achieves state-of-the-art reconstruction, faster convergence, and substantial performance gains in both T2I and editing tasks, validating that representation encoders can be effectively adapted into robust generative components.

• The paper identifies limitations in current encoders and introduces PS-VAE to address off-manifold artifacts and poor reconstruction fidelity.
• It combines KL-regularized semantic autoencoding with pixel-level loss to balance high-level semantic guidance and fine detail synthesis.
• Empirical results demonstrate significant improvements, with rFID reduced to 0.203 and SSIM increased to 0.817 in both generation and editing tasks.

Unifying Semantic Representations and Pixel Fidelity for Text-to-Image Generation and Editing

Introduction

The paper "Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing" (2512.17909) analyzes the limitations of current Latent Diffusion Models (LDMs) when moving beyond the low-level latent spaces encoded by Variational Autoencoders (VAEs). The authors identify that while representation encoders trained via self-supervised learning (e.g., DINOv2, SigLIP2) deliver high-level semantic features that excel in discriminative tasks, their adaptation for generative modeling suffers due to two core obstacles: inadequate regularization of high-dimensional latent spaces (yielding off-manifold artifacts), and insufficient pixel-level reconstruction fidelity (leading to poor synthesis of fine-grained details). The work systematically diagnoses these bottlenecks and introduces the Pixel–Semantic VAE (PS-VAE), providing a methodology for constructing a compact, semantically rich, and high-fidelity generative latent space suitable for unified text-to-image synthesis and instruction-guided image editing.

Analysis of Generative Latent Spaces: RAE vs VAE

The study involves a rigorous empirical and theoretical comparison between Representation Autoencoder (RAE)-based diffusion and VAE-based diffusion. RAE leverages frozen representation encoder features such as DINOv2, which provide strong semantic grounding but suboptimal image reconstruction. VAEs, optimized for pixel-level fidelity, achieve superior structural and texture reproduction but lack semantically organized latent spaces for prompt-driven tasks.

Figure 1: Visualization of RAE and VAE outputs reveals RAE's advantage in prompt-following but exposes severe reconstruction and generative artifacts, especially in structural and texture details.

RAE delivers faster initial coverage in text-to-image generation yet fails to maintain structural integrity and fine-scale details, as evidenced by strong artifacts and detail inconsistencies in both generation and editing. The observed performance gap between RAE and VAE is more pronounced in generation tasks and cannot be explained solely by differences in pixel-level reconstruction metrics.

Off-Manifold Generation from High-Dimensional Representation Spaces

The authors theoretically and experimentally attribute the generative shortcomings of RAE to off-manifold generation within unconstrained high-dimensional representation spaces. The phenomenon is rigorously quantified via a toy experiment: learning to diffuse over a 2D "PS"-shaped manifold embedded in an 8D ambient space. This setting exacerbates the formation of undefined or out-of-distribution (OOD) samples, ultimately leading to model outputs that deviate significantly from the true data manifold.

Figure 2: Increased feature dimensionality amplifies off-manifold sample drift, as shown by the far greater dispersion in 8D versus 2D latent learning tasks.

The manuscript formally decomposes the denoising trajectory, establishing that off-manifold drift and inefficient model capacity allocation arise fundamentally from the mismatch between ambient and intrinsic dimensions in the representation encoder output, underscoring the necessity of compact latent regularization.

Construction of PS-VAE: Semantic-Pixel Compression

To resolve off-manifold generation, the authors project unconstrained encoder features into a compact latent space using a KL-regularized semantic autoencoder (S-VAE), with feature dimensionality reduced (e.g., to 96 channels). This regularization aligns the latent distribution with the valid pixel decoder manifold, improving generative stability and performance.

Subsequently, to recover fine-grained reconstruction fidelity critical for realistic generation and editing, the frozen encoder is unfrozen and trained jointly with a pixel-level loss, simultaneously preserving semantic structure through the original encoder's feature reconstruction. This results in the PS-VAE architecture.

Figure 3: PS-VAE training regularizes the semantic structure while enabling propagation of pixel-level gradients for fine detail preservation.

Empirically, PS-VAE yields state-of-the-art stride-16 VAE reconstruction metrics: rFID is improved from 0.534 (MAR-VAE) to 0.203 and SSIM from 0.715 to 0.817. In open-domain generation (GenEval: 75.8→76.6) and editing (Editing Reward: 0.06→0.22), PS-VAE demonstrates substantial gains.

Unified Generation and Editing: Coverage, Fidelity, and Instruction Adherence

PS-VAE enables faster and more robust convergence across both text-to-image and image-editing benchmarks. Semantic structure and strong regularization afford rapid model learning and superior prompt-following capability, whereas enriched pixel fidelity enables realistic output synthesis. Models trained solely with pixel objectives exhibit degraded prompt-following, while purely semantic models fail at structural and texture reproduction.

Figure 4: PS-VAE achieves superior generative and editing coverage, outperforming semantic-only (RAE) and pixel-only (VAE) across evaluation metrics.

Figure 5: PS-VAE editing examples preserve both semantic correctness and fine-grained visual details, outperforming RAE in consistency and instruction-following.

Scale, Latent Dimensionality, and Model Capacity

Scaling experiments reveal that higher channel counts in the latent space support superior performance ceilings when paired with larger backbone diffusion models. For example, scaling from 653M to 1708M parameters increases GenEval, DPG-Bench, and Editing Reward, confirming the latent's utility for expressive generation. The generation metrics saturate around 96 latent channels—further increases favor pixel fidelity but destabilize semantic alignment.

Figure 6: Scaling backbone parameter count alongside latent channel width improves generative and editing benchmarks, with PS-VAE_{96c} outperforming PS-VAE_{32c} at scale.

Figure 7: Channel ablation demonstrates that 96-dimensional latent spaces optimally balance semantic structure and detail fidelity without overfitting to high-frequency information.

Generalization to Alternate Encoders: SigLIP2

Replacing DINOv2 with SigLIP2 in PS-VAE yields comparable reconstruction and generation metrics (e.g., GenEval: 76.56 vs. 77.14), with negligible loss in zero-shot discriminative ability on VBench/MME benchmarks. This confirms that PS-VAE generalizes across state-of-the-art representation encoders and can serve as the unified backbone for multimodal vision-LLMs.

Failure Modes and Ablations

Direct pixel enrichment in high-dimensional spaces yields shortcut reconstruction: reconstruction metrics improve, but generation degrades due to poor semantic constraint. This is attributed to sparse channel reliance and ineffective propagation of semantic structure in high-dimensional manifolds, supporting the necessity of compact latent regularization.

Practical and Theoretical Implications

The PS-VAE framework demonstrates that robust, semantically structured, and pixel-fidelity latent spaces are essential for unified vision generation and understanding. The methodology eliminates off-manifold artifacts, supports instruction-driven editing, and is compatible with contemporary foundation models (DINOv2, SigLIP2). This design supports bidirectional transfer between discriminative and generative paradigms, providing a candidate architecture for future vision-language foundation models that demand both high-level understanding and text/image synthesis.

Figure 8: PS-VAE_{96c} generates high-fidelity, semantically accurate images for complex prompts and compositions, even at modest ($256 \times 256$) resolutions.

Conclusion

This work demonstrates that high-fidelity generative modeling in text-to-image and editing requires a unified latent space combining semantic structure and pixel-level reconstruction. PS-VAE achieves this via a principled transformation of representation encoder outputs into a compact, regularized latent, yielding state-of-the-art results in both reconstruction and generative tasks. These findings establish a concrete pathway for unifying perception and generation in multimodal models, ensuring latent spaces are simultaneously interpretable semantically and are capable of faithful visual synthesis. Future research directions include scaling PS-VAE to higher resolutions, adaptive fusion with LLMs, and joint optimization with multimodal architectures to further advance unified visual intelligence.