One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation (2512.07829v1)

Abstract: Visual generative models (e.g., diffusion models) typically operate in compressed latent spaces to balance training efficiency and sample quality. In parallel, there has been growing interest in leveraging high-quality pre-trained visual representations, either by aligning them inside VAEs or directly within the generative model. However, adapting such representations remains challenging due to fundamental mismatches between understanding-oriented features and generation-friendly latent spaces. Representation encoders benefit from high-dimensional latents that capture diverse hypotheses for masked regions, whereas generative models favor low-dimensional latents that must faithfully preserve injected noise. This discrepancy has led prior work to rely on complex objectives and architectures. In this work, we propose FAE (Feature Auto-Encoder), a simple yet effective framework that adapts pre-trained visual representations into low-dimensional latents suitable for generation using as little as a single attention layer, while retaining sufficient information for both reconstruction and understanding. The key is to couple two separate deep decoders: one trained to reconstruct the original feature space, and a second that takes the reconstructed features as input for image generation. FAE is generic; it can be instantiated with a variety of self-supervised encoders (e.g., DINO, SigLIP) and plugged into two distinct generative families: diffusion models and normalizing flows. Across class-conditional and text-to-image benchmarks, FAE achieves strong performance. For example, on ImageNet 256x256, our diffusion model with CFG attains a near state-of-the-art FID of 1.29 (800 epochs) and 1.70 (80 epochs). Without CFG, FAE reaches the state-of-the-art FID of 1.48 (800 epochs) and 2.08 (80 epochs), demonstrating both high quality and fast learning.

• The paper introduces the Feature Auto-Encoder that adapts pretrained visual encoders to a compact latent space, enabling efficient image generation.
• The methodology employs a single-attention encoder paired with dual decoders, one for feature reconstruction and one for image synthesis, reducing computational overhead.
• Experiments on ImageNet and MS-COCO demonstrate rapid convergence and enhanced semantic consistency, achieving an FID score of 1.29 under CFG guidance.

One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation

Introduction

The paper "One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation" presents a novel approach to integrating pretrained visual representations, commonly used in visual generative models like diffusion models, into the training pipeline for efficient image generation. The authors identify the mismatch between understanding-oriented features and generation-friendly latent spaces as a significant challenge. They propose a framework called Feature Auto-Encoder (\model{}) to address this disparity by adapting high-dimensional visual features to a low-dimensional latent space suitable for generative models. This adaptation is achieved using a single attention layer, coupled with dual decoders for feature reconstruction and image synthesis.

Figure 1: Comparison between various generative models highlighting the channel dimension disparities.

Methodology

Single-Attention Encoder

The core innovation in \model{} lies in its minimalist encoder design. A single attention layer followed by a linear projection is utilized to transform pretrained patch embeddings into compact latent codes. This design significantly reduces computational overhead while effectively compressing latent space without losing essential information. The attention mechanism efficiently removes redundant information shared across patches, enhancing the model's capacity to leverage rich semantic content from pretrained embeddings.

Double Decoder Architecture

The approach separates the tasks of feature reconstruction and image synthesis into two distinct decoders—feature and pixel decoders. The feature decoder reconstructs the original representation, while the pixel decoder translates reconstructed features into high-quality images. This separation allows each component to specialize, ensuring that the semantic richness of the pretrained embeddings is preserved while enabling high-fidelity image generation.

Figure 2: FID Score Converging Curve for the generative model training indicating rapid convergence.

Experiments and Results

The efficacy of \model{} is demonstrated through extensive experiments on ImageNet and MS-COCO datasets, showcasing its versatility across class-conditional and text-to-image generation tasks. Notably, the diffusion model using \model{} achieves an FID of 1.29 on ImageNet with CFG guidance, demonstrating remarkable generative performance and learning efficiency compared to previous models.

Additionally, qualitative evaluations reveal that the integration of \model{} significantly enhances the model's ability to retain cross-image semantic consistency, which is crucial for maintaining high visual fidelity in generated images.

Figure 3: Random samples from ImageNet showcasing the quality of generated images.

Implications and Future Work

The simplicity and effectiveness of \model{} provide a promising direction for future research in generative modeling. The approach's ability to integrate semantic richness from pretrained models without complex architectural modifications or objectives suggests potential applications in domains requiring rapid adaptation to new visual tasks. Further exploration into enhancing encoder design and decoder training could unlock additional performance improvements.

Figure 4: Demonstration of semantic matching across related images, emphasizing the model's preservation of semantic consistency.

Conclusion

\model{} represents a significant step forward in adapting pretrained visual encoders for efficient image generation by incorporating semantic-rich features into a compact, generation-friendly latent space. The method's simplicity and performance efficacy across benchmarks highlight its potential for broad applicability in generative modeling. Future work should focus on refining encoder-decoder interactions and exploring applications in multimodal generative contexts.