One Small Step in Latent, One Giant Leap for Pixels: Fast Latent Upscale Adapter for Your Diffusion Models (2511.10629v1)

Abstract: Diffusion models struggle to scale beyond their training resolutions, as direct high-resolution sampling is slow and costly, while post-hoc image super-resolution (ISR) introduces artifacts and additional latency by operating after decoding. We present the Latent Upscaler Adapter (LUA), a lightweight module that performs super-resolution directly on the generator's latent code before the final VAE decoding step. LUA integrates as a drop-in component, requiring no modifications to the base model or additional diffusion stages, and enables high-resolution synthesis through a single feed-forward pass in latent space. A shared Swin-style backbone with scale-specific pixel-shuffle heads supports 2x and 4x factors and remains compatible with image-space SR baselines, achieving comparable perceptual quality with nearly 3x lower decoding and upscaling time (adding only +0.42 s for 1024 px generation from 512 px, compared to 1.87 s for pixel-space SR using the same SwinIR architecture). Furthermore, LUA shows strong generalization across the latent spaces of different VAEs, making it easy to deploy without retraining from scratch for each new decoder. Extensive experiments demonstrate that LUA closely matches the fidelity of native high-resolution generation while offering a practical and efficient path to scalable, high-fidelity image synthesis in modern diffusion pipelines.

• The paper presents LUA, which achieves high-resolution synthesis with a single feed-forward operation in latent space.
• It employs a SwinIR-style transformer backbone with scale-specific pixel-shuffle heads to reduce computational cost and support multi-scale upscaling.
• The approach demonstrates robust cross-model transfer and superior fidelity and runtime compared to conventional pixel-space and iterative methods.

Fast Latent Upscaling in Diffusion Pipelines via Lightweight Adapter Integration

Overview

This paper introduces the Latent Upscaler Adapter (LUA), a plug-and-play module for fast, high-fidelity latent space super-resolution in diffusion models. LUA operates between a pretrained generator and a frozen VAE decoder, upscaling the generator’s latent code just prior to decoding. Unlike conventional approaches which require retraining, supplementary diffusion stages, or expensive pixel-space super-resolution (SR), LUA enables high-resolution synthesis with a single feed-forward operation in latent space, supporting multi-scale ($\times2$, $\times4$) upscaling and robust cross-model transfer with minimal adjustment.

Figure 1: LUA is integrated into diffusion pipelines after the generator and before the frozen VAE decoder, enabling fast latent upscaling and single-pass decoding.

---

Motivation and Related Work

Latent Diffusion Models (LDMs) moved generation into a compressed latent representation, increasing efficiency and enabling higher-fidelity outputs within the constraints of training resolution ($512^2$, $1024^2$). Direct high-resolution generation in pixel space is slow and expensive, while post-hoc image super-resolution often introduces artifacts and incurs latency that scales quadratically with resolution. Recent latent domain approaches (DemoFusion, LSRNA) require multi-stage inference, additional diffusion, and tight coupling to specific VAE architectures.

The central challenge is achieving high-resolution synthesis that preserves semantic fidelity and fine textures without extra diffusion passes or pixel-level SR. Bicubic or naive latent resizing diverges from the valid latent manifold, leading to decoding artifacts. LUA addresses this by learning an explicit mapping that preserves high-frequency latent detail, efficiently supports multiple upscaling factors, and generalizes across decoders.

Figure 2: Comparison of upsampling strategies on FLUX outputs—bicubic latent interpolation produces blurred and aliased images; pixel-space SwinIR sharpens edges but introduces noise; the latent-space LUA preserves stable, clean textures while maintaining efficiency.

---

Latent Upscaler Adapter (LUA): Architecture and Training

LUA consists of a shared SwinIR-style transformer backbone and scale-specific pixel-shuffle heads for $\times2$ and $\times4$ upscaling, trained end-to-end in the latent domain. Only the first convolution is adapted for different VAE channel widths.

• Feed-forward mapping: Given latent $z \in \mathbb{R}^{h \times w \times C}$, LUA outputs $$\hat{z} \in \mathbb{R}^{\alpha h \times \alpha w \times C}$$ for scale $\alpha$. Decoding $\hat{z}$ yields a higher-resolution image.
• Computational savings: Operations are on latents ($h \times w$, typically $1/64$ the pixel count after VAE stride $s{=}8$), yielding $16\times$ pixel increase with linear cost scaling, versus quadratic for pixel space SR.
• Multi-scale support: The shared backbone enables both $\times2$ and $\times4$ upscaling via independent lightweight heads, eliminating the need for specialist per-scale models or less effective arbitrary-scale decoders.
• Cross-model transfer: LUA generalizes across SD3, SDXL, and FLUX by swapping a single input layer and brief fine-tuning, with scale heads and backbone reused unchanged.

  Figure 3: LUA demonstrates cross-model $2\times$ latent upscaling—one adapter seamlessly supports SDXL, SD3, and FLUX with only minimal adaptation to the input layer.

  

  Figure 4: SwinIR-style backbone architecture for LUA, illustrating shared encoder–decoder pathways and scale-specific pixel-shuffle heads.

---

Training Strategy: Multi-Stage Latent-Pixel Curriculum

Optimal latent SR requires balancing latent structure, spectral fidelity, decoded image quality, and edge sharpness. A progressive three-stage curriculum is proposed:

1. Stage I: Latent-domain alignment—$\ell_1$ and FFT losses on latent, preserving manifold structure and high-frequency detail.
2. Stage II: Joint latent–pixel consistency—adds downsampling and high-frequency pixel-domain losses, connecting latent fidelity to decoded image appearance and suppressing noise.
3. Stage III: Pixel-domain edge refinement—fine-tunes output images using edge-aware losses (EAGLE) to sharpen boundaries and reduce grid/ringing artifacts.

Empirically, the full curriculum is necessary for optimal decoded structure and perceptual quality. Ablation studies show dropping any stage degrades PSNR and LPIPS metrics.

Figure 5: Effect of the three-stage curriculum—each stage incrementally improves latent reconstruction and decoded image sharpness, with Stage III concentrating high-frequency detail around visually salient regions.

---

Experimental Results

Evaluation on OpenImages shows LUA achieves competitive or superior fidelity and runtime compared to a wide range of high-resolution generation methods: iterative (ScaleCrafter, HiDiffusion), reference-based (DemoFusion, LSRNA), direct high-res sampling (SDXL), and pixel-space SR (SwinIR).

Quantitative highlights:

• At $2048^2$: SDXL+LUA achieves FID 180.80 and pFID 97.90, outperforming SDXL+SwinIR (FID 183.16, pFID 100.09) and requiring less than half the runtime (3.52\,s vs 6.29\,s, batch size 1 on H100).
• At $4096^2$: SDXL+LUA maintains best single-pass fidelity (FID 176.90) and the lowest latency (6.87\,s), markedly improving over multi-stage pipelines (DemoFusion: 225.77\,s, LSRNA-DemoFusion: 91.64\,s).
• At $1024^2$: LUA lags marginally behind direct generation methods (SDXL Direct, LSRNA-DemoFusion), attributed to inherent base latent limitations, but achieves top patch-level fidelity.
• Cross-model, multi-scale adaptation with a single backbone is validated with FID, KID, CLIP metrics nearly matching specialist models and clear runtime advantages.

  Figure 6: Qualitative comparisons at $2048^2$ and $4096^2$—LUA produces sharp, artifact-free textures in extreme-scale images, outperforming direct and pixel-space SR methods in visual coherence, edge fidelity, and runtime.

---

Ablations and Design Choices

• The multi-stage curriculum is essential for edge sharpness and microstructure preservation; pixel-only or latent-only objectives degrade result quality.
• A joint multi-head design for scale factors outperforms separated specialist models and coordinate-based upsamplers (LIIF), both in perceptual metrics and practical resource overhead.

---

Implications and Future Directions

LUA sets a new standard for single-pass latent upscaling in modern diffusion pipelines, combining the efficiency of the latent domain with state-of-the-art fidelity and minimal deployment friction. The architecture (adapter-insertion with minimal fine-tuning) enables rapid adoption across model families and is amenable to hardware acceleration for practical high-res image synthesis.

Limitations and future work:

• Base latent artifacts persist through upscaling; latent refinement prior to upscaling could mitigate inherited errors.
• Extension to video-generative tasks will require temporal latent consistency mechanisms; recurrent or temporally-aware adapters represent promising directions.
• The adapter framework may generalize to non-text-driven generative tasks (e.g., semantic layout to image, depth-to-image) where structural preservation during upscaling is crucial.

---

Conclusion

The Latent Upscaler Adapter (LUA) replaces multi-stage or pixel-space super-resolution in diffusion pipelines with a single, efficient latent domain module. It achieves high-resolution, high-fidelity synthesis at $2\times$ and $4\times$ scales in a fraction of the compute time of conventional approaches, supports cross-generator transfer, and delivers state-of-the-art quantitative and qualitative results on standardized benchmarks. This establishes latent adapter-based SR as an efficient and practical pathway for scalable, semantically faithful high-resolution synthesis in contemporary generative pipelines.