LUA: Latent Upscaler for Diffusion Models

• Latent Upscaler Adapter (LUA) is a modular component that upscales latent representations to enable high-resolution synthesis in diffusion pipelines.
• It integrates a feed-forward SwinIR-style upsampler and a training-free region-adaptive latent upsampler (RALU) between the diffusion generator and VAE decoder.
• Empirical evaluations show LUA achieves notable runtime reductions and improvements in perceptual quality while generalizing across diverse latent spaces.

The Latent Upscaler Adapter (LUA) is a modular component for generative diffusion systems, designed to enable high-resolution synthesis by manipulating the generator’s latent codes prior to final decoding. LUA encompasses two principal methodologies established in recent literature: the fast feed-forward SwinIR-style upsampler (Razin et al., 13 Nov 2025), and the training-free region-adaptive latent upsampling (RALU) framework (Jeong et al., 11 Jul 2025). Both variants integrate seamlessly into modern text-to-image pipelines by situating between the frozen diffusion backbone and the frozen VAE decoder, requiring negligible changes to base model operation and exhibiting strong generalization across diverse latent spaces. LUA addresses the computational and fidelity challenges of scaling beyond native training resolutions, achieving substantial runtime reductions and maintaining perceptual quality even at 2× and 4× upsampling factors.

1. Motivations and Foundations

Diffusion models, including Stable Diffusion (SDXL, FLUX), are traditionally limited to fixed synthesis resolutions imposed by training objectives and decoder architectures. Attempting naïve upsampling—either by direct high-resolution sampling or post-hoc image-space super-resolution—typically incurs quadratic computational cost and visual artifacts such as repetition, texture breakdown, and geometric distortion. Conventional image-space SR (e.g., SwinIR) must operate over all pixels and often yields oversmoothed details and ringing artifacts.

LUA circumvents these bottlenecks by operating in the latent space, where the spatial token count is significantly lower, and the semantic structure is preserved. Two primary approaches have emerged:

• Feed-Forward Latent Upscaling (Razin et al., 13 Nov 2025): Employing a pretrained SwinIR-style backbone with scale-specific pixel-shuffle heads, LUA predicts high-resolution latents from low-resolution outputs in a single forward pass, followed by only one VAE decoding.
• Region-Adaptive Latent Upsampling (RALU, (Jeong et al., 11 Jul 2025)): Utilizing a training-free multi-stage approach, RALU adapts resolution only where necessary, employing low-resolution denoising, artifact-focused upsampling, and noise-timestep rescheduling (NT-DM) to avoid aliasing and noise mismatch artifacts.

Both models function as drop-in adapters that preserve the integrity of the diffusion backbone and decoder, enabling practical and scalable high-quality synthesis.

2. Architecture and Integration

Feed-Forward LUA (Razin et al., 13 Nov 2025)

• Input: Low-resolution latent $z \in \mathbb{R}^{h \times w \times C}$ returned by the diffusion generator $G$.
• Processing: $z$ passes through a 1×1 convolution (for VAE compatibility), shared SwinIR backbone $\varphi(z)$ (layernorm, $N$ residual Swin Transformer blocks, attention and MLP), and either a $2\times$ or $4\times$ pixel-shuffle head ($U_2$, $U_4$).
• Output: Upscaled latent $$\hat{y} = U_\alpha(\varphi(z)) \in \mathbb{R}^{\alpha h \times \alpha w \times C}$$, where $\alpha\in\{2,4\}$; decoded once via $D(\hat{y})$.

RALU LUA (Jeong et al., 11 Jul 2025)

• Stage 1: Downsample full latent $x_1$ by factor 2, perform denoising diffusion on $x_1^L$ (low-res).
• Stage 2: Estimate clean latent via Tweedie’s formula, decode with VAE, perform edge detection (e.g., Canny), score and select top-k artifact-prone patches, upsample corresponding latents (nearest-neighbor), inject correlated noise, and reschedule timesteps for continued denoising on mixed-resolution latent.
• Stage 3: Upsample all remaining latents to full resolution, reschedule noise-timesteps, final DiT sampling and VAE decode.

Integration Point: LUA is inserted between the generator and the VAE decoder, requiring no retraining or modification to either. For region-adaptive upsampling, localized patch selection and upsampling are performed only for artifact-prone areas before global refinement.

Variant	Architecture	Upsample Factor	Placement
Feed-Forward	SwinIR backbone + pixel-shuffle heads	2×, 4×	Before VAE decode
RALU	Multi-stage spatial upsampling	Adaptive	During latent denoising

3. Mathematical Formulation

Feed-Forward LUA

Given diffusion generator output $z_{\text{low}} = G(c, \epsilon)$:

• Upscaling: $$\hat{y} = U_\alpha(z_{\text{low}}) \in \mathbb{R}^{\alpha h \times \alpha w \times C}$$
• Decoding: $\hat{x} = D(\hat{y})$
• Loss function:
  • Stage I: $$L_{\text{SI}} = \alpha_1 \|\hat{y} - z_{\text{HR}}\|_1 + \beta_1 \|\text{FFT}(\hat{y}) - \text{FFT}(z_{\text{HR}})\|_1$$
  • Stage II: Includes coarse image consistency and blur-corrected differences.
  • Stage III: Pixel-space refinement using edge-aware gradient localization $EAGLE$.

RALU LUA

• Clean latent estimate: $$\hat{x}_0 = x_t + \sigma_t^2 \nabla_{x_t} \log p_\theta(x_t, t)$$
• Patch scoring: $s_p = \sum_{(i,j) \in p} E(i, j)$, $E$ from edge detection.
• Upsampling mapping: $Up_{\text{NN}}(x^L) = K \cdot x^L$ (block-diagonal nearest-neighbor kernel).
• Noise-timestep rescheduling:

$$x' = a \ \text{Up}(x_{e_k}) + b z, \quad z \sim N(0, \Sigma')$$

with $a, b, s_{k+1}$ computed as:

$$s_{k+1} = \frac{e_k}{\frac{1 - e_k}{\sqrt{c}} + e_k}$$

Artifactual mismatch suppression is further achieved by matching the stage-mixed sampling PDF to the pretrained DiT’s original PDF, minimizing Jensen-Shannon divergence.

4. Empirical Evaluation

Feed-Forward LUA (Razin et al., 13 Nov 2025)

• Runtime: At 1 024² resolution, SDXL+LUA requires 1.42 s/image versus 2.47 s for pixel-space SwinIR. At 2 048², 3.52 s vs 6.29 s; at 4 096², 6.87 s (LUA) vs 7.29 s.
• Quality Metrics:
  • FID@2 048: LUA = 180.80; SwinIR = 183.16.
  • pFID(1024-crop)@2 048: 97.90 vs 100.09.
  • KID@2 048: 0.0018 vs 0.0020.
  • CLIP-score@2 048: 0.764 vs 0.757.
• Ablation (LPIPS Table 4): Full curriculum yields LPIPS=0.138 (512→1024×2), compared to 0.150 and 0.172 for Stage III and II/III removals, respectively.

RALU LUA (Jeong et al., 11 Jul 2025)

• On FLUX.1-dev (50-step baseline): Up to 7.02× speedup (≈3.37 s), FID ≈28.68 (baseline 30.07), NIQE 6.87 vs 6.75, CLIP-IQA 0.681 vs 0.707, GenEval 0.646 vs 0.665.
• On Stable Diffusion 3 (28-step baseline): 3.02× speedup (1.38 s), FID 23.29 (vs 27.47), NIQE 5.44 (vs 6.09), CLIP-IQA 0.645 (vs 0.692), GenEval 0.597 (vs 0.673).
• Combination with token-caching methods: ~7.94× speedup for FLUX with ≈3% drop in metrics.

Qualitative analysis reveals LUA maintains crisp fine structure (e.g., eyelashes, fur) and avoids artifacts common to image-space super-resolution such as halos and texture drift.

5. Generalization, Ablations, and Transfer

LUA displays robust cross-VAE generalization; adapting the input convolution suffices to handle different VAE channel dimensions (e.g., FLUX/SD3, C=16; SDXL, C=4). Stage III fine-tuning for only 20k steps on a modest latent pair cohort restores latent statistics, obviating full retraining. Joint training of the backbone with multi-scale pixel-shuffle heads outperforms separate models (PSNR=32.54 vs 31.92, LPIPS=0.138 vs 0.150 for $\times2$ upsampling). Implicit upsamplers (LIIF) are consistently outperformed by explicit pixel-shuffle heads.

Removal of latent or latency-aware loss stages degrades perceptual metrics and elevates spectral mismatches, signifying the necessity of full curriculum scheduling. A plausible implication is that latent-pixel coupling and edge-aware refinement constitute an essential synergy for high-fidelity latent upsampling pipelines.

6. Practical Deployment and Limitations

LUA is a lightweight module (≈6M parameters; 6-block SwinIR backbone) requiring only insertion before the VAE decoder and selection of upsampling head. The computational cost is roughly $1/s^2 \approx 1/64$ of equivalent pixel-space SR, as it operates on token grids rather than decoded images. No changes to the base model weights or sampling schedules are required.

Failure modes stem from the fact that LUA upscales whatever latent microstructure the generator provides; artifacts, bias, or irrecoverably lost fine detail in the latent cannot be reconstructed post hoc. At low native resolutions, highest-resolution baselines still produce marginally superior results due to lost information in compact latents.

Future directions include coupling LUA with lightweight latent consistency modules or denoisers to correct for base generator biases and extension to temporal coherence for video applications. LUA’s real-world deployability is enhanced by its training-free variant (RALU) and single-pass design, enabling practical use in latency-critical or on-device systems.

7. Conceptual Significance and Outlook

The Latent Upscaler Adapter encapsulates a decisive paradigm for scalable, adaptive high-resolution synthesis in diffusion pipelines. It bridges the spatial scalability gap imposed by native model resolutions, providing an efficient mechanism for structural and perceptual fidelity preservation at increased scales. LUA’s training-free, modular architecture foregrounds the potential for deployment flexibility, cross-model generalization, and compositional integration with other acceleration strategies.

This suggests that latent-space adapters mark a strategic turning point for generative model scaling, offering both computational and qualitative advances without exacerbating inference demands or retraining overhead.