Region-Adaptive Latent Upsampling (RALU)

• Region-Adaptive Latent Upsampling (RALU) is a spatial acceleration framework for diffusion transformers that adaptively upsamples only artifact-prone regions.
• It employs a three-stage protocol—low-resolution denoising, selective upsampling based on edge detection, and full-resolution refinement—to efficiently balance speed with quality.
• By integrating noise-timestep rescheduling, RALU maintains the statistical consistency of the generative process, achieving up to 7× speed-up with minimal image degradation.

Region-Adaptive Latent Upsampling (RALU) is a training-free, spatial acceleration framework for diffusion transformers (DiTs) that enables high-fidelity image and video generation with substantially reduced inference cost. RALU operates by performing mixed-resolution sampling in the latent space, adaptively upsampling only those regions prone to artifacts, while incorporating mathematically principled noise-timestep rescheduling to seamlessly handle resolution transitions. It achieves up to 7$\times$ speed-up on benchmarks such as FLUX and 3$\times$ on Stable Diffusion 3 with minimal degradation in image quality, and is fully complementary to temporal acceleration techniques such as feature caching (Jeong et al., 11 Jul 2025).

1. Motivation and Problem Statement

Diffusion transformers have demonstrated superior scaling properties and state-of-the-art synthesis quality for image and video generation. Their practicality, however, is constrained by heavy inference costs: the self-attention operation in transformers exhibits $O(N^2)$ complexity with respect to the spatial token count, dominating overall computation. While temporal acceleration strategies (e.g., block- or token-level caching across diffusion steps) address repeated computations over time, they leave the spatial bottleneck unresolved. Naively reducing the spatial resolution introduces two fundamental artifacts:

• Aliasing artifacts occur at object boundaries if low-resolution latents are upsampled too late in the diffusion trajectory.
• Mismatch artifacts—such as grid-like distortions—arise from incompatibilities in noise level and diffusion timestep when transitioning between resolutions without appropriate correction.

RALU directly addresses these issues by (a) targeting only the artifact-prone regions for early upsampling and (b) deploying a noise-timestep distribution-matching mechanism to ensure distributional consistency across resolution changes (Jeong et al., 11 Jul 2025).

2. Three-Stage RALU Protocol

The RALU framework decomposes inference into three spatially adaptive stages, each balancing fidelity against compute cost:

1. Low-resolution denoising: The initial diffusion steps are performed at half-resolution ($R/2 \times R/2$ latents), operating on only $1/4$ the spatial tokens to efficiently model global semantic structure.
2. Region-adaptive upsampling: Artifact-prone regions, identified via edge detection on a decoded image estimate, are selectively upsampled to full resolution. The latent then consists of mixed-resolution tokens, with the DiT applying further diffusion only to the required regions at high spatial detail.
3. Full-resolution refinement: All remaining low-resolution latents are upsampled to full resolution for the final diffusion stage, ensuring detailed structure in the final generation.

Between each stage transition ($1 \rightarrow 2$ and $2 \rightarrow 3$), RALU injects correlated noise and reschedules the diffusion timestep to maintain the generative process on the pretrained model’s statistical trajectory (Jeong et al., 11 Jul 2025).

3. Algorithmic Formulation and Noise–Timestep Rescheduling

At each transition, RALU performs upsampling and noise-timestep correction as follows:

• Region Selection: After the low-resolution diffusion, the denoised latent $\hat{\mathbf{x}}_0$ is estimated using Tweedie's formula:

$$\hat{\mathbf{x}}_0 = (1/\sqrt{\alpha_t})(\mathbf{x}_t - \sqrt{1-\alpha_t} \, \epsilon_\theta(\mathbf{x}_t, t))$$

The estimate is decoded to an image, and a Canny edge detector is applied. The image is partitioned into patches (e.g., $16 \times 16$), and the patches with the highest cumulative edge energy are selected for upsampling.

• Upsampling: Selected patches undergo 2$\times$0 nearest-neighbor upsampling, producing a block-diagonal latent covariance.
• Noise–Timestep Rescheduling (NT-DM): After upsampling, the latent is affinely transformed with noise to match the target distribution at the subsequent diffusion timestep. The transformation

$\times$1

with parameters $\times$2 given by

$\times$3

ensures that the resulting upsampled latent distribution is statistically aligned with the pretrained DiT across all Timestep intervals (Jeong et al., 11 Jul 2025).

• Mixed-Resolution Latents: The composite latent $\times$4 in Stage 2 comprises high-resolution edge patches interleaved with low-resolution background, processed naturally by DiT positional encodings.

4. Computational Complexity and Empirical Performance

RALU achieves substantial reductions in total token FLOPs per generated image:

Framework	Model	Speed-up (×)	FID	Reference
FLUX	DiT-Large	4.1–7.0	28.7	(Jeong et al., 11 Jul 2025)
SD3	DiT-SD3	3.0	23.3	(Jeong et al., 11 Jul 2025)

• Stage 1: $\times$5 FLOPs due to quartered spatial resolution.
• Stage 2: $\times$6 of tokens at full resolution ($\times$7 typically $\times$8–$\times$9), the rest remain low-res.
• Stage 3: Full resolution.

On FLUX ($O(N^2)$0 latent), RALU reduces per-image TFLOPs from $O(N^2)$1 to $O(N^2)$2 (7.02$O(N^2)$3 speed-up); when combined with temporal caching, the speed-up reaches $O(N^2)$4. Quality metrics such as FID and NIQE are minimally impacted, with RALU outperforming alternative spatial sampling and temporal acceleration methods at comparable acceleration factors (Jeong et al., 11 Jul 2025).

5. Implementation Considerations and Integration

• Pretrained Model Compatibility: RALU operates as a post hoc wrapper on any pretrained DiT, requiring no model fine-tuning.
• Region Masking: The region-mask can be implemented as a binary tensor, governing which spatial tokens receive full-resolution processing during Stage 2.
• Caching Synergy: RALU can be seamlessly combined with token-wise temporal feature caching (as in ToCa), further reducing inference latency.
• Region Ratio Tuning: The upsampling ratio trades off fidelity (notably for text alignment and fine structure) against speed, with diminishing returns beyond $O(N^2)$5 upsampled tokens (Jeong et al., 11 Jul 2025).

6. Strengths, Limitations, and Future Directions

Strengths:

• Substantial empirical speed-up (3–7$O(N^2)$6) for high-resolution diffusion transformers without retraining.
• Artifact suppression: By adaptively upsampling edge regions, RALU eliminates aliasing and mismatch artifacts observed with baseline spatial acceleration.
• Orthogonality: RALU complements temporal acceleration techniques, multiplying the achievable speed-up.

Limitations:

• The NT-DM mechanism is derived for the flow-matching objective and 2$O(N^2)$7 upsampling; adaptation to score-based or DDIM models, or arbitrary upsampling scales, requires further theoretical work.
• Region selection relies on VAE decoding and classical edge detection, which is not directly applicable to non-image modalities (e.g., video, audio) without appropriate artifact detectors.

Future Directions:

• Extending NT-DM to arbitrary upsampling ratios and to architectures beyond DiT (e.g., U-Net-based Stable Diffusion, video DiTs).
• Replacing hand-engineered region selection with a learned "region importance" predictor for domains with less structured edge information.
• Joint optimization of stage step counts and region ratio using reinforcement learning for tailored latency–quality trade-offs (Jeong et al., 11 Jul 2025).

7. Relation to Attention-based and Latent Upsampling Methods

RALU is distinct from general attention-based upsampling (as in "Attention-based Image Upsampling" (Kundu et al., 2020)) in that it provides a multi-stage, spatially selective upsampling procedure specifically tailored for accelerating diffusion transformer inference. Whereas attention-based upsampling replaces deconvolutions with adaptive attention kernels to upsample spatial feature maps or latent variables, RALU orchestrates spatial resolution transitions in the latent domain with explicit region adaptivity and distribution matching. Both approaches exploit the flexibility of transformer-based architectures for spatial adaptation, yet RALU’s selective, multi-stage paradigm enables unique computational benefits in generative diffusion contexts.

---

Region-Adaptive Latent Upsampling represents a significant development in accelerating transformer-based generative models, demonstrating that attention to spatial adaptivity, region importance, and statistical correctness at resolution transitions enables both efficient and high-quality synthesis (Jeong et al., 11 Jul 2025).