Denoising, Fast and Slow: Difficulty-Aware Adaptive Sampling for Image Generation

Abstract: Diffusion- and flow-based models usually allocate compute uniformly across space, updating all patches with the same timestep and number of function evaluations. While convenient, this ignores the heterogeneity of natural images: some regions are easy to denoise, whereas others benefit from more refinement or additional context. Motivated by this, we explore patch-level noise scales for image synthesis. We find that naively varying timesteps across image tokens performs poorly, as it exposes the model to overly informative training states that do not occur at inference. We therefore introduce a timestep sampler that explicitly controls the maximum patch-level information available during training, and show that moving from global to patch-level timesteps already improves image generation over standard baselines. By further augmenting the model with a lightweight per-patch difficulty head, we enable adaptive samplers that allocate compute dynamically where it is most needed. Combined with noise levels varying over both space and diffusion time, this yields Patch Forcing (PF), a framework that advances easier regions earlier so they can provide context for harder ones. PF achieves superior results on class-conditional ImageNet, remains orthogonal to representation alignment and guidance methods, and scales to text-to-image synthesis. Our results suggest that patch-level denoising schedules provide a promising foundation for adaptive image generation.

• The paper introduces Patch Forcing, a method that assigns individual denoising schedules to image patches, resulting in improved FID scores and optimized compute allocation.
• The methodology integrates an uncertainty prediction head with a Logit-Normal Truncated Gaussian sampler to dynamically control per-patch noise levels and reduce context leakage.
• Adaptive inference strategies like dual-loop and look-ahead enable faster convergence and enhanced compositional fidelity in resolving complex image regions.

Difficulty-Aware Adaptive Sampling for Image Generation: Patch Forcing

Introduction

Canonical diffusion- and flow-based image generation models rely on spatially uniform denoising schedules, applying identical noise levels and computation budgets to all regions within an image at each denoising iteration. This paradigm inherently neglects the heterogeneity of natural images, where the "denoising difficulty" varies significantly across spatial regions. Uniform treatment is suboptimal: simple or unambiguous regions consume unnecessary compute, while complex regions are under-resourced, lacking both direct steps and contextual information. The paper "Denoising, Fast and Slow: Difficulty-Aware Adaptive Sampling for Image Generation" (2604.19141) introduces Patch Forcing (PF), a flexible framework for spatially-adaptive denoising that leverages per-patch noise trajectories and explicit uncertainty estimation to dynamically allocate computational resources.

Methodology

Patch Forcing generalizes the conventional single-schedule denoising setting to a patchwise adaptive regime. At each training or sampling iteration, the model operates on patches indexed over the spatial domain and assigns each patch its own noise level (timestep), which induces heterogeneous denoising progress within an image. Crucially, PF augments the network with an uncertainty head that predicts the per-patch denoising difficulty (Figure 1), serving as the control signal for adaptive scheduling.

Figure 1: The model outputs a patch-level uncertainty map, distinguishing confident (easy) from uncertain (hard) regions, which is used for dynamic scheduling during inference.

The sampling and training schedule is informed by three main findings:

1. Context from advanced patches helps: Progressing easy/confident regions earlier in the denoising trajectory allows them to provide self-generated, lower-noise context to more difficult patches.
2. Uncertainty prediction aligns with actual generation difficulty: The learned uncertainty head reliably ranks patches according to local reconstruction loss (Figure 2).
3. Providing more context reduces remaining uncertainty: As inpainted or lower-noise values are supplied for subsets of patches, the model's uncertainty sharply decreases in ambiguous/hard areas (Figures 7, 9, 12).

To mitigate the train-test mismatch that arises from naive patch-level timestep sampling (which can result in over-informative patches at train time and neural context leakage), Patch Forcing introduces the Logit-Normal Truncated Gaussian (LTG) timestep sampler. The LTG sampler ensures that the maximum patch-level noise at training matches inference and diffuses patchwise noise scales in a controlled fashion (Figure 3).

Figure 3: Distribution of sampled per-sample $t_{\max}$—the LTG scheduler places tighter control on maximum noise, reducing context leakage.

Adaptive and Difficulty-Aware Inference

At inference, Patch Forcing enables a spectrum of dynamic schedulers parameterized by the predicted uncertainty:

• Parallel: All patches progress synchronously (resembling the standard regime);
• Dual-loop: Confident patches take larger steps toward the clean image, advancing more quickly, while uncertain patches receive more iterative refinement but are updated conditioned on the advanced context from the easy ones;
• Look-ahead: The context for a patch is updated by advancing confident regions to a later timestep, and then the model denoises uncertain regions using the contextually-enhanced input (Figure 4).

These strategies are visualized and compared, illustrating that structured, uncertainty-informed ordering outperforms naive or random patch selection.

Figure 4: Different patchwise sampling strategies. Dual-loop and look-ahead schemes provide more efficient and effective access to generated context.

Figure 5: Patch-wise timestep allocation visualization, demonstrating differentiation of denoising progress per patch with controlled $t_\text{max}$.

The effect of context provision and uncertainty discounting is validated quantitatively: increasing the context resolution provided by confident patches yields a monotonic reduction in loss and uncertainty for the remaining difficult regions (Figures 7, 8, 9, 12).

Figure 6: Validation loss measured on difficult regions, with and without additional context; context provision consistently reduces loss.

Figure 2: Empirical correlation between the predicted uncertainty and the validation loss is high; uncertainty is a reliable proxy for local difficulty.

Figure 7: With context progressively introduced from advanced patches, residual uncertainty in hard regions is reduced.

Numerical Evaluation and Results

Extensive experiments on class-conditional ImageNet and text-to-image (T2I) synthesis tasks illustrate the efficacy of the approach. The Patch Forcing Transformer (PFT) consistently improves FID over SoTA baselines and remains orthogonal and complementary to representation alignment and classifier-free guidance methods. For instance, at comparable network scale and computational budgets, PFT-B/2 achieves FID improvements over the SiT baseline, with further gains attributable to the dual-loop and look-ahead samplers (Figure 8).

Figure 8: PFT-B/2 shows improved FID across a range of function evaluations (NFE); uncertainty-aware samplers (dual-loop, look-ahead) exhibit further gains.

The framework is particularly notable for its minimal architectural overhead, requiring only an additional uncertainty prediction head, and its ability to scale to T2I—where quantitative and qualitative benchmarks confirm competitive or superior compositionality, spatial reasoning, and text rendering capability (Figures 13, 14, 25).

Figure 9: Representative text-to-image samples (512 px), showcasing object fidelity and composition.

Figure 10: Patch Forcing markedly enhances text rendering compared to vanilla Flow Matching.

Practical and Theoretical Implications

Patch Forcing presents a new class of denoising samplers that, by training under heterogeneous, patch-indexed noise schedules, natively produce and exploit intermediate, partially-denoised context for ambiguous or complex regions. The methodology provides several avenues for practical application and future research:

• Efficiency: By adaptively focusing computational effort on uncertain/difficult patches, PF could enable faster or more compute-frugal generation, particularly useful for scenarios with constrained compute budgets.
• Improved Semantic Fidelity: The ability to resolve ambiguous content by leveraging internally generated context is especially beneficial for structured, compositional, or text-critical image domains (e.g., natural/scene text rendering, multi-object compositionality).
• Generalization: The direct use of an uncertainty head and adaptive scheduling is amenable not only to image but also multimodal or sequential domains (e.g., video, text, 3D synthesis), provided spatial or temporal heterogeneity in generative difficulty.
• Compatibility: PF can be combined with any guidance signal, including strong representation alignment methods and classifier-free guidance, with additive benefits.
• Train-Test Consistency: The LTG schedule and insights into context leakage during training offer best practices for designing future adaptive and heterogenous generative pipelines across modalities.
• Model Scaling and Distillation: Extending PF to few-step or distilled models is expected to amplify its advantages, given that larger time increments heighten the impact of intermediate, context-providing denoising.

Conclusion

Patch Forcing demonstrates that spatial adaptivity in denoising schedules, driven by model-internal, learned estimates of local patch difficulty, delivers consistent improvements in generative modeling performance. By leveraging heterogeneous noise levels and difficulty-aware dynamic inference schedules, PF provides an actionable foundation for denoising models that optimally distribute computational resources and exploit intermediate context during synthesis. These results have direct implications for advancing the efficiency, fidelity, and flexibility of diffusion-based generative models (2604.19141).