AsyncPatch Diffusion: spatially-flexible image generation

Abstract: Standard diffusion models corrupt an entire sample with a single shared noise level, forcing all spatial regions to follow the same denoising trajectory. We introduce AsyncPatch Diffusion, a joint-diffusion framework that assigns distinct noise levels to different input dimensions, such as image pixels, or latent tokens. We show how this asynchronous corruption defines a valid generative process while supporting a richer family of spatially heterogeneous denoising trajectories, and prove the first valid ELBO for this process. We show that a single pretrained model can perform spatially adaptive generation, where different regions are denoised on different schedules. A key challenge is training: naive independent noise-level sampling overemphasizes highly heterogeneous configurations and underrepresents homogeneous noise levels, that are crucial during sampling. We address this with a controlled noise-level sampler that regulates both the average corruption level and its spatial variability. AsyncPatch achieves generation quality comparable to conventional diffusion on ImageNet 256 and LSUN, while being natively suited for inpainting without task-specific fine-tuning. We further introduce input guidance, which uses clean or partially corrupted regions to guide the generation of unknown regions, improving local consistency and texture matching. Finally, we demonstrate adaptive generation strategies including uncertainty-guided acceleration and autoregressive sampling.

• The paper introduces AsyncPatch Diffusion, which decouples per-pixel noise schedules to enable spatially flexible image generation and precise inpainting.
• It rigorously derives the first evidence lower bound (ELBO) for a joint diffusion process with independent token timesteps, ensuring valid generative modeling.
• The method leverages controlled timestep sampling and context-aware score modulation to achieve competitive FID and LPIPS metrics against traditional diffusion techniques.

AsyncPatch Diffusion: A Unified Framework for Spatially-Flexible Image Generation

Introduction and Motivation

AsyncPatch Diffusion presents a rigorous generalization of the traditional denoising diffusion model paradigm, extending the underlying forward and reverse processes to support spatially heterogeneous noise schedules at the granularity of image pixels or latent tokens. Standard diffusion models synchronize noise levels across all spatial dimensions, constraining all regions of an image to follow identical denoising trajectories. By contrast, AsyncPatch assigns each dimension its own independent time variable, thereby decoupling corruption and denoising schedules throughout the input. This formulation enables spatially asynchronous and adaptive generation, wherein regions of an image may be denoised according to arbitrary, potentially dynamic, schedules.

A key theoretical advance of this work is the demonstration that this asynchronous joint diffusion process still yields a valid generative model, and the derivation of the first principled evidence lower bound (ELBO) for such models. This is achieved via analysis of the generalized denoising score matching objective under per-token noising, a result of substantial foundational significance for future diffusion-based research.

Formulation: Joint Diffusion with Spatially-Decoupled Timesteps

AsyncPatch operates by dividing the data vector (e.g., pixels, latent tokens) into $N$ components, assigning to each an independent time variable $t_i \in [0,1]$. The forward process maps the clean data to noise via

$$q(\mathbf{z}_\mathbf{t} | \mathbf{x}) = \prod_{i=1}^N q(\mathbf{z}_{i,t_i} | \mathbf{x}_i, t_i) ,$$

where each $q(\mathbf{z}_{i,t_i} | \mathbf{x}_i, t_i)$ is a standard Gaussian diffusion process with SNR-scheduled mean and variance. The reverse (generative) process is defined in terms of a neural network that predicts the joint score, i.e., gradients of the log-density with respect to each noised variable, conditioned on the full, asynchronously-corrupted input.

Critically, the paper formalizes the training objective as a sum of tokenwise Fisher divergences, and proves (via a construction rooted in monotonic random walks in $(0, 1)^N$) that the minimizer of this loss yields the correct data distribution marginal at the end of generation. Furthermore, a nontrivial ELBO is derived—explicitly weighted by a density arising from the distribution of asynchronous noise schedules—which serves as a proper variational lower bound to the log-likelihood.

Training Regimes: Controlled Timestep Sampling Strategies

A primary empirical challenge in spatially-asynchronous models is the design of effective training distributions over timestep fields $\mathbf{t}$. Naively sampling each $t_i$ independently (i.i.d. uniform) overemphasizes highly heterogeneous input states and rarely produces homogeneous regions, creating a mismatch with typical inference-time schedules (e.g., classical generation or structured inpainting).

To address this train-test divergence, the authors devise a controlled sampler, "AsyncPatch sampling," which enforces a target mean and spread for the spatial distribution of timesteps, allowing explicit regulation of global corruption levels and local heterogeneity. Comparison is made with two other regimes:

• Perlin sampling: Uses smooth, structured masks to mimic natural inpainting settings, sampling binary partitions via Perlin noise.
• Patchwise sampling: Assigns block-constant timesteps to random patches, but with less global control.

The effect of these strategies on the sampled timestep distributions is visualized (Figure 1).

Figure 1: Sampled training timesteps and imagewise mean-timestep distributions for Perlin, patchwise, and AsyncPatch sampling, showing the effectiveness of AsyncPatch in balancing global and local diversity.

The AsyncPatch strategy is shown (in both qualitative and quantitative metrics) to maintain image synthesis performance on unconditional generation while simultaneously preserving strong inpainting capability without specialized finetuning.

Inference: Adaptive, Conditional, and Autoregressive Sampling Paths

AsyncPatch supports a broad class of reverse-time scheduling policies $\boldsymbol{\tau}(u)$, enabling a single trained model to execute:

• Standard generation: All timesteps evolve synchronously (baseline diffusion).
• Zero-shot inpainting/outpainting: Arbitrary regions are fixed clean (no noise) during sampling, while others undergo denoising, precisely implementing conditional generation aligned with user-specified masks.
• Autoregressive raster/patchwise generation: Image is recursively revealed in spatial groups according to customized groupwise timestep policies.

This flexibility—the ability to define $\tau_i(u)$ per token as a function of an abstract global progress variable $u$—naturally unifies tasks typically requiring bespoke network architectures or retraining.

Figure 2: Comparison of several possible pixelwise time schedules for synchronous, patchwise, and fully asynchronous generation; right, example forward paths under variable schedules for the same image.

Figure 3: Schematic autoregressive (patchwise order) sampling, showing progressive unmasking and generation conditioned on earlier context.

Empirical results demonstrate that even under highly asynchronous, patchwise, or uncertainty-driven adaptive step schedules, AsyncPatch can yield high-quality samples without the artifacts often observed in heuristic approximate inpainting pipelines.

Input Guidance: Context-Aware Score Modification and Texture Synthesis

An additional novel contribution is "input guidance," a mechanism that leverages clean or partially-corrupted image regions to condition score estimates for denoising the unknown areas. By constructing the predicted score at a clean timestep as a difference from the corresponding score at a more noisy input—weighted via a guidance coefficient—AsyncPatch can explicitly bias the generative trajectory toward consistency with the provided context.

This mechanism enhances local structural coherence in inpainting and enables the model to transfer and extend unique patterns from the known region into the generated area, outperforming both classical patch-based texture synthesis and baseline diffusion methods (Figure 4, Figure 5).

Figure 4: Comparison of inpainting with varying input guidance strength: zero guidance (top) produces plausible but less consistent textures, while increased guidance (bottom) ensures local detail matches the visible region.

Figure 5: Latent-space ImageNet and LSUN samples showing improved completion under wide inpainting masks with AsyncPatch.

Quantitative Evaluation

AsyncPatch achieves FID scores closely matching, and sometimes exceeding, traditional full-image diffusion on ImageNet-256 and LSUN, both in unconditional sampling and zero-shot inpainting (see main text). Notably:

• AsyncPatch FID (ImageNet-256): 8.06 vs LDM baseline 8.24.
• AsyncPatch outperforms previous methods (e.g., RePaint, RAD, Score-SDE) on inpainting FID/LPIPS for both in-distribution and challenging mask shapes, despite not being trained with explicit inpainting supervision.

Theoretical and Practical Implications

The theoretical justification for tokenwise asynchronous time schedules—together with an explicit ELBO—establishes that spatially adaptive denoising is a mathematically-grounded extension, not merely an empirical heuristic. This unlocks the design of generative models with spatially varying computational budgets, locally adaptive acceleration, or task-driven regional conditioning, without sacrificing the generative fidelity of state-of-the-art architectures.

Practically, AsyncPatch enables unified models that can serve as drop-in solutions for unconditional generation, inpainting, and controlled outpainting, eliminating the need for separate finetuned models or compositional compositing pipelines. The native support for patchwise and arbitrary scheduling opens up new possibilities for interactive and real-time synthesis pipelines, as well as enabling research into spatial computation allocation in both convolutional and transformer-based architectures.

Future Directions

Opportunities for further research include:

• Extending AsyncPatch principles to text-conditioned latent diffusion architectures.
• Investigating integration with large-scale transformer backbones and models operating at ultra-high resolutions.
• Exploiting the decoupled time formalism for dynamic, regionally-adaptive compute allocation and stochastic early stopping.
• Studying the benefits for multimodal joint generation and iterative editing pipelines, including integration with plug-and-play guidance or region-specific constraints.

Conclusion

AsyncPatch Diffusion generalizes the diffusion model framework to arbitrary spatial noise schedules, yielding a single model capable of spatially adaptive, context-aware, and zero-shot conditional image generation. The formulation is both theoretically justified (by a novel ELBO) and practically validated through strong results on unconditional, inpainting, and texture synthesis tasks, setting a new standard for unified generative modeling under flexible and dynamic control of denoising processes.