On the Design Fundamentals of Diffusion Models: A Survey

Abstract: Diffusion models are learning pattern-learning systems to model and sample from data distributions with three functional components namely the forward process, the reverse process, and the sampling process. The components of diffusion models have gained significant attention with many design factors being considered in common practice. Existing reviews have primarily focused on higher-level solutions, covering less on the design fundamentals of components. This study seeks to address this gap by providing a comprehensive and coherent review of seminal designable factors within each functional component of diffusion models. This provides a finer-grained perspective of diffusion models, benefiting future studies in the analysis of individual components, the design factors for different purposes, and the implementation of diffusion models.

• The paper provides a comprehensive survey of the design fundamentals of diffusion models, analyzing forward, reverse, and sampling processes.
• It examines methodological choices such as noise scheduling and network architecture parameterizations that critically impact model performance.
• The survey highlights practical implications for optimized data generation and sets the stage for future research in generative modeling.

On the Design Fundamentals of Diffusion Models: A Survey

Introduction to Diffusion Models

Diffusion models stand out as a promising family within the generative model domain, capable of producing novel data by structurally adding and removing noise within a learning framework. The methodology operates through a systematic transformation involving noise manipulation in sequences, effectively learning data distributions for generations. Despite previous exhaustive analyses of various generative models, diffusion models require focused scrutiny to unravel their foundational design. This paper provides an evaluative review centered on the crucial elements, namely the forward process, reverse process, and sampling procedure, offering insights into implementation efficiency and potential improvements.

Figure 1: The overview of diffusion models. The forward process, reverse process, and sampling procedure constitute the core components responsible for noise manipulation, network training, and data generation respectively.

Preliminaries of Diffusion Models

Diffusion models utilize complex mechanisms to facilitate the perturbation and denoising of samples aligned along a timestep-indexed chain.

The Forward Process

The forward process initiates with a systematic noise induction that alters the data samples progressively across designated timesteps. The representation of perturbation follows a sequence of forward transitions:

$$p(x_{T}|x_0)=p(x_1|x_0)\cdots p(x_t|x_{t-1})\cdots p(x_T|x_{T-1}) = \prod_{t=1}^T p(x_t|x_{t-1})$$

Figure 2: The forward process perturbs the original distribution by adding noise through a sequence of transitions across multiple timesteps.

The Reverse Process

Conversely, the reverse process employs an optimal denoising network to systematically remove inserted noise by transforming perturbed samples back to their original states:

$$p_\theta(x_0)=p(x_T)p_\theta(x_{T-1}|x_T)\cdots p_\theta(x_0|x_1) = p(x_T)\prod_{t=1}^Tp_\theta(x_{t-1}|x_t)$$

Figure 3: The reverse process trains a neural network to iteratively remove noise added by the forward process.

The Sampling Procedure

Following training, the sampling procedure leverages the optimized network to generate novel data, concluding a denoising path similar to the reverse process:

$$p_{\theta^*}(x_0) = p(x_T)p_{\theta^*}(x_{T-1}|x_T)\cdots p_{\theta^*}(x_0|x_1) = p(x_T)\prod_{t=1}^Tp_{\theta^*}(x_{t-1}|x_t)$$

Figure 4: The sampling procedure utilizes the trained denoising network and mirrors the transitions of the reverse process.

Design Aspects of Diffusion Models

Delving deeper, this survey highlights key areas of focus within the components:

Forward Process Design

The design of the forward process requires decisions regarding noise schedules and the types of noise to be applied.

• Noise Schedule: Determines the specific noise levels per timestep, balancing between exploration and exploitation to fit distributional intricacy without causing inconvergence.
• Noise Type: Though Gaussian noise is prevalent due to simplicity, alternatives such as Gamma distributions enhance modeling capabilities through increased degrees of freedom.

  Figure 5: Visualization of learning priority shifts within the reverse process, shown through varying colors.

Reverse Process Design

The reverse process targets the refinements necessary for enhancing denoising directions:

• Network Architectures: U-Net and Transformer-based models serve as foundations for denoising networks, exhibiting adaptability across different tasks due to inherent capacity to retain data dimensions.
• Parameterizations of Reverse Mean: Decisive choices in network output handling significantly affect generative accuracy, stable learning, and efficiency.

  Figure 6: Displaying trajectory modeling using score-based parameterization, where scores dictate denoising directions during transitions.

Sampling Procedure Design

Efforts within the sampling phase focus on establishing effective generation through adaptation strategies:

• Guidance Mechanisms: Guidance strength stems from classifier integration, providing distinct control dynamics comparative to vanilla guidance setups.
• Acceleration Techniques: Methods such as truncation and knowledge distillation aim to condense sampling steps, enhancing computational efficiency.

  Figure 7: Sampling via truncation skips non-critical segments for accelerated processing, with grey dashed parts indicating discarded portions.

Conclusion

This survey provides an in-depth analysis of diffusion models, exploring nuanced design fundamentals, insightful methodologies, and the latest advancements. With clear domains identified, ongoing explorations toward optimizing diffusion model components hold promise for addressing challenges and enhancing their real-world applicability across an array of fields.

This study propels future research on diffusion models, encouraging innovations in architectural paradigms, compositional frameworks, and systematic approaches, without disregarding regulatory concerns on their societal effects. As diffusion models continue to shape the generative model landscape, ongoing evaluative studies remain crucial.