Feedback Guidance of Diffusion Models

Abstract: While Classifier-Free Guidance (CFG) has become standard for improving sample fidelity in conditional diffusion models, it can harm diversity and induce memorization by applying constant guidance regardless of whether a particular sample needs correction. We propose FeedBack Guidance (FBG), which uses a state-dependent coefficient to self-regulate guidance amounts based on need. Our approach is derived from first principles by assuming the learned conditional distribution is linearly corrupted by the unconditional distribution, contrasting with CFG's implicit multiplicative assumption. Our scheme relies on feedback of its own predictions about the conditional signal informativeness to adapt guidance dynamically during inference, challenging the view of guidance as a fixed hyperparameter. The approach is benchmarked on ImageNet512x512, where it significantly outperforms Classifier-Free Guidance and is competitive to Limited Interval Guidance (LIG) while benefitting from a strong mathematical framework. On Text-To-Image generation, we demonstrate that, as anticipated, our approach automatically applies higher guidance scales for complex prompts than for simpler ones and that it can be easily combined with existing guidance schemes such as CFG or LIG.

• The paper introduces Feedback Guidance, a dynamic state-dependent method that adjusts the guidance scale to enhance sample fidelity and diversity in conditional diffusion models.
• The methodology leverages feedback predictions and control theory to modulate guidance adaptively along the diffusion trajectory.
• Experimental results on ImageNet and text-to-image tasks show improved FID and FD_DinoV2 metrics, underscoring the method's balanced trade-off between fidelity and diversity.

Feedback Guidance of Diffusion Models

Introduction

The paper "Feedback Guidance of Diffusion Models" (2506.06085) addresses the challenge of enhancing sample fidelity in conditional diffusion models while maintaining diversity. Classifier-Free Guidance (CFG) has been the prevalent method for this task, but it often reduces diversity and induces memorization by applying a static guidance scale along the diffusion trajectory. The authors propose a novel method, Feedback Guidance (FBG), which dynamically adjusts the guidance scale based on the current state of the sample. Unlike CFG, which assumes a multiplicative corruption of the learned model, FBG assumes an additive corruption, allowing for a dynamic and more flexible guidance mechanism.

Methodology

FBG introduces a state-dependent guidance coefficient by making real-time assessments of how much guidance is needed throughout the diffusion process. The guidance scale is no longer a fixed hyperparameter but a dynamic function that reacts to the current prediction's alignment with the conditional model. The core idea is to use feedback predictions about the conditional signal's informativeness, integrating control theory to modulate guidance adaptively.

Figure 1: Example conditional diffusion trajectories demonstrating different guidance scales applied depending on the trajectory's proximity to the desired result.

FBG was derived from the principle that the learned conditional distribution is linearly corrupted by the unconditional distribution. The state-dependent coefficient regulates the guidance by adjusting as the sample approaches the decision windows or maximal sensitivity points in its trajectory. This method contrasts CFG's approach of constant guidance application, offering a refined control mechanism adaptable to the inherent variability of diffusion paths.

Results

The paper benchmarks FBG on datasets such as ImageNet512x512, displaying significant advancements over traditional CFG and Limited Interval Guidance (LIG) techniques. Specifically, for Text-To-Image generation, FBG demonstrates its capabilities by automatically allocating higher guidance scales for complex prompts and minimal scales for simpler ones, showcasing its adaptability and efficiency.

Figure 2: Guidance scale for different trajectories using the prompt: "A snail crawling on a green leaf with water droplets". Demonstrating lower guidance for accurate predictions and higher guidance for poorer initial predictions.

Quantitative results include notable improvements in metrics such as FID and FD$_{\text{DinoV2}$, emphasizing the method's balanced trade-off between fidelity and diversity. Furthermore, experimental results substantiate that FBG can be seamlessly integrated with existing guidance techniques, amplifying their effectiveness without additional computational overhead.

Implications

The introduction of FBG redefines the approach to guidance in diffusion models, emphasizing the need for flexibility and feedback-driven adjustments over rigid frameworks. The state- and time-dependent nature of FBG could inspire future research into more context-aware generative models that inherently adapt to the needs of the sample distribution.

Theoretically, this method challenges the conventional view of guidance as a static process, advocating for a system that dynamically assesses and responds to the evolving state of model predictions. Practically, it holds promise for more efficient and diverse generative models, potentially advancing applications such as content creation, style transfer, and beyond.

Conclusion

The "Feedback Guidance of Diffusion Models" (2506.06085) introduces a compelling alternative to static guidance mechanisms in diffusion models. By leveraging feedback-based control, the authors provide a robust framework that enhances model adaptability and efficacy. These contributions pave the way for future explorations into dynamic guidance systems, highlighting a shift towards more sophisticated approaches in generative modeling.