Diverse Sampling in Diffusion Models with Marginal Preserving Particle Guidance

Abstract: We present EDDY (Exact-marginal Diversification via Divergence-free dYnamics), a guidance mechanism for diffusion and flow matching models that promotes diversity among samples generated while maintaining quality. EDDY exploits symmetries of the Fokker-Planck equation, using drift perturbations that change particle trajectories while preserving the evolving marginal distribution. We instantiate this principle through kernel-based anti-symmetric pairwise matrix fields, constructed from the repulsive directions. The resulting divergence-free dynamics promote diversity at the joint particle level while preserving each particle's marginal distribution without any additional training. As computing the guidance can be computationally expensive in cases such as text-to-image generation with perceptual embeddings, we propose practical approximations as an effective and efficient solution. Experiments on synthetic distributions and text-to-image generation show that EDDY improves diversity while maintaining strong distributional fidelity compared to common baselines.

• The paper introduces EDDY, a novel method that leverages divergence-free particle guidance to maintain exact marginal distributions while enhancing diversity.
• It uses Fokker–Planck symmetries and anti-symmetric matrix fields to design particle interactions that preserve fidelity during inference.
• Empirical results on synthetic and text-to-image benchmarks demonstrate that EDDY achieves a superior diversity-quality trade-off compared to existing methods.

Exact-Marginal Diversification in Diffusion Models via Divergence-Free Particle Guidance

Overview

"Diverse Sampling in Diffusion Models with Marginal Preserving Particle Guidance" (2605.06553) introduces EDDY, a theoretically principled and practically efficient mechanism for diverse sampling in score-based generative models. The central innovation is a particle interaction scheme that promotes diversity by leveraging symmetries of the Fokker–Planck equation to maintain marginal distributions, enabling training-free diversity enhancement at inference. The paper provides theoretical analysis, practical algorithmic formulations, and comprehensive empirical evaluation across synthetic and text-to-image generation benchmarks.

Motivation and Background

Sampling multiple outputs corresponding to the same conditioning (such as a prompt in text-to-image models) often yields highly similar samples, especially when classifier-free guidance or other fidelity-boosting mechanisms are applied. Recent literature has attempted to address this via particle interactions during inference (e.g., repulsive potentials, noise optimization), but these methods commonly distort the target marginal distribution, degrading sample fidelity or introducing artifacts.

EDDY is motivated by the observation that certain perturbations to the drift field—specifically, anti-symmetric matrix fields under the Stein operator—preserve the marginal distribution, as dictated by symmetries of the Fokker–Planck equation. This opens the possibility of designing interactions that achieve diversity without sacrificing quality, an important theoretical and practical advance over previous repulsion-based techniques.

Theoretical Contributions

EDDY's construction is grounded in the following:

• Fokker–Planck symmetries: For any anti-symmetric matrix field $A$, the guidance drift $\psi_t = \mathcal{A}(A)$ (the Stein operator applied row-wise) leaves the marginal evolution invariant.
• Per-particle marginal preservation: For a batch of $n$ interacting particles, the guidance can be formulated so that each particle's drift incorporates anti-symmetric pairwise contributions, guaranteeing that the marginal of each particle remains exactly the intended $p_t$.
• Kernel-based interactions: The pairwise interaction matrix uses kernel gradients and divergence-free matrix kernels to create repulsive fields. When instantiated with RBF kernels, all quantities admit closed-form expressions, facilitating efficient implementation.

This is illustrated in the transport fields induced by traditional Particle Guidance (PG) versus EDDY. PG induces purely repulsive fields that distort marginals; EDDY constructs divergence-free, anti-symmetric fields ensuring fidelity preservation.

Figure 1: PG (left) produces mean repulsive fields distorting the sampling distribution; EDDY (right) generates divergence-free, anti-symmetric transport fields that preserve the target marginal.

Algorithmic Formulations

EDDY is instantiated in two forms:

• EDDY-RBF: Uses closed-form kernel derivatives for RBF kernels in low-dimensional or analytically tractable settings.
• General EDDY (feature-space kernels): In perceptual embedding spaces (e.g., DINO or CLIP features for text-to-image), kernel derivatives are computationally prohibitive. EDDY employs finite-difference and Hutchinson trace estimation to approximate necessary quantities, maintaining efficient sampling while preserving the guiding principles.

The per-particle guidance is applied only in the early, high-noise portion of the sampling trajectory, resulting in significant computational savings with minimal impact on quality.

Empirical Evaluation

Synthetic Benchmarks

In 2D Gaussian mixtures, EDDY-RBF demonstrably increases mode coverage with guidance strength while maintaining statistical indistinguishability from independent sampling at the marginal level (as confirmed by Kolmogorov–Smirnov, Mann–Whitney, and Welch t-tests). This illustrates exact diversity enhancement with theoretical fidelity preservation.

Figure 2: Mode coverage increases with EDDY guidance strength $w_g$, confirming enhanced diversity at matched marginal fidelity.

Text-to-Image Generation

EDDY is benchmarked against PG and CADS on FLUX.1-dev and Stable Diffusion XL (SDXL) across 2048 MS-COCO prompts, measuring diversity (pairwise DINO similarity) and quality (CLIPScore, CMMD, Aesthetic, FID). EDDY consistently achieves Pareto-dominance over PG, delivering higher quality metrics at matched diversity levels. Unlike CADS, which perturbs conditioning and sacrifices prompt adherence, EDDY preserves both prompt fidelity and photorealism.

Figure 3: Qualitative comparison using the prompt "No one is in the room but there are chairs. high quality, 8k." EDDY produces diverse, photorealistic, and prompt-adherent samples.

Figure 4: Diversity–quality trade-off curves on MS-COCO; EDDY Pareto-dominates PG across all quality metrics at matched diversity.

EDDY exhibits occasional blurring (not artifacts) on FLUX.1-dev, reflecting an axis of diversity compatible with the distribution. On SDXL, image quality is robust and unaffected by blurring.

Limitations

EDDY incurs approximately 25% additional runtime relative to I.I.D. and PG, primarily due to kernel computations and feature-space approximations. Exact marginal preservation is weakened (but empirically robust) in feature-space settings due to finite-sample kernel approximations. For very high-dimensional or non-RBF kernels, this overhead may be significant.

Implications and Future Directions

Theoretical results establishing marginal-preserving particle interactions in score-based models lay groundwork for principled diverse generation. Practical implications include:

• Training-free diversity enhancement: No modifications to model weights or retraining required, facilitating rapid deployment in new settings.
• Robust fidelity across modalities: The approach is adaptable to arbitrary kernel choices and perceptual embeddings.
• Principled diversity-quality trade-off: Pareto-optimal behavior across major generative metrics.

Future directions may involve scaling EDDY to ultra-high-dimensional modalities, developing adaptive kernel selection strategies, integrating importance weighting for further bias correction, and extending the theoretical analysis to more general SDEs and weakly supervised generative frameworks.

Conclusion

EDDY is a mathematically elegant, empirically robust method for diverse sampling in diffusion and flow matching models, leveraging Fokker–Planck symmetries via anti-symmetric, divergence-free drift perturbations to guarantee marginal preservation. Comprehensive testing across synthetic and vision tasks demonstrates superior diversity-fidelity trade-offs relative to prior methods, with minimal artifacts and strong prompt adherence. This work advances both the theoretical foundation and practical utility of particle-based diversity in modern generative modeling.