Training-Free Generative Sampling via Moment-Matched Score Smoothing

Abstract: Diffusion models generate samples by denoising along the score of a perturbed target distribution. In practice, one trains a neural diffusion model, which is computationally expensive. Recent work suggests that score matching implicitly smooths the empirical score, and that this smoothing bias promotes generalization by capturing low-dimensional data geometry. We propose moment-matched score-smoothed overdamped Langevin dynamics (MM-SOLD), a training-free interacting particle sampler that enforces the target moments throughout the sampling trajectory. We prove that, in the large-particle limit, the empirical particle density converges to a deterministic limit whose one-particle stationary marginal is a Gibbs--Boltzmann density obtained by exponentially tilting a naive score-smoothed diffusion target. The mean and covariance of this distribution agree with the empirical moments of the training data. Experiments on 2D distributions and latent-space image generation show that MM-SOLD enables fast, robust, training-free sampling on CPUs, with sample fidelity and diversity competitive with neural diffusion baselines.

• The paper introduces MM-SOLD, a training-free generative sampler that constrains particle moments to match the dataset's empirical mean and covariance.
• It employs score smoothing with moment-matching to prevent memorization and barycentric collapse, thereby ensuring robust sample diversity and fidelity.
• Empirical results demonstrate that MM-SOLD outperforms prior training-free methods and rivals neural diffusion models in both low- and high-dimensional settings.

Training-Free Generative Sampling via Moment-Matched Score Smoothing

Overview and Motivation

The paper "Training-Free Generative Sampling via Moment-Matched Score Smoothing" (2605.14276) addresses the challenge of generating high-fidelity samples from empirical datasets using score-based diffusion models without neural network training. Standard diffusion models rely on costly score-learning via neural networks, while training-free approaches employing closed-form Gaussian mixture scores suffer from memorization or barycentric collapse, insufficiently reflecting the manifold geometry of data. The authors propose MM-SOLD (Moment-Matched Score-Smoothed Overdamped Langevin Dynamics), an interacting particle sampler that constrains empirical moments (mean, covariance) of generated particles to match those of the original dataset throughout the sampling trajectory. This construction yields a sample distribution closer to the underlying data geometry while preserving diversity.

Score Smoothing and Limitations of Closed-Form Diffusion Models

The empirical score of a finite dataset is computable exactly for Gaussian mixture models, but if directly used for sampling, it leads to concentration on training samples—i.e., memorization. Prior work suggests neural approximations implicitly smooth the empirical score, promoting generalization but introducing scale-sensitive bias: insufficient smoothing leaves memorization, excessive smoothing collapses the distribution to barycenters. Figure 1 visualizes this phenomenon for isotropic Gaussian smoothing on a 2D circle, showing how smoothing bandwidth $\sigma$ impacts sample distribution: naive smoothing collapses the geometry, while moment-matching restores it.

Figure 1: Naive (top) and moment-matched (bottom) densities under isotropic Gaussian smoothing with increasing $\sigma$; moment-matched correction maintains circular geometry.

MM-SOLD: Algorithmic Design and Large-Particle Limit

MM-SOLD operates as an interacting particle system evolving over a constrained moment-matching manifold. Each sampling iteration projects drift and stochastic noise into the tangent space of this manifold, maintaining agreement of particle mean and covariance with the training data. The sampling dynamics are performed in a whitened (mean- and covariance-centered) coordinate system, enabling efficient implementation. The theoretical large-particle stationary marginal of MM-SOLD is proven to be a Gibbs–Boltzmann density, obtained via exponential tilting of the naive score-smoothed target with linear and quadratic corrections for mean and covariance. These corrections are determined by population expectations, estimated empirically from the training data.

Comparative Experimental Results in 2D and High-Dimensional Latent Spaces

On canonical 2D distributions ("Checkerboard", "Two Spirals"), MM-SOLD yields sampling that is robust to both score-smoothing bandwidth $\sigma$ and Monte Carlo sample number $M$, consistently outperforming training-free baselines such as $\sigma$-CFDM in sample fidelity and diversity.

Figure 2: Comparison of MM-SOLD (blue) with $\sigma$-CFDM (red) on 2D distributions; moment matching preserves intrinsic data structure.

In the high-dimensional latent space of handwritten digits, as encoded by a Nuclear Norm-Regularized AutoEncoder (NRAE), MM-SOLD achieves robust generative sampling competitive with (or better than) neural DDPMs in classification accuracy (98.0% versus 97.9%), fidelity (KID), recall, and duplication rate, without any model training. Sample diversity and fidelity persist across a wide search of smoothing parameters, with MM-SOLD consistently maintaining low memorization rates.

Figure 3: Real digit-8 images (left), Latent DDPM samples, $\sigma$-CFDM samples, and MM-SOLD samples (right), decoded from NRAE latent space.

Extension to Large-Scale Image Generation

On CelebA-HQ at $256\times256$ resolution, MM-SOLD, despite running entirely on CPUs and requiring no training, closes the gap to neural latent diffusion models on FID, KID, and recall metrics, outperforming $\sigma$-CFDM in all cases and maintaining very low duplication rates.

Figure 4: Real CelebA-HQ images (left), Latent DDPM samples, $\sigma$-CFDM samples, and MM-SOLD samples (right), decoded from NRAE latent space.

MM-SOLD consistently shows robustness to hyperparameters, with heatmaps confirming performance superiority across a grid of $\sigma$0 and $\sigma$1 values (Figures 6 and 8). Ablation studies further demonstrate stability with respect to step size, particle number, and training-set size, exhibiting faster mixing and lower sample quality degradation.

Theoretical Implications and Future Directions

The moment-matched stationary law of MM-SOLD quantifies a KL projection onto distributions with empirical mean and covariance constraints, providing a transparent variational characterization. Practically, MM-SOLD enables fast, training-free generative sampling on CPUs, mitigating the sensitivity of score smoothing to bandwidth selection and avoiding barycentric collapse. Its explicit enforcement of global sample moments represents a significant advance over unconstrained samplers and offers a pathway for out-of-distribution robustness and adaptation in generative modeling.

The theoretical characterization lacks a finite-sample bound between the MM-SOLD limiting law and the true data distribution, leaving open the issue of statistical convergence and non-affine score bias. Relaxing the constraint on particle number ($\sigma$2) for full moment matching and integrating further inductive biases from deep generative architectures could enhance fidelity on complex, high-dimensional data manifolds.

Conclusion

MM-SOLD presents a principled, computationally efficient framework for training-free generative sampling by combining score smoothing and moment-matching constraints. Empirical results demonstrate strong performance in both low- and high-dimensional settings, outperforming prior non-trained diffusion baselines and approaching neural diffusion models in latent spaces. The method offers practical implications for generative modeling under data scarcity or computational budget constraints, and its theoretical basis suggests directions for convergence analysis and integration with architecture-driven inductive biases.