On the Memorization of Consistency Distillation for Diffusion Models

Abstract: Diffusion models are central to modern generative modeling, and understanding how they balance memorization and generalization is critical for reliable deployment. Recent work has shown that memorization in diffusion models is shaped by training dynamics, with generalization and memorization emerging at different stages of training. However, deployed diffusion models are often further distilled, introducing an additional training phase whose impact on memorization is not well understood. In this work, we analyze how distillation reshapes memorization behavior in diffusion models, taking consistency distillation as a representative framework. Empirically, we show that when applied to a teacher model that has memorized data, consistency distillation significantly reduces transferred memorization in the student while preserving, and sometimes improving, sample quality. To explain this behavior, we provide a theoretical analysis using a random feature neural network model [Bonnaire et al., 2025], showing that consistency distillation suppresses unstable feature directions associated with memorization while preserving stable, generalizable modes. Our findings suggest that distillation can serve not only as an acceleration tool, but also as a mechanism for improving the memorization-generalization trade-off.

• The paper demonstrates that consistency distillation actively reduces memorization, even when distilled from overfit teachers, thereby preventing instance-level data leakage.
• It employs a spectral filtering mechanism via RFNN analysis, selectively transferring generalization-associated modes while minimizing sample-specific updates.
• Empirical evaluations on datasets like CIFAR-10 and ImageNet show that lowered memorization can maintain or sometimes enhance generative quality under optimal architectural settings.

Consistency Distillation and Memorization in Diffusion Models

Introduction

This work provides a rigorous empirical and theoretical analysis of memorization dynamics in distilled diffusion models, focusing specifically on consistency distillation (CD). Due to the increasing adoption of diffusion models in generative modeling and the practice of further distilling these models for efficient inference, understanding how distillation reshapes memorization and generalization phenomena is essential. The authors demonstrate that CD actively and robustly reduces memorization in the student model—even when distilled from a strongly overfit (memorizing) teacher—while preserving or occasionally improving generative quality. Theoretical evidence based on Random Feature Neural Network (RFNN) analysis elucidates a structured spectral filtering mechanism that underlies this memorization suppression.

Empirical Findings: Memorization Suppression and Sample Quality

The primary empirical contribution is an extensive evaluation of memorization and generalization in CD across various datasets (CIFAR-10, ImageNet, and Stable Diffusion v1.5). Distillation reliably yields student models with drastically lower instance-level memorization than their teacher models, even when quality metrics such as FID or CLIP score are held constant.

Figure 1: Consistency distillation reliably reduces memorization ratios throughout training while the evolution of FID depends on the initial teacher's memorization regime.

The left panel of Figure 1 demonstrates that, during CD on CIFAR-10, student memorization ratios remain close to zero, rarely increasing over continued training. The right panel shows that when the teacher has moderate memorization, FID of the student can even surpass that of the teacher—indicating that the suppression of memorization does not necessitate a trade-off with generative utility. In severe overfitting scenarios, FID improvement is not observed, indicating a regime where excessive teacher memorization degrades the transfer of sample quality.

Further experiments under localized memorization regimes (e.g., overexposure only for select samples or classes) confirm that CD selectively suppresses memorization in overexposed regions while preserving the global generalization capacity of the student.

Mechanistic Explanation: Selective Spectral Filtering in Consistency Distillation

The theoretical core of the study comprises an analysis of consistency distillation using RFNNs. The student’s update dynamics are governed by a curvature operator, derived from a second-moment matrix of teacher-induced perturbations in feature space. The key result is that CD induces highly structured, non-isotropic updates: generalization-associated directions (high-curvature modes) are selectively retained, while memorization-associated (low-curvature, sample-specific) directions receive minimal transfer.

Figure 2: The spectral density of the teacher curvature matrix $\mathbf{U}$ exhibits separation between memorization-associated (low eigenvalues) and generalization-associated (high eigenvalues) modes.

Modes corresponding to memorization exhibit near-isotropic, low-visibility structure—manifested as low eigenvalues in the curvature spectrum (Figure 2). CD’s curvature operator, through its non-isotropic term, acts primarily on generalization modes, allocating almost all positive update energy there. This results from spectral filtering: consistent with empirical observations, nearly zero memorization-associated signal is transferred or reinforced in the student, while general representational structure persists.

Figure 3: Mode-wise visibility $a_i$ shows that generalization-associated modes have high alignment with the feature span, while memorization modes have negligible visibility and are minimally updated.

The theoretical analysis is reinforced by a rigorous per-mode decomposition. The visibility $a_i$ and the signed response $\alpha_i$ are negligible for memorization modes, and the allocation of positive update is almost exclusively within the high-curvature, generalization-associated subspace.

Architectural Robustness and Optimization Stability

An important practical dimension is architectural mismatch between teacher and student, or the student’s initialization. Results indicate that with random initialization or mismatched architectures, student memorization is suppressed even further, often to undetectable levels, but at the cost of optimization stability in sample quality (see Figure 4). When the student is initialized with compatible parameters from the teacher and architectures are matched, a small portion of memorization can be retained, but optimization stability and generative quality are superior. Thus, architectural compatibility mediates a trade-off between suppression of memorization and optimization tractability.

Figure 4: While student memorization remains near zero across configurations, lack of architectural match or fine-tuning significantly degrades FID at late stages of training.

Spectral Geometry and Regularization Implications

The theoretical apparatus unveils that the curvature spectrum’s separation is critical for effective spectral filtering: regimes with clear distinction between memorization and generalization directions enable CD to allocate positive update selectively and robustly. As capacity increases or data becomes more limited (increasing the $\psi_p/\psi_n$ ratio), this spectral separation diminishes, explaining cases where distillation may less effectively transfer generalizable structure or may be forced to attenuate more of the generative signal.

Additionally, the introduction of ridge regularization in the inversion of teacher curvature is essential for controlling “memorization leakage” into generalization modes during student updates (see detailed Appendix analysis).

Practical and Theoretical Implications

The findings have significant implications for the development and deployment of distilled diffusion models:

• Practical Safety: CD provides an effective procedural “guardrail” against undesirable memorization (e.g., instance-level data copying) in publicly deployed or privacy-sensitive generative systems.
• Preservation of Utility: The fact that sample quality is often preserved—or even improved—in the student at matched or lower memorization levels is notable for applications requiring both privacy and high fidelity.
• Design of Future Distillation Algorithms: The analysis suggests that distillation methods which induce strong non-isotropic filtering and avoid trajectory-based overfitting (as seen in progressive distillation) are preferable for safety-critical deployment.
• Generalization Theory: The work sharpens the theoretical understanding of how implicit regularization and update geometry interact in overparameterized generative models.

The spectral perspective on memorization filtering may generalize to other generative frameworks and inform the design of learning objectives or model pruning techniques which aim to minimize unintended memorization.

Conclusion

This work delivers both empirical and theoretical validation that consistency distillation actively suppresses memorization in diffusion models, even when the teacher exhibits strong overfitting, without compromising generative quality in most regimes. The mechanistic analysis via RFNNs reveals that CD achieves this by transferring only dynamically stable, generalizable modes and suppressing sample-specific directions. The framework and analysis established here serve as a foundation for both privacy-preserving generative modeling and for future theoretical investigations into implicit regularization dynamics in overparameterized neural networks.

Citation: "On the Memorization of Consistency Distillation for Diffusion Models" (2604.23552)