SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation

Abstract: Distribution Matching Distillation (DMD) is a widely used paradigm for accelerating inference in few-step video diffusion models. However, DMD-style video distillation faces two coupled challenges: the fake score must track a continuously evolving generator, making training costly when frequent updates are required, while reverse-KL-style matching can be mode-seeking and conservative for preserving strong motion dynamics. To address these issues, we propose \textbf{Score Gradient Matching Distillation (SGMD)}. SGMD adopts a fake-score perspective by directly optimizing the fake score toward the teacher, while using teacher stop-gradient Fisher as a stable distribution-matching objective. We provide a gradient analysis that motivates this objective choice under ideal tracking. Building on this, SGMD introduces a pair of dual potentials: negative-residual (NR) for outer-loop correction and residual-contraction (RC) for inner-loop tracking. Empirically, compared to DMD2, SGMD achieves an approximately $\sim 3\times$ training speedup and substantially improves motion dynamics for 4-step distilled models while preserving temporal consistency. A human study confirms that SGMD is preferred in motion quality and overall preference, while visual quality and text alignment remain comparable. Code is available at https://github.com/ModelTC/LightX2V.

• The paper introduces SGMD, a novel framework using a teacher stop-gradient Fisher objective to stabilize optimization and enhance motion dynamics.
• The paper presents a lightweight two-step update scheme that reduces training time by 3x while preserving temporal consistency and visual quality.
• The paper validates SGMD through empirical results and human studies, demonstrating superior dynamic motion and competitive static image quality compared to DMD baselines.

Authoritative Summary of "SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation" (2605.30116)

Introduction and Motivation

Modern large-scale video diffusion models yield compelling motion dynamics and visual quality but remain computationally expensive due to high parameter counts and multi-step sampling. Few-step distillation techniques aim to reduce inference steps with minimal architectural changes. Distribution Matching Distillation (DMD) approaches have dominated this regime yet face significant practical challenges: the fake-score tracker must closely follow the generator, incurring high training costs, and conventional reverse-KL objectives tend to yield mode-seeking behavior, suppressing dynamic content in motion-rich outputs.

SGMD: Methodological Contributions

SGMD introduces a fundamentally different framework for few-step video diffusion distillation:

• Teacher Stop-Gradient Fisher Objective: The distillation loss employs a stop-gradient Fisher divergence, avoiding unreliable teacher input gradients on OOD samples. This loss offers stable gradients and aligns directionally with reverse-KL under ideal tracking scenarios, but is less conservative, thus favoring dynamic motion.
• Fake-Score Perspective & Dual Potentials: SGMD reframes the distillation optimization, treating the fake score as the primary objective and the generator as a tracker. The mechanism is realized via dual potential functions:
  • Negative-Residual (NR): Correction applied in the generator update, pulling it toward fake-score predictions.
  • Residual-Contraction (RC): Applied to the fake score, contracting the residual and improving alignment with the generator.
• Lightweight Two-Step Update Scheme: Each iteration consists of one generator and one fake-score update, explicitly controlling tracking lag and obviating the need for multiple auxiliary updates or expensive second-order gradients.
• Explicit Tuning of Tracking Correction: The trade-off between distribution matching and tracking correction is controlled via a weight parameter $\lambda$, enabling empirical selection for the best dynamics-quality trade-off.

Numerical Results and Empirical Evaluation

SGMD was empirically validated using Wan2.1-T2V-14B as a teacher base model on a large-scale text-to-video task. Key findings include:

• ~3x Training Speedup: SGMD reduces fake-score updates from 5 (DMD2-style) to 1 per iteration, yielding a threefold reduction in training time while maintaining competitive output quality.
• Enhanced Motion Dynamics: SGMD substantially outperformed DMD2 and other baselines in optical flow (motion intensity) and dynamic degree, as measured via UniMatch flow and VBench dynamic metrics.
• Temporal Consistency and Quality: Visual quality and text-video semantic alignment remain comparable to strong DMD baselines, with best FVD among all distilled models.
• Human Preference Evaluation: In a pairwise human subject study, SGMD was strongly preferred in terms of motion quality and overall video preference, whereas static aspects (visual quality and text alignment) were largely ties.

Theoretical Analysis

SGMD provides a principled solution to the two-timescale problem inherent in DMD:

• The teacher stop-gradient Fisher objective stabilizes the optimization, avoiding catastrophic behavior from teacher gradient propagation on OOD states.
• Dual potentials decouple outer-loop direction correction from inner-loop contraction, ensuring stable score matching without excessive tracking lag.
• Gradient analysis reveals that Fisher-style (as opposed to reverse-KL) objectives yield less conservative, global matching signals, resulting in more dynamic visual outputs.
• The explicit dual potential design permits direct compensation for tracking lag, a limitation in prior SIM/SiD approaches where the implicit correction strength is fixed.

Comparative Analysis and Ablations

SGMD is contrasted against SIM, SiD, and DMD2:

• SIM and SiD partially address the tracking lag via implicit gradients and fixed/explicit reweighting. However, their generator update is not strictly unbiased with respect to Fisher divergence and offers no explicit trade-off parameter.
• SGMD's explicit dual potentials allow for finer control over tracking and dynamics, as validated through empirical parameter sweeps.
• Objective ablation studies confirm that the teacher stop-gradient Fisher divergence is primarily responsible for improved motion dynamics, while the dual potentials restore training stability and quality.

Practical Implications and Future Directions

SGMD advances the field by providing a framework that achieves efficient few-step video diffusion distillation, marrying strong motion dynamics and temporal consistency with stable, low-cost training.

• Deployment: The reduction in computational overhead favors practical and scalable video generation applications, including interactive content creation.
• Model Distillation Paradigms: The fake-score perspective and dual potential design are likely extensible to broader generative modeling tasks beyond video, potentially impacting image and audio diffusion distillation.
• Objective Design: SGMD’s explicit trade-off parameter and score-matching methodology open avenues for adaptive, task-specific distillation regimes, enabling tailored solutions for motion-sensitive versus detail-sensitive video domains.
• Theoretical Developments: The score-gradient matching paradigm may inspire new bilevel optimization and tracking frameworks, especially for generative models confronted with fast-evolving data distributions and OOD scenarios.

Conclusion

SGMD establishes a new class of practical, stable, and efficient distillation schemes for video diffusion models. Through the adoption of a fake-score-centric optimization strategy, teacher stop-gradient Fisher objectives, and tunable dual potentials, SGMD achieves substantial speedup and motion enhancement without compromising static visual or semantic quality. The methodological innovations and empirical outcomes collectively support SGMD as a principled direction for future few-step diffusion distillation research and applications.