Teacher Stop-Gradient Fisher Objective

• The paper demonstrates that the teacher stop-gradient Fisher objective improves training stability and preserves high-quality motion in video diffusion models.
• It employs a pointwise Fisher divergence to directly match student and teacher score fields, avoiding unstable gradients from reverse-KL.
• Empirical results on VBench-T2V show significant gains in motion dynamics and inference speed, highlighting its practical advantages.

The teacher stop-gradient Fisher objective is a score-based distribution-matching criterion used in Score Gradient Matching Distillation (SGMD) for accelerating inference and refining motion dynamics in few-step video diffusion models. It addresses key stability and expressivity issues that arise in standard Distribution Matching Distillation (DMD) paradigms by leveraging a pointwise Fisher divergence with a stop-gradient on the teacher, offering improved training stability and preserving high-quality motion in aggressively distilled regimes (Wu et al., 28 May 2026).

1. Precise Formulation

Let $G_\theta$ denote the student generator parametrized by $\theta$, and $S_\text{fake}(x_t, t; \phi)$ the “fake-score” network with parameters $\phi$. The teacher score $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$ is always treated with stop-gradient. The forward noising process is: $$x_t = \alpha_t x_0 + \sigma_t \epsilon, \quad \epsilon \sim \mathcal{N}(0, I),$$ where $\alpha_t$ and $\sigma_t$ are schedule values.

The teacher stop-gradient Fisher objective is: $$L_\text{Fisher}(\theta, \phi) = \mathbb{E}_{x_0 \sim q_\theta, \epsilon, t}\Bigl[\, c(t)\, \| S_\text{fake}(x_t, t; \phi) - S_\text{real}(\mathrm{sg}[x_0], t) \|^2\,\Bigr],$$ where

• $\mathrm{sg}[\cdot]$ is the stop-gradient operator,
• $\theta$0
• $\theta$1 is the distribution defined by $\theta$2,
• $\theta$3 is the frozen teacher score.

Equivalently, with $\theta$4,

$\theta$5

2. Theoretical Rationale and Distinction from Reverse-KL

Fisher Divergence vs. Reverse-KL

Reverse-KL (minimizing $\theta$6) is highly mode-seeking, tending to avoid allocating mass to low-density regions of the target $\theta$7. In video diffusion distillation, this yields conservative student models with muted motion, suppressing crucial temporal dynamics. The Fisher divergence

$\theta$8

matches the score fields directly, providing a smoother, more global guidance that empirically encourages stronger motion dynamics and better coverage, particularly in few-step regimes.

Necessity of Teacher Stop-Gradient

Naively including backpropagation through teacher score inputs $\theta$9 during distillation exposes the optimization to unreliable or even divergent gradients, as such $S_\text{fake}(x_t, t; \phi)$0 often lies outside the original teacher domain. Applying the stop-gradient operator to $S_\text{fake}(x_t, t; \phi)$1 eliminates this unstable path, yielding a valid (though one-sided) Fisher-matching objective with substantially improved optimization stability.

Consistency Under Ideal Tracking

Assuming idealized tracking conditions—$S_\text{fake}(x_t, t; \phi)$2 and $S_\text{fake}(x_t, t; \phi)$3—the generator update induced by minimizing $S_\text{fake}(x_t, t; \phi)$4 exactly aligns with the gradient step prescribed by reverse-KL, ensuring that the overall matching direction remains correct. This guarantees that Fisher (with stop-grad) retains the desirable theoretical properties of reverse-KL while being empirically superior in challenging few-step settings.

3. Gradient Derivation and Properties Under Ideal Tracking

Consider the process $S_\text{fake}(x_t, t; \phi)$5. For perfect tracking, the Fisher objective reduces to: $S_\text{fake}(x_t, t; \phi)$6 A variation $S_\text{fake}(x_t, t; \phi)$7 in $S_\text{fake}(x_t, t; \phi)$8 induces a change $S_\text{fake}(x_t, t; \phi)$9. By information geometry, the update direction for $\phi$0 aligns with the reverse-KL gradient: $\phi$1 mirrored by the Fisher objective gradient. This ensures directional consistency and theoretical soundness, as shown by Proposition 3.1 in SGMD (Wu et al., 28 May 2026).

4. Practical Implementation in Score Gradient Matching Distillation

A distilled iteration, implemented in PyTorch-style pseudocode, consists of: $$x_t = \alpha_t x_0 + \sigma_t \epsilon, \quad \epsilon \sim \mathcal{N}(0, I),$$2 The critical implementation requirement is that gradients never flow into the teacher score components due to the torch.no_grad() (stop-gradient).

5. Integration with Dual Potentials in SGMD

SGMD frames the distillation as a bilevel optimization, decoupling generator adaptation from fake-score learning. The dual potentials—Negative-Residual (NR) and Residual-Contraction (RC)—control the coupling between generator and fake-score network:

• Negative-Residual: $\phi$2, corrects $\phi$3 to match $\phi$4.
• Residual-Contraction: $\phi$5, corrects $\phi$6 to match $\phi$7.

Gradients in the $\phi$8-space induced by these terms are exactly opposite: $\phi$9 The tracking residual $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$0 closes the loop, ensuring synchronization between the generator and score field. The overall SGMD iteration (with $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$1) updates $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$2 with $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$3 and $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$4 with $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$5, efficiently restoring score-consistency and enabling stable, aggressive few-step distillation.

6. Empirical Performance and Effects

Empirical comparisons on VBench-T2V under 4-step sampling settings isolate the impact of $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$6:

• DMD2 (reverse-KL, $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$7): FVD = 85.05, OptFlow = 77.46, DynDeg = 80.56
• TSG-Fisher (stop-gradient Fisher, $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$8): FVD = 82.98, OptFlow = 71.50, DynDeg = 94.25
• SGMD (Fisher + dual potentials, $S_\text{real}(x_t, t) = S_\text{teacher}(x_t, t)$9): FVD = 84.77, OptFlow = 75.64, DynDeg = 93.06

Switching from reverse-KL to teacher stop-gradient Fisher produces a substantial increase in DynDeg (motion dynamics) from 80.56 to 94.25, indicating that Fisher’s global field-matching signal encourages more vivid motion. OptFlow decreases somewhat (indicating a mode-covering/quality trade-off), but FVD improves alongside motion metrics. Pure Fisher without dual potentials requires more fake-score updates for stability, while SGMD achieves full performance with a single update (%%%%43$\phi$44%%%% speedup).

Human studies indicate strong preference for SGMD’s motion quality and overall video realism, with text alignment and visual metrics remaining competitive relative to DMD2.

7. Summary and Significance

The teacher stop-gradient Fisher objective establishes a one-sided, Fisher-style score matching loss with robust optimization properties in distillation scenarios. It:

• Offers a stable alternative to reverse-KL for distribution matching in fast video diffusion,
• Eliminates unreliable teacher gradients via stop-gradient,
• Retains theoretical alignment with reverse-KL under exact tracking,
• Is central in SGMD’s dual potential framework, enabling both strong motion dynamics and training efficiency in few-step video models (Wu et al., 28 May 2026).