Velocity Distillation in Flow Models

• Velocity Distillation is a family of few-step distillation methods that leverages teacher-provided velocity fields to reduce the computational cost of iterative ODE-based sampling.
• It supervises transport-level dynamics by aligning integrated or average velocities rather than matching instantaneous dynamics, enhancing stability and efficiency.
• Empirical comparisons show that VD methods can significantly speed up inference—often achieving up to 9× improvements—while maintaining nearly the same quality as full iterative processes.

Searching arXiv for the cited papers and closely related "velocity distillation" / "mean flow distillation" work. Velocity Distillation (VD) refers to a family of few-step distillation procedures for flow-based generative models in which a pretrained teacher supplies a velocity field and the student is trained from quantities derived from that field rather than from a full iterative sampler. In recent literature, the term is used in several closely related senses: a KL-based marginal-distribution distillation objective built on learned velocities in 3D generation, an integral-velocity distillation procedure for few-step speech generation, and mean-flow distillation based on average velocity fields. This suggests that VD functions less as a single standardized loss than as a common viewpoint on transport distillation in continuous-time generative models (Zhou et al., 4 Sep 2025, Wang et al., 9 Oct 2025, Zhao et al., 9 Jun 2026, Gao et al., 2 Jun 2026).

1. Terminological scope and problem setting

The shared motivation across VD formulations is the same: flow-based models often require iterative ODE-based sampling with substantial inference cost, and distillation seeks to replace that long-horizon transport with a student that uses far fewer function evaluations. In the 3D setting, the target is a one-step mapping from an arbitrary diffusion marginal $q_t(\mathbf x_t)$ back to the data distribution $q_{\rm data}$. In speech synthesis, the target is few-step generation for token-to-spectrogram and text-to-spectrogram tasks. In Mean Flow Distillation, the target is a single-step student generator $G_\theta$ trained by aligning time-integrated mean velocities rather than instantaneous ones (Zhou et al., 4 Sep 2025, Wang et al., 9 Oct 2025, Zhao et al., 9 Jun 2026).

The literature distinguishes several non-identical supervisory targets under the broader VD label.

Formulation	Student target	Noted property
MDT-dist VD	Marginal distributions via a KL-gradient estimator using $\mathbf u_\theta$ and $\mathbf v_{\rm pre}$ as approximate scores	No second-derivative detachment is required; gradients are unbiased
IntMeanFlow IVD	Teacher’s integral velocity $\bar v_{\rm teacher}(x,t,r)$	No JVPs and no self-bootstrap
Mean Flow Distillation	Average velocity fields $U^P$ and $U^Q$ over $[s,t]$	Temporal low-pass filter; Mean Flow Matching Theorem

A recurrent source of confusion is the assumption that VD is synonymous with direct instantaneous velocity matching. The 3D distillation paper explicitly separates Velocity Matching (VM) from Velocity Distillation (VD), and both IntMeanFlow and MFD supervise average or integral velocities rather than instantaneous ones. This suggests that contemporary usage of the term is organized around transport-level supervision from a teacher velocity field, but not around a unique loss construction (Zhou et al., 4 Sep 2025, Wang et al., 9 Oct 2025, Zhao et al., 9 Jun 2026).

2. Velocity fields, transport, and average velocity

The common mathematical substrate is the continuous-time flow. In continuous normalizing flows, one defines a time-dependent vector field $v(x(t),t)$ such that

$q_{\rm data}$0

Training typically matches the model’s instantaneous velocity $q_{\rm data}$1 to a ground-truth velocity that transports an easy-to-sample base distribution $q_{\rm data}$2 to a target data distribution $q_{\rm data}$3 over $q_{\rm data}$4. The instantaneous velocity is

$q_{\rm data}$5

The average, or mean, velocity over an interval $q_{\rm data}$6 is

$q_{\rm data}$7

This quantity captures the overall transport from $q_{\rm data}$8 to $q_{\rm data}$9 rather than the infinitesimal velocity at each time (Wang et al., 9 Oct 2025).

In diffusion models viewed through the probability-flow ODE, the corresponding velocity field can be written as

$G_\theta$0

Few-step distillation then amounts to approximating the exact flow map $G_\theta$1 by a small composition of student maps $G_\theta$2. The theoretical perspective developed for diffusion distillation emphasizes that local one-step approximation errors are amplified by the time-integrated Jacobian bound of the probability-flow ODE, so the difficulty of VD depends not only on score approximation but also on dynamical stability (Gao et al., 2 Jun 2026).

This division between local approximation and global transport control underlies the divergence between instantaneous-velocity, integral-velocity, and distribution-level distillation schemes. Methods that supervise integrated or average velocities are motivated by the idea that matching transport over intervals can be more stable than matching highly local dynamics.

3. Velocity Distillation in marginal-data transport distillation

In “Few-step Flow for 3D Generation via Marginal-Data Transport Distillation,” the primary objective is to distill a pretrained model to learn the Marginal-Data Transport (MDT). If a pretrained teacher provides a velocity field $G_\theta$3 satisfying

$G_\theta$4

the student network $G_\theta$5 is intended to replicate the entire transport $G_\theta$6. The corresponding MDT objective is

$G_\theta$7

with $G_\theta$8. Because that time-integral is not directly computable, the paper introduces two surrogates: Velocity Matching (VM) and Velocity Distillation (VD) (Zhou et al., 4 Sep 2025).

VM differentiates the transport target in time and matches the induced student velocity

$G_\theta$9

to the teacher’s instantaneous velocity $\mathbf u_\theta$0 through an MSE loss. In practice, the derivative is discretely approximated and gradients through the finite-difference term are detached for stability. The paper states that VM stably matches velocity fields, but inevitably provides biased gradient estimates. VD is introduced to remove that bias at the distribution level. Starting from $\mathbf u_\theta$1, the gradient is rewritten in score-matching form and then approximated using the probability-flow ODE connection

$\mathbf u_\theta$2

This yields the VD gradient estimator

$\mathbf u_\theta$3

The paper characterizes VD as a score-like distillation on marginals that restores unbiased, distribution-level supervision, and reports that VM+VD outperforms VM alone by approximately $\mathbf u_\theta$4–$\mathbf u_\theta$5 FD points and $\mathbf u_\theta$6 ULIP on geometry. In its ablation, VM alone yields $\mathbf u_\theta$7, $\mathbf u_\theta$8, and $\mathbf u_\theta$9, while VM+VD yields $\mathbf v_{\rm pre}$0, $\mathbf v_{\rm pre}$1, and $\mathbf v_{\rm pre}$2. Qualitatively, VD is described as removing spurious surfaces and completing thin structures better than VM alone or CM baselines (Zhou et al., 4 Sep 2025).

4. Integral and mean-velocity formulations

The speech paper “IntMeanFlow: Few-step Speech Generation with Integral Velocity Distillation” develops Integral Velocity Distillation (IVD) as a direct response to two limitations of applying MeanFlow to TTS: GPU memory overhead from Jacobian-vector products and training instability due to self-bootstrap processes. Instead of predicting instantaneous velocity $\mathbf v_{\rm pre}$3, the student predicts the integral velocity

$\mathbf v_{\rm pre}$4

Discretizing $\mathbf v_{\rm pre}$5 into $\mathbf v_{\rm pre}$6 steps lets the total displacement

$\mathbf v_{\rm pre}$7

serve as a discrete approximation of the integral velocity. The distillation loss is

$\mathbf v_{\rm pre}$8

where

$\mathbf v_{\rm pre}$9

Because IVD uses only the teacher’s instantaneous velocity samples to estimate the integral, no time derivatives or JVPs are needed, and because the student is directly supervised by the teacher’s velocities, there is no self-bootstrap stage that risks collapse (Wang et al., 9 Oct 2025).

The paper pairs IVD with Optimal Step Sampling Search (O3S), whose goal is, for a fixed total number of sampling steps $\bar v_{\rm teacher}(x,t,r)$0, to find time points $\bar v_{\rm teacher}(x,t,r)$1 that maximize a speech-quality metric on a development set without increasing $\bar v_{\rm teacher}(x,t,r)$2. The search initializes uniformly, then performs a ternary search over each interior step until convergence. This makes the choice of sampling schedule part of the distillation pipeline rather than a fixed postulate (Wang et al., 9 Oct 2025).

Mean Flow Distillation (MFD) generalizes the average-velocity idea to a broader flow-matching framework. Let $\bar v_{\rm teacher}(x,t,r)$3 denote the teacher’s instantaneous velocity field transporting $\bar v_{\rm teacher}(x,t,r)$4, and let an auxiliary flow $\bar v_{\rm teacher}(x,t,r)$5 approximate the student distribution $\bar v_{\rm teacher}(x,t,r)$6. For $\bar v_{\rm teacher}(x,t,r)$7, the average velocity fields are defined through a fixed ODE integrator $\bar v_{\rm teacher}(x,t,r)$8 by

$\bar v_{\rm teacher}(x,t,r)$9

The MFD loss is

$U^P$0

Under Lipschitz continuity and a strictly positive-definite Bregman divergence, the Mean Flow Matching Theorem states that if $U^P$1, then $U^P$2. The same paper argues that MFD acts as a temporal low-pass filter: if instantaneous supervision has zero-mean noise $U^P$3, then averaging over $U^P$4 Euler steps reduces the variance term from $U^P$5 to $U^P$6, so $U^P$7 (Zhao et al., 9 Jun 2026).

5. Stability, bias, and time discretization

The theoretical analysis of diffusion-model flow distillation sharpens the conditions under which VD is easy or hard. If $U^P$8 is approximated on an interval $U^P$9 by $U^Q$0 with uniform error at most $U^Q$1, and $U^Q$2, then Grönwall’s inequality yields

$U^Q$3

The resulting message is that local approximation error is not the entire problem; its amplification is governed by the integrated stability profile $U^Q$4 (Gao et al., 2 Jun 2026).

In the Gaussian-mixture Ornstein–Uhlenbeck setting, the same framework derives an explicit time-dependent Lipschitz constant

$U^Q$5

and a flow-map stability bound

$U^Q$6

It further identifies a Lipschitz-mismatch regime in which one-step distillation is structurally unfavorable: if the teacher’s tail-map Lipschitz constant exceeds the student class’s representable Lipschitz budget, then even an arbitrarily accurate fixed-time score approximation does not guarantee small end-to-end error. This is a direct theoretical limitation on aggressive compression to a single step (Gao et al., 2 Jun 2026).

A corresponding remedy is a stability-balanced nonuniform time grid. Defining the cumulative stability coordinate

$U^Q$7

one chooses the grid by uniform spacing in $U^Q$8-space so that each segment has approximately equal stability exponent. Experiments support this prediction: with $U^Q$9 segments, the relative end-to-end MSE is reduced by up to $[s,t]$0 compared with the uniform-grid baseline. A plausible implication is that step placement is a central part of VD design, not merely a numerical afterthought; the O3S procedure in IntMeanFlow reaches a similar conclusion from the speech-synthesis side (Gao et al., 2 Jun 2026, Wang et al., 9 Oct 2025).

6. Empirical profile, misconceptions, and prospective extensions

The empirical record reported in the cited papers is strongly task-dependent but consistent in one respect: few-step VD can preserve much of a teacher’s quality while sharply reducing latency. In TRELLIS-based 3D generation, the teacher uses $[s,t]$1 steps$[s,t]$2 and attains $[s,t]$3, $[s,t]$4, and $[s,t]$5 at $[s,t]$6. MDT-dist with $[s,t]$7 steps attains $[s,t]$8, $[s,t]$9, and $v(x(t),t)$0 at $v(x(t),t)$1, a $v(x(t),t)$2 speedup, while $v(x(t),t)$3 steps attain $v(x(t),t)$4, $v(x(t),t)$5, and $v(x(t),t)$6 at $v(x(t),t)$7, a $v(x(t),t)$8 speedup. At the same $v(x(t),t)$9 setting, the method reports $q_{\rm data}$00 versus CM $q_{\rm data}$01, PCM $q_{\rm data}$02, and sCM $q_{\rm data}$03, with geometry $q_{\rm data}$04 versus $q_{\rm data}$05, $q_{\rm data}$06, and $q_{\rm data}$07 respectively (Zhou et al., 4 Sep 2025).

In speech synthesis, IntMeanFlow reports $q_{\rm data}$08-NFE inference for token-to-spectrogram and $q_{\rm data}$09-NFE inference for text-to-spectrogram tasks while maintaining high-quality synthesis. For English Text2Mel, the $q_{\rm data}$10-NFE teacher has WER $q_{\rm data}$11, SIM-o $q_{\rm data}$12, UTMOS $q_{\rm data}$13, UV.MOS $q_{\rm data}$14, and RTF $q_{\rm data}$15. IntMeanFlow $q_{\rm data}$16-step + O3S with a teacher at $q_{\rm data}$17 NFE reports WER $q_{\rm data}$18, SIM-o $q_{\rm data}$19, UTMOS $q_{\rm data}$20, UV.MOS $q_{\rm data}$21, CMOS $q_{\rm data}$22, SMOS $q_{\rm data}$23, and RTF $q_{\rm data}$24, described as a $q_{\rm data}$25 inference speedup with negligible quality drop. For Token2Mel, the original $q_{\rm data}$26-NFE system reports WER $q_{\rm data}$27, SIM-o $q_{\rm data}$28, UTMOS $q_{\rm data}$29, UV.MOS $q_{\rm data}$30, and RTF $q_{\rm data}$31, while IntMeanFlow $q_{\rm data}$32-step reports WER $q_{\rm data}$33, SIM-o $q_{\rm data}$34, UTMOS $q_{\rm data}$35, UV.MOS $q_{\rm data}$36, CMOS $q_{\rm data}$37, SMOS $q_{\rm data}$38, and RTF $q_{\rm data}$39, a $q_{\rm data}$40 speedup with imperceptible quality loss. Eliminating JVPs reduced peak GPU memory by $q_{\rm data}$41, and no training collapse was observed (Wang et al., 9 Oct 2025).

MFD extends the same general program beyond 3D and speech. On 4D occupancy forecasting, the teacher at $q_{\rm data}$42 steps reports IoU $q_{\rm data}$43 and mIoU $q_{\rm data}$44, while single-step MFD reports IoU $q_{\rm data}$45 and mIoU $q_{\rm data}$46; the paper states that MFD retains $q_{\rm data}$47 of the teacher’s IoU while increasing throughput from $q_{\rm data}$48 to $q_{\rm data}$49 FPS. On text-to-image with SANA $q_{\rm data}$50B, the teacher at $q_{\rm data}$51 steps reports Aesthetic $q_{\rm data}$52, while $q_{\rm data}$53-step MFD reports Aesthetic $q_{\rm data}$54 and FID $q_{\rm data}$55. On CIFAR-10, the teacher at $q_{\rm data}$56 steps has FID $q_{\rm data}$57 and recall $q_{\rm data}$58, while $q_{\rm data}$59-step MFD reports FID $q_{\rm data}$60 and recall $q_{\rm data}$61, i.e. $q_{\rm data}$62 of teacher recall. The paper also reports smooth losses, narrow loss distributions, and low gradient variance (Zhao et al., 9 Jun 2026).

Two clarifications follow from these results. First, VD is not equivalent to one specific objective: VM, VD, IVD, and MFD all use teacher velocities, but they supervise different mathematical objects. Second, one-step distillation is not universally viable: the diffusion-theoretic analysis identifies settings in which a one-step student is structurally mismatched to the teacher dynamics (Gao et al., 2 Jun 2026). The prospective extensions named in the cited work are correspondingly diverse: extending IVD to unconditional audio or music generation, integrating IVD with consistency-model distillation for further NFE reduction, learning adaptive sampling schedules jointly with model parameters rather than via post-hoc search, and applying the 3D VD recipe to flow-matching models such as YM–Liu’s FM, conditional flows, vector-quantized flows, and certain diffusion solvers that admit a deterministic ODE viewpoint (Wang et al., 9 Oct 2025, Zhou et al., 4 Sep 2025).