--- title: Velocity Distillation in Flow Models url: https://www.emergentmind.com/topics/velocity-distillation-vd type: topic --- # Velocity Distillation in Flow Models Searching arXiv for the cited papers and closely related "velocity distillation" / "mean flow distillation" work. Velocity Distillation (VD) refers to a family of few-step distillation procedures for flow-based generative models in which a pretrained teacher supplies a velocity field and the student is trained from quantities derived from that field rather than from a full iterative sampler. In recent literature, the term is used in several closely related senses: a KL-based marginal-distribution distillation objective built on learned velocities in 3D generation, an integral-velocity distillation procedure for few-step speech generation, and mean-flow distillation based on average velocity fields. This suggests that VD functions less as a single standardized loss than as a common viewpoint on transport distillation in continuous-time generative models [2509.04406] [2510.07979] [2606.11155] [2606.03820]. ## 1. Terminological scope and problem setting The shared motivation across VD formulations is the same: flow-based models often require iterative ODE-based sampling with substantial inference cost, and distillation seeks to replace that long-horizon transport with a student that uses far fewer function evaluations. In the 3D setting, the target is a one-step mapping from an arbitrary diffusion marginal \(q_t(\mathbf x_t)\) back to the data distribution \(q_{\rm data}\). In speech synthesis, the target is few-step generation for token-to-spectrogram and text-to-spectrogram tasks. In Mean Flow Distillation, the target is a single-step student generator \(G_\theta\) trained by aligning time-integrated mean velocities rather than instantaneous ones [2509.04406] [2510.07979] [2606.11155]. The literature distinguishes several non-identical supervisory targets under the broader VD label. | Formulation | Student target | Noted property | |---|---|---| | MDT-dist VD | Marginal distributions via a KL-gradient estimator using \(\mathbf u_\theta\) and \(\mathbf v_{\rm pre}\) as approximate scores | No second-derivative detachment is required; gradients are unbiased | | IntMeanFlow IVD | Teacher’s integral velocity \(\bar v_{\rm teacher}(x,t,r)\) | No JVPs and no self-bootstrap | | Mean Flow Distillation | Average velocity fields \(U^P\) and \(U^Q\) over \([s,t]\) | Temporal low-pass filter; Mean Flow Matching Theorem | A recurrent source of confusion is the assumption that VD is synonymous with direct instantaneous velocity matching. The 3D distillation paper explicitly separates Velocity Matching (VM) from Velocity Distillation (VD), and both IntMeanFlow and MFD supervise average or integral velocities rather than instantaneous ones. This suggests that contemporary usage of the term is organized around transport-level supervision from a teacher velocity field, but not around a unique loss construction [2509.04406] [2510.07979] [2606.11155]. ## 2. Velocity fields, transport, and average velocity The common mathematical substrate is the continuous-time flow. In continuous normalizing flows, one defines a time-dependent vector field \(v(x(t),t)\) such that $$ \frac{d}{dt}\,x(t) \;=\; v\bigl(x(t),\,t\bigr). $$ Training typically matches the model’s instantaneous velocity \(v(x(t),t)\) to a ground-truth velocity that transports an easy-to-sample base distribution \(p_0\) to a target data distribution \(p_1\) over \(t\in[0,1]\). The instantaneous velocity is $$ v\bigl(x(t),t\bigr)\;=\;\frac{d}{dt}\,x(t). $$ The average, or mean, velocity over an interval \([0,T]\) is $$ \bar v\bigl(x,0,T\bigr) \;=\; \frac{x(T)-x(0)}{T} \;\approx\; \frac{1}{T}\int_{0}^{T} v\bigl(x(t),t\bigr)\,dt. $$ This quantity captures the overall transport from \(x(0)\) to \(x(T)\) rather than the infinitesimal velocity at each time [2510.07979]. In diffusion models viewed through the probability-flow ODE, the corresponding velocity field can be written as $$ v(x,t) = -\beta_t\bigl( x + \nabla_x \log p_t(x) \bigr). $$ Few-step distillation then amounts to approximating the exact flow map \(\Phi_{0\leftarrow T}\) by a small composition of student maps \(\Psi_{t_k\leftarrow t_{k-1}}\). The theoretical perspective developed for diffusion distillation emphasizes that local one-step approximation errors are amplified by the time-integrated Jacobian bound of the probability-flow ODE, so the difficulty of VD depends not only on score approximation but also on dynamical stability [2606.03820]. This division between local approximation and global transport control underlies the divergence between instantaneous-velocity, integral-velocity, and distribution-level distillation schemes. Methods that supervise integrated or average velocities are motivated by the idea that matching transport over intervals can be more stable than matching highly local dynamics. ## 3. Velocity Distillation in marginal-data transport distillation In “Few-step Flow for 3D Generation via Marginal-Data Transport Distillation,” the primary objective is to distill a pretrained model to learn the Marginal-Data Transport (MDT). If a pretrained teacher provides a velocity field \(\mathbf v_{\rm pre}(\cdot,t)\) satisfying $$ \mathbf x_0 \;=\;\mathbf x_t \;-\;\int_0^t \mathbf v_{\rm pre}(\mathbf x_\tau,\tau)\,d\tau, $$ the student network \(\boldsymbol\phi_\theta(\mathbf x_t,t)\) is intended to replicate the entire transport \(\int_0^t\mathbf v_{\rm pre}\,d\tau\). The corresponding MDT objective is $$ \mathcal L_{\rm MDT}(\theta)\;=\;\E_{t,\mathbf x_0,\mathbf z}\;\Big\|\;t\,\boldsymbol\phi_\theta(\mathbf x_t,t)\;-\;\int_0^t \mathbf v_{\rm pre}(\mathbf x_\tau,\tau)\,d\tau\Big\|^2, $$ with \(\mathbf x_t=(1-t)\mathbf x_0 + t\mathbf z\). Because that time-integral is not directly computable, the paper introduces two surrogates: Velocity Matching (VM) and Velocity Distillation (VD) [2509.04406]. VM differentiates the transport target in time and matches the induced student velocity $$ \mathbf u_\theta(\mathbf x_t,t) = \boldsymbol\phi_\theta(\mathbf x_t,t) + t\,\frac{d}{dt}\boldsymbol\phi_\theta(\mathbf x_t,t) $$ to the teacher’s instantaneous velocity \(\mathbf v_{\rm pre}\) through an MSE loss. In practice, the derivative is discretely approximated and gradients through the finite-difference term are detached for stability. The paper states that VM stably matches velocity fields, but inevitably provides biased gradient estimates. VD is introduced to remove that bias at the distribution level. Starting from \(\KL(p_\theta^t\parallel q_t)\), the gradient is rewritten in score-matching form and then approximated using the probability-flow ODE connection $$ -\nabla_{\mathbf x}\log p_\theta^t(\mathbf x) \approx \mathbf u_\theta(\mathbf x,t), \qquad -\nabla_{\mathbf x}\log q_t(\mathbf x) \approx \mathbf v_{\rm pre}(\mathbf x,t). $$ This yields the VD gradient estimator $$ \nabla_\theta\mathcal L_{\rm VD} = \E_{t,\mathbf z',\mathbf z''} \Bigl[ -\bigl(\mathbf u_\theta(\mathbf x_t',t)-\mathbf v_{\rm pre}(\mathbf x_t',t)\bigr)\, \frac{\partial \mathbf x_t'}{\partial\theta} \Bigr]. $$ The paper characterizes VD as a score-like distillation on marginals that restores unbiased, distribution-level supervision, and reports that VM+VD outperforms VM alone by approximately \(5\)–\(10\) FD points and \(+1.0\) ULIP on geometry. In its ablation, VM alone yields \( \mathrm{FD}_{\text{incep}}\approx 18.4\), \( \mathrm{FD}_{\text{dinov2}}\approx 172\), and \(\mathrm{ULIP}\approx 35.99\), while VM+VD yields \( \mathrm{FD}_{\text{incep}}\approx 18.09\), \( \mathrm{FD}_{\text{dinov2}}\approx 164.2\), and \(\mathrm{ULIP}\approx 36.88\). Qualitatively, VD is described as removing spurious surfaces and completing thin structures better than VM alone or CM baselines [2509.04406]. ## 4. Integral and mean-velocity formulations The speech paper “IntMeanFlow: Few-step Speech Generation with Integral Velocity Distillation” develops Integral Velocity Distillation (IVD) as a direct response to two limitations of applying MeanFlow to TTS: GPU memory overhead from Jacobian-vector products and training instability due to self-bootstrap processes. Instead of predicting instantaneous velocity \(v(x,t)\), the student predicts the integral velocity $$ v_{\rm int}(x,T) = \frac{1}{T}\int_{0}^{T} v_{\rm teacher}\bigl(x(t),t\bigr)\,dt. $$ Discretizing \([0,T]\) into \(K\) steps lets the total displacement $$ \Delta x = \sum_{k=0}^{K-1}(t_{k+1}-t_k)\,v_{\rm teacher}\bigl(x_{t_k},t_k\bigr) \approx \int_{0}^{T}v_{\rm teacher}\bigl(x(t),t\bigr)\,dt $$ serve as a discrete approximation of the integral velocity. The distillation loss is $$ L_{\rm distill} = \mathbb E_{t,r,\,x_t\sim p_t} \Bigl\| u_{\rm student}(x_t,t,r) - \bar v_{\rm teacher}(x,t,r) \Bigr\|^2, $$ where $$ \bar v_{\rm teacher}(x,t,r) = \frac{1}{r-t}\int_t^r v_{\rm teacher}\bigl(x(\tau),\tau\bigr)\,d\tau. $$ Because IVD uses only the teacher’s instantaneous velocity samples to estimate the integral, no time derivatives or JVPs are needed, and because the student is directly supervised by the teacher’s velocities, there is no self-bootstrap stage that risks collapse [2510.07979]. The paper pairs IVD with Optimal Step Sampling Search (O3S), whose goal is, for a fixed total number of sampling steps \(N\), to find time points \(T=\{t_0=098\%\) of the teacher’s IoU while increasing throughput from \(12\) to \(25\) FPS. On text-to-image with SANA \(1.6\)B, the teacher at \(20\) steps reports Aesthetic \(6.335\), while \(4\)-step MFD reports Aesthetic \(6.541\) and FID \(33.28\). On CIFAR-10, the teacher at \(100\) steps has FID \(17.9\) and recall \(0.261\), while \(1\)-step MFD reports FID \(22.7\) and recall \(0.212\), i.e. \(81\%\) of teacher recall. The paper also reports smooth losses, narrow loss distributions, and low gradient variance [2606.11155]. Two clarifications follow from these results. First, VD is not equivalent to one specific objective: VM, VD, IVD, and MFD all use teacher velocities, but they supervise different mathematical objects. Second, one-step distillation is not universally viable: the diffusion-theoretic analysis identifies settings in which a one-step student is structurally mismatched to the teacher dynamics [2606.03820]. The prospective extensions named in the cited work are correspondingly diverse: extending IVD to unconditional audio or music generation, integrating IVD with consistency-model distillation for further NFE reduction, learning adaptive sampling schedules jointly with model parameters rather than via post-hoc search, and applying the 3D VD recipe to flow-matching models such as YM–Liu’s FM, conditional flows, vector-quantized flows, and certain diffusion solvers that admit a deterministic ODE viewpoint [2510.07979] [2509.04406]. Source: https://www.emergentmind.com/topics/velocity-distillation-vd