Velocity Distillation in Generative Models

• Velocity distillation is a method where a compact student model learns the continuous velocity field from a high-sampling-cost teacher, capturing the full generative trajectory.
• It utilizes distinct approaches such as IntMeanFlow, π-Flow, ArcFlow, and VD to match instantaneous or integrated teacher velocities without costly evaluations.
• Empirical results across image, speech, and 3D domains demonstrate significant acceleration and quality retention, despite challenges in optimal step placement and teacher dependency.

Velocity distillation refers to a set of model distillation methodologies for flow-based and diffusion generative models in which a few-step student network is trained to directly reproduce the time-dependent velocity field learned by a much larger, high-sampling-cost teacher system. Unlike conventional approaches that distill towards denoised targets or match final sample distributions, velocity distillation propagates the full generative trajectory by matching the vector field (velocity) that defines the probability flow ODE underlying generation. This paradigm has become central for scaling diffusion, flow, and score-based generative modeling to regimes requiring fast, high-fidelity, few-shot synthesis across modalities such as images, speech, and 3D structures.

1. Formal Foundations of Velocity Distillation

Most diffusion and flow-matching models define generation as solving a deterministic ODE of the form: $$\frac{d}{dt}z_t = v(z_t, t; \theta),\quad z_0\sim p_0,\,z_1\sim p_1,\,t\in[0,1]$$ where $v(x, t)$, the instantaneous velocity field, governs the evolution of data from noise $p_1$ to data $p_0$. The teacher model, typically trained with hundreds of network function evaluations (NFEs), learns to accurately estimate $v(x, t)$ across continuous time.

Velocity distillation is the process of training a smaller, few-step student to match this teacher's velocity field. The student may either regress the instantaneous velocity at queried states (as in flow matching and policy-based architectures) or the average/integral velocity over longer intervals (as in MeanFlow/IntMeanFlow). This vector field matching ensures that the student reconstructs not only endpoints but also the generative dynamics, enabling high-quality, low-NFE synthesis (Wang et al., 9 Oct 2025, Chen et al., 16 Oct 2025, Lukoianov et al., 2024).

2. Methodological Variants and Loss Formulations

Velocity distillation is realized using several distinct mathematical objectives and architectural strategies:

• Integral Velocity Distillation (IntMeanFlow): The student network $s_\phi(x; \tau_s, \tau_e)$ learns to predict the average velocity

$$\bar{v}(x; \tau_s, \tau_e) = \frac{1}{\tau_e - \tau_s} \int_{\tau_s}^{\tau_e} v(x, t)\,dt$$

by minimizing a squared loss against the time-averaged teacher trajectory. The student only matches the teacher's integrated velocity, never relying on its own predictions, eliminating the need for costly Jacobian-vector products (JVPs) and avoiding instability from self-bootstrap procedures (Wang et al., 9 Oct 2025).

• Policy-based Velocity Distillation ($\pi$-Flow): The student outputs a closed-form policy $\pi_\phi$ at an initial anchor $(x_{t_\text{src}}, t_\text{src})$, which analytically provides $v(x_t, t)$ at future sub-times at negligible cost. Training uses an on-policy imitation loss:

$$\mathcal{L}_\text{FM} = \mathbb{E}_{t, x_t} \| v_\theta(x_t, t) - \pi_\phi(x_t, t; t_{\text{src}}) \|_2^2$$

On-policy training ensures that the student corrects its own drift and reproduces the teacher's dynamic velocity field without quality-diversity collapse (Chen et al., 16 Oct 2025).

• Analytic Integration (ArcFlow): The student parameterizes the velocity field as a mixture of K continuous “momentum” processes,

$$v_\text{student}(x, t; \theta) = \sum_{k=1}^K \pi_k(x)\,v_k(x)\,[\gamma_k(x)]^{1-t}$$

enabling closed-form integration over long intervals for each synthesis step. Distillation minimizes mean-squared velocity error along the teacher's high-NFE trajectory (Yang et al., 9 Feb 2026).

• Distributional Velocity Distillation (VD): In complex settings like 3D generation, VD matches the student’s marginal density induced via velocity fields (KL divergence), not just per-point velocities. For student $\phi_\theta$ and teacher $v_\text{pre}$,

$$L_\text{VD}(\theta) = \mathbb{E}_{t, z', z''} \big[ - (u_\theta(x_t', t) - v_\text{pre}(x_t', t)) \cdot \partial x_t'/\partial \theta \big]$$

This unbiased gradient matches full density marginals, complementing direct velocity matching (Zhou et al., 4 Sep 2025).

3. Empirical Results and Practical Impact

Velocity distillation has demonstrated substantial empirical benefits across modalities:

• Speech Synthesis: IntMeanFlow achieves $1$-NFE token-to-spectrogram and $3$-NFE text-to-spectrogram synthesis with $\approx 10\times$ lower real-time factor and minimal degradation in WER or speaker similarity compared to 32-NFE teacher models. GPU memory reduced by up to $5\times$; training is consistently stable (Wang et al., 9 Oct 2025).
• Image Generation: $\pi$-Flow attains ImageNet $256^2$ FID of $2.85$ at $1$-NFE, outperforming MeanFlow and equaling or surpassing multi-step teacher quality. For large-scale text-to-image (FLUX.1-12B, Qwen-Image-20B), $\pi$-Flow at $4$-NFE matches or increases diversity and preserves alignment, outperforming prior few-step methods (Chen et al., 16 Oct 2025). ArcFlow delivers $40\times$ speedup (2 NFEs) over multi-step teachers on Qwen/FLUX without significant quality degradation (e.g., FID $12.4$ vs. teacher's $3.78$) (Yang et al., 9 Feb 2026).
• 3D Generation: MDT-dist (Velocity Distillation + Velocity Matching) achieves $6.5$–$9\times$ speedup on A800 GPUs for TRELLIS-based 3D synthesis, maintains high visual and geometric fidelity, and outperforms existing 3D consistency model distillations by $5$–$30\%$ on FID metrics (Zhou et al., 4 Sep 2025).

Quantitative results are summarized below:

Domain/Method	NFE	Teacher FID/pFID	Student FID/pFID	Speedup
IntMeanFlow (Speech)	1-3	–	≈+1–2% quality	10×
π-Flow (ImageNet)	1	3.43 (MeanFlow)	2.85	–
ArcFlow (Qwen-Image)	2	3.78	12.40	40×
MDT-dist (TRELLIS 3D)	2×2	65.24	110.9	6.5×

4. Architectural and Optimization Considerations

• Removal of JVPs and Self-Bootstrap: IntMeanFlow obtains average velocities by integrating teacher outputs and never requires backpropagation through ODE integrators or computation of JVPs—operations that are both GPU-intensive and incompatible with some memory-efficient attention layers (Wang et al., 9 Oct 2025).
• Imitation via On-Policy Rollouts: $\pi$-Flow's imitation distillation samples student rollouts, matching velocities along these on-policy trajectories and avoiding the error accumulation typical in off-policy distillation pipelines (Chen et al., 16 Oct 2025).
• Search for Optimal Step Grids: The Optimal Step Sampling Search (O3S) algorithm, introduced within IntMeanFlow, identifies time grids that maximize synthesis quality for fixed NFE without extra inference cost, e.g., concentrating steps in noisier early regions (Wang et al., 9 Oct 2025).
• Momentum Mixtures and Analytic Solvers: ArcFlow models velocity evolution within each large ODE step with an exponential mixture, facilitating analytic integration and mitigating discretization errors inherent in linear shortcut methods (Yang et al., 9 Feb 2026).

5. Scope of Applicability and Limitations

Velocity distillation is broadly modal-agnostic. The core requirement is the existence of a high-quality pretrained teacher capable of providing the relevant instantaneous (or integrated) velocity field. Applications span text-to-speech, text-to-image, image synthesis, and 3D object generation (Wang et al., 9 Oct 2025, Zhou et al., 4 Sep 2025, Chen et al., 16 Oct 2025). A plausible implication is that these methods will generalize to other domains (e.g., audio, video) governed by continuous flow or diffusion ODEs.

Principal limitations include:

• Teacher Dependency: All distillation objectives rely on a fully trained (and often NFE-expensive) teacher.
• Interval Bias: For large integration intervals or very small NFE, student predictions may average over highly nonlinear teacher fields, introducing bias (Wang et al., 9 Oct 2025).
• Step Placement: Optimal allocation/placement of steps (i.e., adaptive NFE, dynamic interval widths) remains an open problem, though algorithms like O3S address fixed-K settings.

6. Theoretical Insights and Related Discourses

A salient unification is that score distillation, DDIM, and velocity distillation all operate by leveraging the underlying ODE velocity field of the generative process. Score Distillation via Reparametrized DDIM demonstrates that the guidance used in 3D SDS is a high-variance discretization of the DDIM velocity path—variance reduction on the velocity difference enables recovery of high-frequency detail in 3D asset synthesis, closing the gap to 2D sample quality (Lukoianov et al., 2024).

Further, velocity distillation may either directly regress velocity fields (VM), perform KL-based density distillation (VD), or combine both for variance and bias trade-offs. Analytic integration approaches (e.g., ArcFlow) explicitly parameterize the non-linear evolution of velocities, yielding high-fidelity trajectories even at minimal NFE (Yang et al., 9 Feb 2026).

7. Future Directions

Current research explores extending velocity distillation to:

• Adaptive and Conditional Intervalization: Dynamic restep placement or conditionalizing intervals on input complexity.
• Joint Optimization of Step Count and Grid: Moving beyond fixed-K O3S to end-to-end NFE/grid co-optimization (Wang et al., 9 Oct 2025).
• Unifying Density and Trajectory Matching: Combining VM and VD for minimal bias and maximal distributional fidelity (Zhou et al., 4 Sep 2025).
• Modal Scalability: Application to video, high-resolution multi-view synthesis, and non-Gaussian diffusion processes.

Velocity distillation, by directly targeting the generative vector field, constitutes a principal driver for enabling fast, stable, and high-fidelity few-step generative modeling across modalities. Its theoretical underpinnings and practical realization continue to guide advances in efficient generative model deployment.