---
title: 'MeanFlow Pretraining: Efficient One-Step Generation'
url: https://www.emergentmind.com/topics/meanflow-pretraining
type: topic
---

# MeanFlow Pretraining: Efficient One-Step Generation

MeanFlow Pretraining is a methodology for training one-step or few-step generative models via direct regression to the time-averaged velocity field between noise and data. The approach enables high-fidelity sampling in a single function evaluation (1 NFE), dramatically reducing computational cost compared to classical diffusion or flow models that require hundreds of integration steps. Central to MeanFlow is a differential identity linking the instantaneous velocity (as in flow-matching) and the average velocity over a finite interval, which serves as the core target for training. Recent advances—Modular MeanFlow, Decoupled MeanFlow, Rectified MeanFlow, and MeanFlow pretraining in latent or reinforcement learning settings—unify, generalize, and optimize this framework for image synthesis, trajectory modeling, and policy generation, achieving state-of-the-art quality and efficiency.

## 1. Mathematical Foundation: MeanFlow Identity and Average Velocity

Let $x_0 \sim p_{\mathrm{data}}$ (data), $x_1 \sim p_{\mathrm{prior}}$ (noise), and define interpolated states $x_t = (1-\alpha)x_0 + \alpha x_1$ for $t \in [0,1]$, with $\alpha = (t-r)/(1-r)$ and $0\leq r < t \leq 1$ [2508.17426]. The instantaneous velocity field is $v(x_t, t)$, while the MeanFlow average velocity over $[r,t]$ is
$$
u(x_t,r,t) = \frac{1}{t-r} \int_{r}^{t} v(x_{\tau},\tau)\ d\tau.
$$
Crucially, the MeanFlow identity relates the two:
$$
v(x_t,t) = u(x_t,r,t) + (t-r) \frac{d}{dt}u(x_t,r,t),
$$
where $\frac{d}{dt}u = \partial_t u + (\nabla_x u)\cdot v(x_t,t)$. For practical training, $v$ in the Jacobian term may be replaced by $u$ to yield the regression target:
$$
u + (t-r)\big(\partial_t u + \nabla_x u \cdot u\big) \approx \frac{x_1-x_0}{t-r}.
$$
This identity underpins all loss formulations for MeanFlow pretraining.

## 2. Loss Functions, Gradient Modulation, and Training Schedules

Several training losses have been introduced for stable MeanFlow pretraining:
- **Full second-order MeanFlow loss** (not used in practice due to higher-order gradient cost):  
  $$
  \mathcal{L}_{\text{full}}(\theta) = \mathbb{E}_{x_0, x_1, r < t} \left\| u_\theta(x_t,r,t) + (t-r)\left( \partial_t u_\theta + \nabla_x u_\theta \cdot u_\theta \right) - \frac{x_1-x_0}{t-r} \right\|^2
  $$
  [2508.17426].
- **Gradient-modulated MeanFlow loss**: Introduces a partial stop-gradient operator $\mathrm{SG}_\lambda[z] = \lambda z + (1-\lambda)\mathrm{stopgrad}(z)$, where $\lambda \in [0,1]$ controls gradient flow. The training objective becomes:
  $$
  \mathcal{L}_\lambda(\theta) = \mathbb{E}_{x_0, x_1, r < t} \left\| u_\theta(x_t, r, t) + (t - r)\ \mathrm{SG}_\lambda \left[ \partial_t u_\theta(x_t,r,t) + \nabla_x u_\theta(x_t,r,t)\cdot \left(\frac{x_1-x_0}{t-r}\right)\right] - \frac{x_1-x_0}{t-r} \right\|^2.
  $$
- **Curriculum-style warmup**: $\lambda$ is scheduled from $0$ (pure stop-gradient, "coarse" estimation) to $1$ (full backprop, "maximum expressivity") over an initial warmup phase, then fixed at $1$. This procedure allows the network to first stabilize on simple targets before learning detailed dynamics, which is essential to avoid gradient explosion [2508.17426]. A linear schedule is given by $\lambda(s) = \min(1,\, s / T_{\text{warmup}})$ over training step $s$.

Empirically, improper settings of $\lambda$ (e.g., jumping to $1$ too early or staying too low) lead to instability or underfitting. Monitoring gradient norms and the schedule is crucial [2508.17426].

## 3. Pretraining Pipelines: Variants and Structural Optimizations

Recent work has generalized and structurally optimized the MeanFlow paradigm:
- **Joint MeanFlow Training** [2511.19065]: Jointly trains both instantaneous ($v_\theta$) and average ($u_\theta$) velocity heads:
  $$
  \mathcal{L}_v = \mathbb{E} \left[ \|v_\theta(z_t, t) - (\epsilon - x)\|_2^2 \right],\ 
  \mathcal{L}_u = \mathbb{E} \left[ \| u_\theta(z_t, r, t) - \mathrm{sg}(u_{\text{tgt}}(z_t; r, t))\|_2^2 \right]
  $$
  with $u_{\text{tgt}}$ a differentiable target using the learnable $v_\theta$. Crucially, accurate $v_\theta$ formation is a hard prerequisite for good $u_\theta$ learning, necessitating a curriculum that accelerates $v_\theta$ and schedules gap sizes ($\Delta t = t-r$) in $u_\theta$ [2511.19065].
  
- **Decoupled MeanFlow (DMF)** [2510.24474]: Reinterprets pretrained DiT (diffusion transformer) flow models as flow-maps (average velocity predictors) without architectural change. The DiT backbone is split into encoder (sees $t$ embedding) and decoder (sees $r$ embedding); only the second-stage decoder is retrained for arbitrary interval jumps. Training combines flow-matching and MeanFlow losses with adaptive Cauchy loss reweighting.

- **Rectified MeanFlow (Re-MeanFlow)** [2511.23342]: Addresses MeanFlow’s difficulty on curved flows by reparameterizing couplings using a single “reflow” step—first training a flow model, generating rectified couplings via ODE solving, truncating highly curved pairs, then training the MeanFlow predictor. This preconditioning improves convergence and sample quality.

- **Latent-Space and RAE-based MeanFlow** [2511.13019]: MeanFlow is combined with frozen, semantically rich representation autoencoders (e.g., DINO-based). Naive MF training in latent space causes gradient explosion; to counteract this, Consistency Mid-Training initializes the MeanFlow predictor along ODE trajectories from a pre-trained teacher. Two-stage training (distillation, then bootstrapping) provides stable, efficient 1-step generation.

- **Reinforcement Learning Policies** [2511.13035]: Original two-stage MeanFlow Q-learning involves velocity pretraining and distillation. A residual reformulation unifies this into a single policy network $g_\theta(s,a_t,b,t) = a_t - u_\theta(s,a_t,b,t)$, avoiding the expressivity bottleneck of distillation and supporting stable policy learning in a single stage.

## 4. Empirical Results and Training Dynamics

MeanFlow pretraining delivers high one-step sample quality (FID), fast convergence, and robust performance:

| Model / Setting                 | Dataset / Size     | 1-Step FID ↓ | 2-Step FID ↓ | Notes                                 |
|---------------------------------|--------------------|--------------|--------------|----------------------------------------|
| Modular MeanFlow (MMF)          | ImageNet 256×256   | 3.43         | 2.93         | DiT-XL/2 backbone [2508.17426]         |
| Accelerated MF (DTD)            | ImageNet 256×256   | 2.87         | 2.64         | Task affinity + curriculum [2511.19065]|
| Decoupled MF (DMF-XL/2+)        | ImageNet 256×256   | 2.16         | 1.64         | No arch change, Cauchy loss [2510.24474]|
| Rectified MF (Re-MeanFlow)      | ImageNet 256×256   | 3.41         | -            | Truncated, 1 reflow [2511.23342]       |
| MF+RAE (distill)                | ImageNet 256×256   | 2.03         | 1.89         | RAE latent, no guidance [2511.13019]   |

Re-MeanFlow on ImageNet 64 shows FID = 2.87 (EDM2-S), matching or outperforming previous few-step flow-based methods. In the RL context, residual MeanFlow policies outperform prior two-stage flow-policy baselines on 65/73 OGBench and D4RL tasks, maintaining near-zero “bound loss” and expressivity for multimodal action distributions [2511.13035].

Empirical dynamics highlight that accurate instantaneous velocity learning must precede average velocity, and small-gap average velocity supervision stabilizes the progression towards large-jump one-step models [2511.19065]. Failure to respect these curriculum constraints destabilizes or slows convergence.

## 5. Implementation and Practical Strategies

Key hyperparameters and strategies common to successful MeanFlow pretraining:
- Batch sizes: typically $128$–$512$ (larger batches recommended during initial warmup) [2508.17426][2511.13019].
- Learning rates: $10^{-4}$ with cosine decay; Adam or AdamW optimizer [2508.17426][2510.24474][2511.13019].
- Exponential moving average (EMA) parameter stabilization is consistently beneficial (e.g., $\beta=0.9999$) [2511.23342].
- Curriculum over interval gap: linearly or progressively increase gap size in $u_\theta$ head (e.g., $\beta(\Delta t, s)=1-s+\lambda s(1-\Delta t)$) [2511.19065].
- For efficiency, Jacobian-vector-products (JVP) can be computed via forward-mode autodiff; finite-difference approximations can stabilize weakly trained teachers in latent-space MeanFlow [2511.13019].
- Distance truncation in Re-MeanFlow avoids instability from highly curved rectified paths; 10% discarding is usually effective [2511.23342].

In Decoupled MeanFlow, architectural reuse (encoder/decoder split) and combined FM/MF losses enable direct upgrading of pretrained diffusion models without additional weights or layers [2510.24474].

## 6. Applications and Extensions: Image Synthesis, Latent Models, and Policy Learning

MeanFlow pretraining is widely adopted in:
- **Image synthesis**: Modular MeanFlow, DMF, Re-MeanFlow, and latent-space MF approaches all achieve state-of-the-art FID in one or few steps on ImageNet and other datasets [2508.17426][2510.24474][2511.23342][2511.13019].
- **Latent generative models**: MeanFlow leverages powerful representation autoencoders for semantically meaningful generation, significantly reducing computational requirements compared to SD-VAE pipelines [2511.13019].
- **Offline RL policy learning**: Residual MeanFlow reformulation (one-step Q-learning) enables fast, expressive, stable policy deployment in both tabular and continuous-control settings, outperforming both Gaussian and compositionally distilled flow policies [2511.13035].

Curriculum-based, modular, and decoupled MeanFlow strategies continue to generalize as foundational techniques for scalable, efficient, and robust generative modeling.

## 7. Limitations, Pitfalls, and Best Practices

MeanFlow training can exhibit instability due to higher-order gradients unless proper gradient modulation, curriculum scheduling, and architectural choices are applied [2508.17426][2511.19065]. Key pitfalls:
- Jumping directly to full gradient backpropagation ($\lambda=1$) can cause oscillatory, divergent losses.
- Small $\lambda$ throughout restricts expressivity, leading to poor FID.
- Instability with very small interval gaps $(t-r)$; can be mitigated by thresholding $t-r$ and appropriately reweighting loss terms [2508.17426].
- In latent-space MF, naive initialization without trajectory-aware warm-start (e.g., CMT) leads to severe gradient explosion [2511.13019].

Best practices include:
- Monitoring both the regression loss and gradient norms throughout curriculum ramp-up.
- Applying distance truncation and adaptive loss scaling as needed.
- Optionally fine-tuning on long-intervals ($r=0$, $t=1$) late in training to polish one-step accuracy.
- Using adaptive schedules for curriculum and guidance mixing.

MeanFlow pretraining provides a mathematically principled, empirically robust approach for one-step generative modeling, acceleration of flow-based architectures, and efficient reinforcement learning policy generation—with modular, scalable strategies now established across the generative modeling landscape [2508.17426][2511.19065][2510.24474][2511.23342][2511.13019][2511.13035].

Source: https://www.emergentmind.com/topics/meanflow-pretraining