Consistency Mid-Training (CMT)

• Consistency Mid-Training is an intermediate training paradigm that bridges diffusion pre-training and flow map learning to yield trajectory-consistent mappings.
• It employs numerical ODE solvers to accurately regress from intermediate states to clean data, stabilizing target mappings and accelerating convergence.
• CMT reduces computational costs and hyperparameter tuning while outperforming traditional methods on efficiency and stability in few-step generative models.

Consistency Mid-Training (CMT) is an intermediate training paradigm developed to address the instability, inefficiency, and optimization difficulties endemic to few-step generative models such as Consistency Models (CM), Mean Flow (MF), and other flow map approaches. By inserting a targeted “mid-training” phase between initial diffusion pre-training and the final flow map learning (post-training), CMT yields trajectory-consistent initializations that enable stable and efficient learning of long-jump maps in high-dimensional generative modeling.

1. Theoretical Motivations and Conceptual Framework

The principal motivation for Consistency Mid-Training is that direct conversion from a diffusion model’s pre-trained weights (optimized for infinitesimal time-step denoising) to a global flow map (capable of mapping from arbitrary intermediate states to clean data in one or two steps) is both computationally inefficient and prone to instability. In paradigms such as CM and MF, post-training targets (clean samples or pseudo-targets) drift during training, and instability is exacerbated by reliance on stop-gradient strategies or fragile pseudo-labels (Hu et al., 29 Sep 2025).

CMT intervenes by explicitly introducing a compact mid-training stage that learns, for each diffusion ODE trajectory, an accurate mapping from any intermediate state $x_{t_i}$ at time $t_i$ to a clean data target $x_0$. This process can be formally described as learning $f_\theta(x_{t_i}, t_i) \approx x_0$ for all relevant $t_i$ along a solver-generated trajectory starting from the prior distribution. The initialization provided by CMT more closely approximates the ideal “oracle” flow map:

$$x_{t \rightarrow 0}(x_t) = x_t + \int_t^0 v(x_u, u) du$$

where $v(x_u, u)$ is the drift from the teacher diffusion model.

2. CMT Methodology

CMT is structured in three distinct phases:

a. Pre-Training

A high-quality diffusion model is trained using standard probability flow ODE machinery. This establishes a well-behaved drift $v(\cdot, \cdot)$, providing clean mapping trajectories from a simple prior (typically Gaussian) to the data distribution.

b. Mid-Training (CMT Stage)

For each sampled trajectory:

• A numerical ODE solver (e.g., third-order DPM-Solver++) is used to compute the states $\{x_{t_i}\}_{i=0}^N$ from the prior sample $x_T$ to the clean data $x_0$.
• The mid-training model is then trained to regress from each $x_{t_i}$ to $x_0$:

$$\mathcal{L}_{\text{CMT}}(\theta) = \mathbb{E}_{i} \mathbb{E}_{x_{t_i}} \left[d\left(f_\theta(x_{t_i}, t_i), x_0\right)\right]$$

where $d(\cdot, \cdot)$ is an appropriate metric (e.g., LPIPS, $\ell_2^2$).

This yields a trajectory-consistent initialization, as the model learns to “jump” directly to $x_0$ from any intermediate ODE state along the teacher-generated path.

c. Post-Training (Flow Map Training)

The weights from CMT are used to initialize post-training of the flow map model (using ECT, ECD, or other distillation strategies). Given good initialization, the loss landscape is stabilized and convergence is dramatically accelerated (Hu et al., 29 Sep 2025).

3. Comparisons With Prior Consistency and Flow Map Methods

CMT’s mid-training approach contrasts with prior CM and MF strategies in several ways:

Method	Target Specification	Initialization	Notable Issues Addressed
CM/CD/MF	Pseudo-targets, stop-grad	Diffusion or random	Instability, slow conv.
CMT	Trajectory-aligned targets	CMT-initialized (ODE)	Improved init, robust conv.

• In CM (with distillation or standalone consistency training), the training dynamics are governed by pseudo-targets produced either from teacher models or from adjacent time steps, with stop-gradient strategies complicating optimization.
• In MF, the average drift over intervals must be learned, which demands costly Jacobian-vector products and is sensitive to batch size and schedules.
• CMT leverages a fixed, numerically stable reference trajectory, replacing heuristic time-sampling and fragile pseudo-targets with a global, fixed mapping reference.

4. Empirical Results and Performance

CMT has demonstrated leading performance on a range of image generation tasks, particularly in few-step regimes:

• CIFAR-10 (32×32): two-step FID of 1.97 using CMT initialization with post-training, outperforming diffusion baselines using 35 steps and earlier state-of-the-art CMs (Hu et al., 29 Sep 2025).
• ImageNet 64×64: CMT reaches a two-step FID of 1.32, requiring only 6.4–12.8M “images processed” versus hundreds of millions for conventional methods.
• ImageNet 512×512 (class-conditional): two-step FID of 1.84, with up to 93% reduction in GPU-hour cost.
• ImageNet 256×256 (latent): one-step FID 3.34, halving the total flow map training time compared to MF from scratch.

These results underscore the reduction in required training data and compute—up to 98% less—when compared against baselines, with improved stability (lower variance and bias in gradients) and competitive or superior generation quality.

5. Practical Benefits and Implementation

The introduction of CMT yields significant advances in the training and deployment of flow map models:

• Elimination of Heuristic Schedules: CMT obviates the need for tuning annealing parameters (e.g., $\Delta t$), sampling weightings, and stop-gradient heuristics.
• Stable and Consistent Initialization: By aligning with ODE solver outputs, models benefit from numerically stable and fixed target mappings throughout mid-training.
• Lower Resource Requirements: The approach enables state-of-the-art generation from limited compute budgets, smaller batch sizes, or less aggressive hardware scheduling.
• Simplified Hyperparameter Tuning: CMT reduces the hyperparameter search space, as stability allows for the use of standard optimizers and constant learning rates.

6. Broader Impact and Research Directions

By decoupling the pre-learning of ODE trajectories from post-training, CMT opens the door to efficient and scalable training regimes for a wide spectrum of generative modeling tasks, including high-resolution vision synthesis and potentially text-to-image or conditional generative tasks.

Future research avenues highlighted in (Hu et al., 29 Sep 2025) include:

• Extending CMT to alternative modalities (latent, text, conditional generation).
• Exploring advanced loss metrics (e.g., perceptual or latent domain variants of LPIPS).
• Improving ODE solver techniques for even more data/compute-efficient mid-training.
• Further theoretical refinement of bias and variance reductions attributable to mid-training, with possible transfer to adversarial or reinforcement learning generative settings.

7. Theoretical and Optimization Implications

CMT fundamentally reduces the “drift” and instability associated with direct flow map learning from diffusion models. Theoretically, replacing dynamic pseudo-targets by fixed ODE solver–generated references reduces alignment error, stabilizes gradients, and ensures that the model’s initialization captures the entire continuous ODE trajectory. Because CMT-generated targets are strictly trajectory-consistent and globally fixed, final flow map training converges quickly without the pathological degeneracies of standard CM or MF training.

This foundational rationale is further supported by recent advances in optimization of consistency models, such as variance-reduced learning techniques (Wang et al., 24 Oct 2024), manifold-aligned loss functions (Kim et al., 1 Oct 2025), and efficient differential consistency tuning (Geng et al., 20 Jun 2024), which can be integrated into or combined with CMT for further benefit.

---

In summary, Consistency Mid-Training establishes a principled, trajectory-consistent intermediate training regime that robustly bridges the gap between diffusion-model pre-training and efficient, stable few-step generative modeling. It addresses core optimization bottlenecks, delivers improvements in data and computational efficiency, and lays the groundwork for future research in advanced flow map architectures and generative applications (Hu et al., 29 Sep 2025).