Consistency Distillation with Score Regularization (rCM)

• The paper introduces a principled mixture of a forward consistency objective and a reverse score regularization term to enable fast, high-quality generation.
• The methodology employs a teacher-student framework that leverages continuous-time consistency constraints along probability-flow ODEs paired with DMD-style score corrections.
• Empirical results show that rCM distillation outperforms pure consistency or score-only training by improving VBench scores and significantly reducing sampling steps.

Consistency Distillation with Score Regularization, conventionally abbreviated rCM, denotes a diffusion-distillation framework that combines a consistency model objective with a score-regularization term, typically implemented with a DMD-style reverse-divergence loss. In the reported formulations, the consistency term supplies a forward-type, trajectory-preserving constraint along the teacher probability-flow ODE, while the score term operates on student-generated samples and contributes a mode-seeking correction that pushes the student toward high-density teacher modes. The resulting student is a few-step generator rather than a full-step diffusion sampler, and the framework has been instantiated for large-scale video generation, large-scale text-to-image and text-to-video distillation, and autoregressive streaming video generation (Xiaoyu et al., 27 Apr 2026, Zheng et al., 9 Oct 2025, Zheng et al., 24 Jun 2026).

1. Historical and conceptual lineage

The immediate ancestry of rCM lies in the consistency-model program introduced as a way to learn a direct mapping from a noisy state $x_t$ to the clean endpoint of its probability-flow ODE trajectory, with the defining self-consistency property that all points on the same ODE trajectory map to the same output. That framework already supported both consistency distillation from a pretrained diffusion model and consistency training without a teacher, and it formalized the central consistency map $f_\theta(x,t)$ together with one-step and few-step sampling (Song et al., 2023).

Subsequent work on consistency training clarified that the training dynamics are sensitive to the way teacher and student are coupled. In particular, the analysis of improved consistency training identified a flaw associated with using Exponential Moving Average as the teacher inside the direct consistency-training objective, proposed eliminating EMA from the teacher consistency model during training, replaced LPIPS with a Pseudo-Huber loss, adopted a lognormal noise schedule, and doubled total discretization steps every set number of training iterations (Song et al., 2023). Although this line did not use the rCM name, it sharpened the theoretical and optimization substrate on which later consistency-distillation variants were built.

In parallel, several score-distillation papers progressively tightened the conceptual relation between consistency constraints and score-based guidance. The text-to-3D paper on Guided Consistency Sampling rewrote SDS in sample space and cast SDS, VSD, ISD, and CDS as consistency-style alignments between points on forward and reverse trajectories (Li et al., 2024). SegmentDreamer then reformulated SDS through Segmented Consistency Trajectory Distillation, explicitly separating self-consistency and cross-consistency along PF-ODE segments and interpreting the result as a score-driven consistency loss with a tighter distillation-error bound (Zhu et al., 7 Jul 2025). The explanatory synthesis accompanying Collaborative Score Distillation further interpreted kernel-smoothed score sharing across related samples as a form of multi-sample score regularization, and explicitly connected that mechanism to a general “Consistency Distillation with Score Regularization” viewpoint (Kim et al., 2023).

This suggests that rCM did not emerge in isolation. A plausible implication is that it consolidated several already-visible design tendencies: consistency constraints along PF-ODE trajectories, explicit decomposition between coverage-preserving and guidance-preserving terms, and the use of score-based corrections to stabilize or sharpen distilled generators.

2. Core objective and mathematical structure

In the video-distillation formulation used for Alice v1, rCM is defined as a teacher-student framework in which the student is a consistency model $F_\theta:(x_t,t)\mapsto x_0$ and the total objective is a weighted sum of a continuous-time consistency term and a DMD-style score-regularization term (Xiaoyu et al., 27 Apr 2026):

$$\mathcal{L}_{\text{rCM}}(\theta) = \mathcal{L}_{\text{sCM}}(\theta) + \lambda\,\mathcal{L}_{\text{DMD}}(\theta).$$

The consistency component is written as

$$\mathcal{L}_{\text{sCM}}(\theta) = \mathbb{E}_{t \sim \mathcal{U}[0,T],\, x_t} \left[ w(t) \left\| F_\theta(x_t,t) - F_{\theta^-}(x_t,t) - \frac{g}{\|g\|_2^2 + c} \right\|_2^2 \right],$$

with

$g = w(t)\frac{dF_{\theta^-}(x_t,t)}{dt}.$

The total derivative is evaluated along the teacher probability-flow ODE and decomposes as

$$\frac{dF_{\theta^-}}{dt} = (\nabla_{x_t}F_{\theta^-})\,v_\phi(x_t,t) + \partial_t F_{\theta^-}.$$

This term enforces local invariance of the student prediction along teacher trajectories.

The score-regularization component is DMD-style and is evaluated on student-generated $x_0\sim p_\theta$. In the Alice formulation, it uses the discrepancy between student and teacher denoising outputs to define a stop-gradient target, and the paper interprets that discrepancy as a reverse-KL or score-matching signal that makes the student concentrate on high-density teacher modes rather than covering the full teacher distribution (Xiaoyu et al., 27 Apr 2026). The paper’s conceptual table states the division explicitly: the consistency term behaves like a forward KL (mode-covering) objective, while the DMD term behaves like a reverse KL (mode-seeking) objective.

The large-scale image/video paper retains the same decomposition but expresses it in velocity form under a TrigFlow parameterization with

$x_t = \cos t\, x_0 + \sin t\, \varepsilon.$

There the score-regularized continuous-time consistency model is written as (Zheng et al., 9 Oct 2025)

$$\mathcal{L}_{\text{rCM}}(\theta) = \mathcal{L}_{\text{sCM}}(\theta) + \lambda\,\mathcal{L}_{\text{DMD}}(\theta),$$

with a normalized consistency loss of the form

$f_\theta(x,t)$0

where

$f_\theta(x,t)$1

In that presentation, the DMD term is described as a long-skip regularizer: the sCM component enforces local consistency along teacher trajectories, whereas the DMD component constrains the student’s own terminal samples through teacher scores, thereby correcting long-range jump errors and counteracting the mode-covering tendency of pure sCM (Zheng et al., 9 Oct 2025).

Two features are common across these formulations. First, the student is trained to support few-step generation for arbitrary times along the diffusion trajectory, rather than merely reproducing a fixed discrete sampler. Second, the framework relies on a deliberate balance between forward and reverse divergences. The papers do not present rCM as pure consistency distillation plus an auxiliary heuristic; they present it as a principled mixture of a trajectory-preserving objective and a student-sample-based score objective.

3. Training protocols, data curation, and systems

Representative rCM implementations differ mainly in scale and systems design, but they share the same division between a forward consistency component and a reverse score component.

System	Forward component	Reverse component
Alice v1 (Xiaoyu et al., 27 Apr 2026)	sCM on teacher trajectories	DMD-style score regularization
Large-scale rCM (Zheng et al., 9 Oct 2025)	continuous-time sCM with JVP	DMD as long-skip regularizer
Causal-rCM (Zheng et al., 24 Jun 2026)	TF-sCM or TF-dCM under causal masks	SF-DMD on autoregressive rollouts

Alice v1 uses a four-stage curriculum on a 14-billion-parameter student derived from a 50-step Wan2.2 teacher. Stage 1 is “Consistency foundation,” lasts 100K steps, uses 480p 3-second synthetic data, targets 8 steps, and optimizes pure sCM with $f_\theta(x,t)$2. Stage 2 lasts 50K steps, progressively scales from 480p to 720p, switches to 4 target steps, and uses full rCM with $f_\theta(x,t)$3. Stage 3 lasts 20K steps at 720p and 5 seconds, mixes 70% synthetic with 30% real physics-heavy curated data, adds LPIPS and VGG-feature perceptual terms, and keeps $f_\theta(x,t)$4. Stage 4 lasts 10K steps on preference pairs, shifts to DPO-style optimization, and reduces the DMD weight to $f_\theta(x,t)$5 while the internal sCM weight is set to $f_\theta(x,t)$6 (Xiaoyu et al., 27 Apr 2026).

The Alice training pipeline makes the score term inseparable from data curation. It generates about $f_\theta(x,t)$7M GPT-4 prompts, samples them with Wan2.2, filters to the top 30% by VBench sub-scores, optical flow checks, CLIP aesthetic scores, and face or physics checks, and then performs hard example mining for failure modes such as physics, hands, faces, and motion coherence. For hard categories, approximately $f_\theta(x,t)$8 additional teacher samples are generated; successful hard examples are oversampled, while failed examples are retained but downweighted to $f_\theta(x,t)$9 sampling probability (Xiaoyu et al., 27 Apr 2026). The paper explicitly attributes Alice’s student-above-teacher behavior to the combination of reverse-KL mode seeking, quality-filtered synthetic data, and consistency-based implicit regularization.

The large-scale image/video rCM paper shifts the emphasis from curriculum to infrastructure. Its central systems contribution is a parallelism-compatible FlashAttention-2 JVP kernel for continuous-time consistency training on models with over 10 billion parameters and high-dimensional video tasks (Zheng et al., 9 Oct 2025). That implementation is paired with FSDP sharding, context parallelism, selective activation checkpointing, BF16-aware time-derivative handling, and teacher/student wrapping into TrigFlow so that the continuous-time sCM objective can be trained on Cosmos-Predict2 and Wan2.1 without abandoning application-scale resolution or video length (Zheng et al., 9 Oct 2025).

Across these implementations, the dominant computational burden is not the score regularizer but the JVP required by the continuous-time consistency term. This is why the literature surrounding rCM treats score regularization and systems design as linked rather than separate problems: once continuous-time consistency is scaled, the reverse-divergence term becomes a practical complement rather than an abstract addition.

4. Empirical performance and reported mechanisms

The empirical claim that most sharply distinguishes rCM from earlier fast-sampling distillation methods is that distillation need not trade quality for speed. Alice v1 reports 5-second 720p videos at 24fps in 4 denoising steps, approximately 8 seconds on an H100, which is a $F_\theta:(x_t,t)\mapsto x_0$0 speedup over its 50-step Wan2.2 teacher, while improving VBench from 84.0 to 91.2. In the same comparison table, Veo3 and Sora2 are listed at approximately 90 and 88 respectively, and the paper reports competitive human-preference performance alongside release of weights, code, synthetic data pipelines, and evaluation scripts (Xiaoyu et al., 27 Apr 2026).

Alice’s ablations are central to the interpretation of rCM. On the paper’s reported VBench comparison, consistency only gives 82.1, score only gives 83.5, and the combined rCM objective gives 91.2 (Xiaoyu et al., 27 Apr 2026). The same paper also reports that moving from unfiltered synthetic data to quality filtering and then to hard example mining increases VBench from 86.3 to 88.7 and then to 91.2, while the Physics subscore rises from 0.76 to 0.80 and then to 0.86 (Xiaoyu et al., 27 Apr 2026). These results are presented as evidence that the forward component preserves coverage, the reverse component sharpens high-quality modes, and curated hard examples provide targeted gradients in teacher failure regimes.

The large-scale rCM paper reports a different but related empirical pattern. On large text-to-image and text-to-video teachers, pure sCM shows systematic deficits in fine-detail generation, text rendering, and temporal coherence, and the paper attributes these deficits to error accumulation and the mode-covering character of the forward-divergence objective (Zheng et al., 9 Oct 2025). Adding the DMD regularizer yields 1-step to 4-step students that match or surpass DMD2 on quality metrics while preserving more diversity. For example, on Wan2.1 T2V, the 14B teacher is reported at VBench 83.6, while rCM reaches 84.9 in 4 steps and 85.1 in 2 steps; on Cosmos-Predict2 T2I, the 14B teacher is reported at GenEval overall 0.84, while rCM reaches 0.83 in 4 steps and 0.82 in 1 step (Zheng et al., 9 Oct 2025). The same study reports throughput improvements from roughly 0.7 FPS for the 1.3B Wan teacher to 14.6 FPS, 23 FPS, and 32.3 FPS for 4-step, 2-step, and 1-step rCM students, respectively (Zheng et al., 9 Oct 2025).

A recurrent empirical theme is that the reverse-divergence component improves fidelity without fully sacrificing diversity. The large-scale paper explicitly contrasts rCM with DMD2 and states that rCM offers notable advantages in diversity while matching or surpassing DMD2 on quality metrics, and that it does so without GAN tuning or extensive hyperparameter searches (Zheng et al., 9 Oct 2025). This suggests that the forward term is doing more than stabilizing optimization; it is actively preserving a broader subset of teacher modes than a purely reverse-divergence student would preserve.

5. Autoregressive extension: Causal-rCM

Causal-rCM extends the rCM philosophy from bidirectional diffusion distillation to autoregressive video diffusion with causal diffusion transformers. Its central identification is structural: in the autoregressive setting, teacher forcing supplies the offline, forward-divergence counterpart of consistency training, while self forcing supplies the on-policy, reverse-divergence counterpart of DMD. The paper therefore maps the original rCM decomposition into TF-CM and SF-DMD (Zheng et al., 24 Jun 2026).

Teacher-forcing CM is implemented in both discrete-time and continuous-time forms. The continuous-time variant, TF-sCM, is the paper’s first implementation of teacher-forcing-based continuous-time CMs for autoregressive video diffusion, enabled by a custom-mask FlashAttention-2 JVP kernel that supports causal masking, FSDP2, context parallelism, and long sequences (Zheng et al., 24 Jun 2026). Under RF parameterization, the paper defines a teacher-forcing tangent along the causal teacher PF-ODE and uses the normalized continuous-time CM loss

$F_\theta:(x_t,t)\mapsto x_0$1

where $F_\theta:(x_t,t)\mapsto x_0$2 contains the teacher velocity, the EMA student velocity, and a JVP term through the causal masked transformer (Zheng et al., 24 Jun 2026). This is the causal analogue of the score-regularized tangent normalization used in non-autoregressive rCM.

The reported reason for using TF-sCM is not merely formal symmetry with the original rCM. The paper states that TF-sCM converges approximately $F_\theta:(x_t,t)\mapsto x_0$3 faster than TF-dCM as an initializer, reaching strong VBench performance in roughly $F_\theta:(x_t,t)\mapsto x_0$4k iterations versus $F_\theta:(x_t,t)\mapsto x_0$5k for the discrete alternative (Zheng et al., 24 Jun 2026). After that initialization, SF-DMD refines the few-step student on self-generated autoregressive rollouts with KV caches, truncated backpropagation, and DMD loss on generated videos.

The headline result is that a distilled 2-step causal Wan2.1-1.3B model reaches a VBench-T2V score of 84.63 with only 1 or 2 sampling steps (Zheng et al., 24 Jun 2026). In the same study, Causal-rCM is reported as state-of-the-art in both frame-wise and chunk-wise streaming settings, and the framework is further applied to Cosmos 3 to obtain an action-conditioned interactive world model (Zheng et al., 24 Jun 2026). The paper also reports real-time throughput figures for the causal student, including 8.3 FPS and 17.4 FPS for 4-step frame-wise and chunk-wise settings, and up to 25.6 FPS in 1-step or 2-step noisy-context chunk-wise settings (Zheng et al., 24 Jun 2026).

A notable negative result is that joint CM+DMD training lowered VBench ceilings in the causal setting, whereas a staged pipeline—causal teacher training, TF-CM distillation, then SF-DMD refinement—worked best (Zheng et al., 24 Jun 2026). This is important because it indicates that the forward-plus-reverse decomposition is robust as a conceptual principle, but not yet universally robust as a single-stage optimizer.

6. Relations, limitations, and nomenclature

rCM is best understood as one member of a broader family of methods that mix consistency constraints with score-based guidance, but the name should not be generalized indiscriminately. Guided Consistency Sampling, SegmentDreamer’s SCTD, and Collaborative Score Distillation all exhibit explicit or implicit complementarities between consistency-style transport constraints and score-driven corrections, yet they are not presented under the rCM name (Li et al., 2024, Zhu et al., 7 Jul 2025, Kim et al., 2023). This suggests a family resemblance rather than terminological identity.

Several limitations recur across the reported rCM literature. Alice v1 states that long-horizon physics, deformable bodies, multi-body interactions, text rendering, and audio remain difficult, and it explicitly notes that rCM does not invent fundamentally new physics but refines what the teacher and curated data already contain (Xiaoyu et al., 27 Apr 2026). The large-scale image/video paper identifies fine-detail degradation in pure sCM and traces it to error accumulation and the mode-covering nature of the forward-divergence objective, which is precisely why the score regularizer is introduced (Zheng et al., 9 Oct 2025). Causal-rCM reports fragility in frame-wise deep rollouts, a mismatch between the best pre-SF initializer and the best final SF-refined model, and an unresolved tension between staged and joint optimization (Zheng et al., 24 Jun 2026).

Two recurring misconceptions are addressed directly by the literature. The first is that consistency distillation is necessarily a speed-quality tradeoff. Alice v1 explicitly argues the opposite and attributes student-above-teacher performance to the combination of mode-seeking score regularization, targeted synthetic data with hard example mining, and consistency-based implicit regularization that removes “lucky path” dependence on specific noise samples (Xiaoyu et al., 27 Apr 2026). The second is terminological: RCM is also the acronym of the Riemannian Consistency Model, a distinct framework for few-step generation on manifolds such as spheres, flat tori, and $F_\theta:(x_t,t)\mapsto x_0$6. That paper uses RCD and RCT for Riemannian consistency distillation and training, and its acronym should not be conflated with score-regularized consistency distillation in Euclidean image or video models (Cheng et al., 1 Oct 2025).

In its current form, rCM is therefore not a single fixed loss but a design pattern with a stable core: a continuous- or discrete-time consistency objective provides forward, coverage-preserving structure, and a student-sample-based score objective provides reverse, mode-seeking correction. The available evidence indicates that this combination is especially effective when paired with curriculum design, careful data curation, and JVP-capable systems infrastructure, and that it remains adaptable enough to span non-autoregressive video generation, large-scale image/video diffusion distillation, and causal world-model training (Xiaoyu et al., 27 Apr 2026, Zheng et al., 9 Oct 2025, Zheng et al., 24 Jun 2026).