Reward Guided Latent Consistency Distillation

• RG-LCD is a framework that augments latent consistency distillation with reward maximization, enabling accelerated sampling while preserving or enhancing quality.
• It integrates latent proxy reward models to handle differentiable and non-differentiable rewards, ensuring preference alignment and semantic fidelity across multiple modalities.
• Empirical results demonstrate significant speedups and state-of-the-art performance in text-to-image, text-to-video, biomolecular design, and offline reinforcement learning tasks.

Reward Guided Latent Consistency Distillation (RG-LCD) is a framework for aligning the fast generative capabilities of consistency-distilled diffusion models with user- or task-defined reward functions. By interleaving latent consistency distillation with reward maximization, RG-LCD enables accelerated sampling (often in one or a few steps) while retaining or even exceeding the quality, preference alignment, and semantic fidelity provided by the original, multi-step diffusion teacher. The framework is task- and modality-agnostic, supporting image, video, protein sequence, molecular, and reinforcement learning domains, and is compatible with both differentiable and non-differentiable reward models through latent reward proxies.

1. Theoretical Foundations

RG-LCD builds upon the foundation of Latent Consistency Distillation (LCD), where a student latent consistency model (LCM) is trained to match the output of a multi-step teacher latent diffusion model (LDM) in only a few inference steps. LCD leverages a loss between the student's single- or few-step prediction and reference multi-step outputs from the teacher or an EMA copy, typically using the Huber or MSE distance in latent space. For instance, for text-to-image synthesis, LCD matches $f_\theta(z_{t_{n+k}}, \omega, c, t_{n+k})$ to a teacher-generated target at timestep $t_n$ using a designated solver and schedule (Li et al., 2024).

The core innovation of RG-LCD is to augment the standard LCD loss with an explicit reward maximization term:

$$\mathcal{L}_{\text{RG-LCD}}(\theta, \theta^-) = \mathcal{L}_{\text{LCD}}(\theta, \theta^-) - \beta J(\theta)$$

where $J(\theta)$ denotes the expected reward, $\mathcal{R}(\cdot, c)$, evaluated on the decoded student output and $\beta$ is a tradeoff hyperparameter (Li et al., 2024).

This paradigm is extended to cover broader cases (offline RL, biomolecular generation, and video synthesis) by using trajectory-level, sequence-level, or latent reward models, and by supporting non-differentiable reward functions through reward model surrogates (Su et al., 1 Jul 2025, Ding et al., 2024, Duan et al., 9 Jun 2025).

2. Algorithmic Structure

RG-LCD training consists of a tight coupling between consistency-based supervision and reward optimization. The pipeline, as instantiated for text-to-image, is as follows (Li et al., 2024):

1. Latent Consistency Supervision:
   • For a latent $z_{t_{n+k}}$, the student predicts $z_0$ in one step; the teacher provides a reference $z_{t_n}$ via a multi-step solver.
   • Loss $\mathcal{L}_{\text{LCD}}$ penalizes the discrepancy between student and teacher outputs.
2. Reward Maximization:
   • The student prediction is decoded to RGB and evaluated under a reward model, which may encode preference alignment, photorealism, or domain-specific criteria.
   • For non-differentiable or high-cost reward models, a latent reward model (LRM) is trained as a regressor or via preference-based loss.
3. Joint Optimization:
   • The total objective is a linear combination of LCD loss and the (potentially proxy) reward, with hyperparameter weighting.
   • Training may alternate updates of the LRM and the student network, as in text-to-video (Ding et al., 2024).
4. Evaluation and Refinement:
   • Fast inference is enabled by minimal step count; results are assessed by preference metrics (human or algorithmic), as well as statistical measures (FID, HPS, task-specific reward).

This procedure generalizes to sequence, trajectory, and other high-dimensional spaces inherent to RL, biomolecular, or video tasks, often using the RL trajectory reward or simulated soft-optimal rollouts as in (Duan et al., 9 Jun 2025, Su et al., 1 Jul 2025).

3. Reward Model Integration and Latent Proxy Models

Direct optimization toward powerful reward models (e.g., CLIPScore, HPSv2, ImageReward) can induce overfitting and high-frequency artifacts ("reward hacking"), especially for models with large receptive fields trained on resized samples. To mitigate this, RG-LCD introduces the Latent Proxy Reward Model (LRM) (Li et al., 2024, Ding et al., 2024):

• Training the LRM: The LRM is learned to regress or classify the outputs of a target reward model based on either regression or contrastive alignment in latent space, using real and generated samples.
• Preference Alignment: The LRM is optionally trained to match pairwise or triplet preferences (via KL divergence on softmaxed reward) between the latent predictions and the expert reward's judgments.

Once the LRM is aligned, gradients of the reward with respect to the student latents can be computed efficiently. This enables both differentiable and non-differentiable reward functions, and empirically suppresses noise and artifacts in the outputs (Li et al., 2024, Ding et al., 2024).

4. Domain-Specific Extensions

Text-to-Image Synthesis (Li et al., 2024)

• RG-LCD enables 2-step or 4-step synthesis matching or surpassing human preference rates for teacher LDMs using 50-step DDIM.
• Automatic metrics such as HPSv2.1 and FID on MS-COCO confirm superior performance compared to standard LCD and baseline LCMs.

Text-to-Video Generation (Ding et al., 2024)

• RG-LCD (as implemented in DOLLAR) combines Variational Score Distillation (VSD), Consistency Distillation (CD), and LRM-fine-tuning.
• Students match or exceed the teacher’s quality on VBench, achieve $15.6\times$–$278.6\times$ inference acceleration (1- to 4-step sampling), and are robust to non-differentiable reward models.

Biomolecular and Sequence Design (Su et al., 1 Jul 2025)

• RG-LCD is generalized via iterative policy distillation and value-weighted maximum likelihood over reward-based soft-optimal posteriors, supporting arbitrary, non-differentiable reward functions.
• Outperforms both RL and best-of-N baselines in protein (SS-match, globularity), DNA enhancer, and molecule docking tasks.

Offline RL and Trajectory Planning (Duan et al., 9 Jun 2025)

• Reward-Aware Consistency Trajectory Distillation (RACTD) applies the RG-LCD paradigm to trajectory diffusion models for RL, achieving 8.7% improvement over SOTA with up to $142\times$ inference speedup.
• Pseudocode and loss structure closely follow the combination of consistency-based and reward-based terms as in general RG-LCD.

5. Empirical Results and Comparative Analysis

RG-LCD provides state-of-the-art acceleration and preference alignment across modalities. Key results include:

• Text-to-image (HPSv2.1 on PartiPrompt): 2-step RG-LCM (HPS) achieves higher human preference than the teacher SD v2.1 with 50 DDIM steps (62.1% vs 37.9%) (Li et al., 2024).
• Text-to-video (VBench, HPSv2): RG-LCD + HPSv2 delivers total scores 82.57, surpassing the teacher’s 80.25, with $>15\times$–$278\times$ inference acceleration (Ding et al., 2024).
• Offline RL (MuJoCo, Maze2d): RACTD achieves an average of 97.6 on MuJoCo (vs 89.3 for Diffusion QL), using single-step inference (Duan et al., 9 Jun 2025).
• Biological/macromolecule design: VIDD (RG-LCD) achieves best-in-class median rewards across protein, DNA, and docking tasks, exceeding previous baseline methods (Su et al., 1 Jul 2025).

Domain	Acceleration	Human/Task Alignment	Notable Metrics/Score	Reference
Text-to-image	$25\times$	Outperforms teacher	HPSv2.1: 30.9 (2-step HPS)	(Li et al., 2024)
Text-to-video	$15$–$278\times$	Exceeds teacher	VBench: 82.57 (4-step HPSv2)	(Ding et al., 2024)
RL, trajectories	$20$–$142\times$	$+8.7\%$ rel. gain	MuJoCo: 97.6 (RACTD)	(Duan et al., 9 Jun 2025)
Biomolecule	N/A	Best among baselines	Protein SS-match: 0.69	(Su et al., 1 Jul 2025)

Quantitative results consistently show that reward-guided terms compensate for the sample quality lost by aggressive step reduction in distillation.

6. Limitations and Failure Modes

Empirical analysis reveals several caveats:

• Over-optimization toward imperfect reward models can degrade text alignment and generate artifacts, especially with large $\beta$.
• Most vision reward models are limited to low-resolution assessment, missing fine-scale high-frequency artifacts that can arise in the samples (Li et al., 2024).
• RG-LCD presumes access to a differentiable RM or an accurate LRM; training surrogate reward models remains nontrivial for some tasks.
• Consistency distillation objectives and reward-guided optimization can introduce loss-surface instability.

Practical deployment may require adaptive reward weighting, architectural modifications for higher resolution, or more advanced surrogate models for complex, non-differentiable metrics (Li et al., 2024, Duan et al., 9 Jun 2025).

7. Prospects and Future Research

Ongoing research in RG-LCD seeks to:

• Develop high-resolution, domain-robust reward models, possibly eschewing rigid 224×224 downsampling.
• Extend RG-LCD to novel domains including multi-modal (3D, audio) generation or transformer-based models.
• Enable more stable and adaptive reward integration via schedule or uncertainty-based weighting.
• Apply RG-LCD to multi-task, meta-learning, and settings with weak, noisy, or adversarial reward supervision.

Advances in proxy reward modeling, consistency objectives, and scalable distillation architectures remain pivotal for achieving the full promise of RG-LCD across increasingly complex and high-dimensional generative domains.