---
title: 'ReVal: Value-Based RL for Model Fine-Tuning'
url: https://www.emergentmind.com/topics/reval-value-based-rl
type: topic
---

# ReVal: Value-Based RL for Model Fine-Tuning

ReVal (Value-Based RL) comprises a family of recent value-based reinforcement learning algorithms advancing sample-efficient, scalable, and theoretically sound fine-tuning for high-capacity generative models—most notably diffusion models and large language models (LLMs). In contrast to dominant on-policy, policy-gradient-trained paradigms, ReVal methods recast the task as value (or Q-function) learning via Bellman updates and shape learning signals throughout the generation process. This addresses traditional limitations of RL for model fine-tuning, particularly sparse or non-differentiable terminal rewards, trajectory-level credit assignment, poor sample efficiency, and the challenge of maintaining fidelity to powerful pretrained networks. This article surveys major instantiations of the ReVal framework, with a technical focus on its mathematical formulation, algorithmic objectives, learning workflow, treatment of non-differentiable rewards, empirical findings, and implementation best practices.

## 1. Formulation: Value-Based RL for Model Fine-Tuning

The core of ReVal is the reinterpretation of conditional generative model sampling (e.g., backward diffusion, stepwise LLM token prediction) as trajectory optimization in an MDP:

- **Diffusion models:** The reverse diffusion process $\pi_{\theta_0}(x_{t-1} \mid x_t, c)$ is represented as a Markov decision process $ \mathcal{M} = (\mathcal{S}, \mathcal{A}, \rho_0, \mathcal{P}, \mathcal{R}) $ for $ t=0, ..., T $, with state $s_t = (x_t, c)$, action $a_t = x_{t-1}$, and reward $\mathcal{R}(s_t, a_t)$ typically nonzero only at $t=0$ [2505.15791].

- **LLMs:** Autoregressive token generation is modeled as Q-learning with the policy’s logits encoding $Q_\theta(s_h, a_h)$, where $s_h$ is the history and $a_h$ the token at step $h$ [2603.23355].

The ReVal approach then replaces the sparse/singular reward with a dense, predictive value function $V_\phi$, and uses Bellman update logic to inform learning at every step.

## 2. Learning Objectives and Optimization

ReVal algorithms comprise two principal learning objectives:

- **Value Function Training (Reward-to-Go Regression):**
  $$
  \mathcal{L}_{\text{value}}(\phi) = \mathbb{E}_{c, t, x_{0:T}}\Bigl[V_\phi(x_t, c) - r(x_0, c)\Bigr]^2
  $$
  Here $V_\phi(x_t, c)$ approximates the expected total reward from the intermediate state, facilitating dense supervision even in the presence of sparse or non-differentiable task rewards [2505.15791].

- **Main RL Objective (KL-Regularized Value-Guided Fine-Tuning):**
  $$
  \mathcal{L}_{\rm VARD}(\theta)
    = \mathbb{E}_{c, t, (x_t, x^0_t)}\Bigl[ -V_\phi(x_t, c) + \eta\,\|x_t - x^0_t\|^2\Bigr]
  $$
  The penalty term $\|x_t - x^0_t\|^2$ (equivalent to $D_{\rm KL}(\pi_\theta \| \pi_{\theta_0})$ up to scale) constrains policy drift from the pretrained model, ensuring prior similarity and training stability [2505.15791].

For LLMs, the Bellman-residual loss is minimized:
$$
\mathcal{L}_{\rm ReVal}(\theta) = \frac{1}{|\mathcal{D}|} \sum_{\tau\in\mathcal{D}} \sum_{h=1}^H [ Q_\theta(s_h,a_h) - (T_B Q_{\theta^-})(s_h, a_h) ]^2
$$
with $T_B Q$ a soft Bellman operator incorporating shaped trajectory-level and stepwise rewards [2603.23355].

## 3. Algorithmic Workflow

The ReVal fine-tuning loop decomposes into:

- **Value Pretraining:** Freeze base model, collect MDP trajectories, and regress the value network $V_\phi$ against observed terminal rewards, using all intermediate states in each trajectory.

- **KL-Regularized Fine-Tuning:** Unfreeze the main model (e.g., a diffusion U-Net or LLM), and optimize parameters to maximize the value function at randomly sampled intermediate states, regularizing proximity to the prior model.

- **Continuous Value Network Refresh:** Periodically update $V_\phi$ using new trajectories sampled under the fine-tuned model, ensuring value consistency as the policy evolves [2505.15791].

- **Replay Buffer and Off-Policy Updates (LLMs):** Store and replay complete trajectories, regularly resampling batches for Bellman-residual minimization, and periodically resetting the target/reference network. This off-policy learning decouples sample collection from parameter updates, optimizing data reuse [2603.23355].

### ReVal/VARD Workflow Outline (Diffusion)
```python
# Algorithm 1: Value pretraining
for iterations:
    c ~ p(c)
    x_{0:T} ~ π_{θ0}(·|c)
    r = r(x0, c)
    for t in 0..T:
        regress V_φ(xt, c) to r
    update φ on ℒ_value

# Algorithm 2: Fine-tuning
for iterations:
    c ~ p(c); t ~ U[0,T]
    (xt, x0_t) ~ (π_θ, π_{θ0})(·|xt+1, c)
    compute loss ℓ = -V_φ(xt, c) + η ||xt - x0_t||^2
    update θ; fine-tune φ as needed
```
[2505.15791]

### ReVal Workflow Outline (LLMs)
```python
# Replay Buffer
initialize buffer B = ∅; θ ← θ_ref
for t in 1..T:
    D_new ← on-policy rollouts from θ
    B ← B ∪ D_new; (evict old if |B| > M)
    for k in 1..K:
        D_replay ← sample batch from B
        update θ with gradient of ℒ_ReVal(D_replay)
    every N steps: reset θ^- ← θ
```
[2603.23355]

## 4. Handling Sparse and Non-Differentiable Rewards

ReVal methods address the inefficiency of direct policy-gradient RL with non-differentiable/sparse rewards by:

- **Monte Carlo Regression on the Value Function:** Instead of propagating gradients through the reward function, the value function $V_\phi$ is regressed against final outcomes using sampled trajectories, even when rewards are non-differentiable (e.g., human preference, discrete class labels).

- **Backpropagation-Compatible Dense Signal:** Since $V_\phi$ is differentiable with respect to intermediate states, gradients with respect to model parameters can be propagated through $-V_\phi(x_t)$, constructing a training signal accessible at every diffusion or generation step, independent of the original reward's differentiability [2505.15791].

This approach yields smoother optimization, reduced variance in updates, and enables RL-style fine-tuning for a broader class of objectives, including those inaccessible to policy gradients alone.

## 5. Empirical Evaluation and Comparative Findings

ReVal methods deliver consistent empirical improvements across domains:

- **Diffusion Model Fine-Tuning [2505.15791]:**
  - On protein secondary structure, FoldFlow-2-VARD increases target β-sheet content beyond baselines.
  - For JPEG (in)compressibility, VARD successfully manipulates image statistics to match the reward signal.
  - On human-preference score tasks (Aesthetic, PickScore, ImageReward), VARD attains higher scores and fewer reward hacks versus ReFL and DRaFT-1; only KL-regularized VARD maintains prompt fidelity and prior similarity.
  - Generalization is validated on unseen HPDv2 prompts and cross-reward settings, with superior FID-reward tradeoffs.

- **LLM Post-Training [2603.23355]:**
  - Achieves up to 5.2x speedup over GRPO for mathematical reasoning.
  - Outperforms GRPO by +2.7% (AIME24) and +4.5% (GPQA) on DeepSeek-R1-Distill-1.5B; similar gains on Qwen 2.5-Math-7B.
  - Increased efficiency is most pronounced in low-rollout regimes, with an 18% reduction in wall-clock time to SOTA.
  - Ablation studies link performance to reference reset frequency, KL regularization scale, and reward design (normalized advantage performs best).

## 6. Implementation Details and Extensions

The following outlines major hyperparameter and architectural settings:

| Domain        | Value Model                   | Fine-tune Steps | $\eta$ (KL weight) | Base-model LR  | Value LR     | GPUs           | Time/run |
|---------------|------------------------------|-----------------|--------------------|---------------|--------------|----------------|----------|
| Proteins      | GVP-Transformer              | all 50          | 0.1                | 1e-6–5e-5     | 5e-6–1e-6    | 8×H800         | 30 min   |
| Compression   | ResNet-18                    | last 10         | 1                  | 1e-6–5e-5     | 5e-6–1e-6    | 8×H800         | 30 min   |
| Aesthetic     | ViT+LoRA                     | last 10         | 100                | 1e-6–5e-5     | 5e-6–1e-6    | 8×H800         | 30 min   |

- Fine-tuning most often targets the final steps of diffusion, optimizing compute.
- Off-policy buffers in LLM ReVal benefit from large capacity (e.g., 5120), with batch sizes 1024, and off-policy updates per iteration (typically $K=2$).
- Reference model (for KL/policy shaping) is periodically reset, typically every 200 steps.
- VARD supports extension to vector-valued value functions (multi-objective RL), scaling to larger backbones or continuous-time generators, and hybridization with adversarial/classifier guidance [2505.15791].

## 7. Implications and Open Directions

ReVal-type algorithms concretely demonstrate:

- **Unified value-policy training:** By parameterizing value and policy in a single model (especially at LLM scale), memory and computation overhead are minimized, and off-policy learning is feasible even with large, long-horizon or expensive-to-sample models.
  
- **Scalability:** The off-policy, replay-enhanced architecture supports high data reuse, critical for long sequences or expensive environments.

- **Limitations:** Further research is needed into more advanced replay (e.g., prioritized schemes), stability in highly non-stationary settings, and better integration of “internal consistency” shaping for complex generative domains.

- **Contrast with Related Work:** While some contemporaneous proposals (e.g., VSPO [2512.07478]) share the principle of using value estimates to guide training (sometimes via empirical batch statistics rather than explicit critics), classical ReVal methods uniquely pair critic-predicted value functions with explicit RL objectives and Bellman-consistent updates, yielding a dense, stable fine-tuning signal [2505.15791, 2603.23355].

ReVal represents a robust and extensible paradigm for value-based model fine-tuning, with empirical and theoretical support for applications in diffusion models, autoregressive LLMs, and beyond.

Source: https://www.emergentmind.com/topics/reval-value-based-rl