ReVal: Value-Based RL for Model Fine-Tuning

• ReVal is a novel value-based reinforcement learning framework that reframes model fine-tuning by learning dense rewards via Bellman updates, overcoming sparse reward limitations.
• It employs dual objectives with value function training through reward-to-go regression and KL-regularized fine-tuning to maintain fidelity to powerful pretrained models.
• Empirical evaluations demonstrate significant improvements in sample efficiency and stability across domains such as diffusion models and large language models.

ReVal (Value-Based RL) comprises a family of recent value-based reinforcement learning algorithms advancing sample-efficient, scalable, and theoretically sound fine-tuning for high-capacity generative models—most notably diffusion models and LLMs. In contrast to dominant on-policy, policy-gradient-trained paradigms, ReVal methods recast the task as value (or Q-function) learning via Bellman updates and shape learning signals throughout the generation process. This addresses traditional limitations of RL for model fine-tuning, particularly sparse or non-differentiable terminal rewards, trajectory-level credit assignment, poor sample efficiency, and the challenge of maintaining fidelity to powerful pretrained networks. This article surveys major instantiations of the ReVal framework, with a technical focus on its mathematical formulation, algorithmic objectives, learning workflow, treatment of non-differentiable rewards, empirical findings, and implementation best practices.

1. Formulation: Value-Based RL for Model Fine-Tuning

The core of ReVal is the reinterpretation of conditional generative model sampling (e.g., backward diffusion, stepwise LLM token prediction) as trajectory optimization in an MDP:

• Diffusion models: The reverse diffusion process $\pi_{\theta_0}(x_{t-1} \mid x_t, c)$ is represented as a Markov decision process $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \rho_0, \mathcal{P}, \mathcal{R})$$ for $t=0, ..., T$, with state $s_t = (x_t, c)$, action $a_t = x_{t-1}$, and reward $\mathcal{R}(s_t, a_t)$ typically nonzero only at $t=0$ (Dai et al., 21 May 2025).
• LLMs: Autoregressive token generation is modeled as Q-learning with the policy’s logits encoding $Q_\theta(s_h, a_h)$, where $s_h$ is the history and $a_h$ the token at step $h$ (Wang et al., 24 Mar 2026).

The ReVal approach then replaces the sparse/singular reward with a dense, predictive value function $V_\phi$, and uses Bellman update logic to inform learning at every step.

2. Learning Objectives and Optimization

ReVal algorithms comprise two principal learning objectives:

• Value Function Training (Reward-to-Go Regression):

$$\mathcal{L}_{\text{value}}(\phi) = \mathbb{E}_{c, t, x_{0:T}}\Bigl[V_\phi(x_t, c) - r(x_0, c)\Bigr]^2$$

Here $V_\phi(x_t, c)$ approximates the expected total reward from the intermediate state, facilitating dense supervision even in the presence of sparse or non-differentiable task rewards (Dai et al., 21 May 2025).

• Main RL Objective (KL-Regularized Value-Guided Fine-Tuning):

$$\mathcal{L}_{\rm VARD}(\theta) = \mathbb{E}_{c, t, (x_t, x^0_t)}\Bigl[ -V_\phi(x_t, c) + \eta\,\|x_t - x^0_t\|^2\Bigr]$$

The penalty term $\|x_t - x^0_t\|^2$ (equivalent to $D_{\rm KL}(\pi_\theta \| \pi_{\theta_0})$ up to scale) constrains policy drift from the pretrained model, ensuring prior similarity and training stability (Dai et al., 21 May 2025).

For LLMs, the Bellman-residual loss is minimized:

$$\mathcal{L}_{\rm ReVal}(\theta) = \frac{1}{|\mathcal{D}|} \sum_{\tau\in\mathcal{D}} \sum_{h=1}^H [ Q_\theta(s_h,a_h) - (T_B Q_{\theta^-})(s_h, a_h) ]^2$$

with $T_B Q$ a soft Bellman operator incorporating shaped trajectory-level and stepwise rewards (Wang et al., 24 Mar 2026).

3. Algorithmic Workflow

The ReVal fine-tuning loop decomposes into:

• Value Pretraining: Freeze base model, collect MDP trajectories, and regress the value network $V_\phi$ against observed terminal rewards, using all intermediate states in each trajectory.
• KL-Regularized Fine-Tuning: Unfreeze the main model (e.g., a diffusion U-Net or LLM), and optimize parameters to maximize the value function at randomly sampled intermediate states, regularizing proximity to the prior model.
• Continuous Value Network Refresh: Periodically update $V_\phi$ using new trajectories sampled under the fine-tuned model, ensuring value consistency as the policy evolves (Dai et al., 21 May 2025).
• Replay Buffer and Off-Policy Updates (LLMs): Store and replay complete trajectories, regularly resampling batches for Bellman-residual minimization, and periodically resetting the target/reference network. This off-policy learning decouples sample collection from parameter updates, optimizing data reuse (Wang et al., 24 Mar 2026).

ReVal/VARD Workflow Outline (Diffusion)

for iterations: c ~ p(c) x_{0:T} ~ π_{θ0}(·|c) r = r(x0, c) for t in 0..T: regress V_φ(xt, c) to r update φ on ℒ_value for iterations: c ~ p(c); t ~ U[0,T] (xt, x0_t) ~ (π_θ, π_{θ0})(·|xt+1, c) compute loss ℓ = -V_φ(xt, c) + η ||xt - x0_t||^2 update θ; fine-tune φ as needed

(Dai et al., 21 May 2025)

ReVal Workflow Outline (LLMs)

initialize buffer B = ∅; θ ← θ_ref for t in 1..T: D_new ← on-policy rollouts from θ B ← B ∪ D_new; (evict old if |B| > M) for k in 1..K: D_replay ← sample batch from B update θ with gradient of ℒ_ReVal(D_replay) every N steps: reset θ^- ← θ

(Wang et al., 24 Mar 2026)

4. Handling Sparse and Non-Differentiable Rewards

ReVal methods address the inefficiency of direct policy-gradient RL with non-differentiable/sparse rewards by:

• Monte Carlo Regression on the Value Function: Instead of propagating gradients through the reward function, the value function $V_\phi$ is regressed against final outcomes using sampled trajectories, even when rewards are non-differentiable (e.g., human preference, discrete class labels).
• Backpropagation-Compatible Dense Signal: Since $V_\phi$ is differentiable with respect to intermediate states, gradients with respect to model parameters can be propagated through $-V_\phi(x_t)$, constructing a training signal accessible at every diffusion or generation step, independent of the original reward's differentiability (Dai et al., 21 May 2025).

This approach yields smoother optimization, reduced variance in updates, and enables RL-style fine-tuning for a broader class of objectives, including those inaccessible to policy gradients alone.

5. Empirical Evaluation and Comparative Findings

ReVal methods deliver consistent empirical improvements across domains:

• Diffusion Model Fine-Tuning (Dai et al., 21 May 2025):
  • On protein secondary structure, FoldFlow-2-VARD increases target β-sheet content beyond baselines.
  • For JPEG (in)compressibility, VARD successfully manipulates image statistics to match the reward signal.
  • On human-preference score tasks (Aesthetic, PickScore, ImageReward), VARD attains higher scores and fewer reward hacks versus ReFL and DRaFT-1; only KL-regularized VARD maintains prompt fidelity and prior similarity.
  • Generalization is validated on unseen HPDv2 prompts and cross-reward settings, with superior FID-reward tradeoffs.
• LLM Post-Training (Wang et al., 24 Mar 2026):
  • Achieves up to 5.2x speedup over GRPO for mathematical reasoning.
  • Outperforms GRPO by +2.7% (AIME24) and +4.5% (GPQA) on DeepSeek-R1-Distill-1.5B; similar gains on Qwen 2.5-Math-7B.
  • Increased efficiency is most pronounced in low-rollout regimes, with an 18% reduction in wall-clock time to SOTA.
  • Ablation studies link performance to reference reset frequency, KL regularization scale, and reward design (normalized advantage performs best).

6. Implementation Details and Extensions

The following outlines major hyperparameter and architectural settings:

Domain	Value Model	Fine-tune Steps	$\eta$ (KL weight)	Base-model LR	Value LR	GPUs	Time/run
Proteins	GVP-Transformer	all 50	0.1	1e-6–5e-5	5e-6–1e-6	8×H800	30 min
Compression	ResNet-18	last 10	1	1e-6–5e-5	5e-6–1e-6	8×H800	30 min
Aesthetic	ViT+LoRA	last 10	100	1e-6–5e-5	5e-6–1e-6	8×H800	30 min

• Fine-tuning most often targets the final steps of diffusion, optimizing compute.
• Off-policy buffers in LLM ReVal benefit from large capacity (e.g., 5120), with batch sizes 1024, and off-policy updates per iteration (typically $K=2$).
• Reference model (for KL/policy shaping) is periodically reset, typically every 200 steps.
• VARD supports extension to vector-valued value functions (multi-objective RL), scaling to larger backbones or continuous-time generators, and hybridization with adversarial/classifier guidance (Dai et al., 21 May 2025).

7. Implications and Open Directions

ReVal-type algorithms concretely demonstrate:

• Unified value-policy training: By parameterizing value and policy in a single model (especially at LLM scale), memory and computation overhead are minimized, and off-policy learning is feasible even with large, long-horizon or expensive-to-sample models.
• Scalability: The off-policy, replay-enhanced architecture supports high data reuse, critical for long sequences or expensive environments.
• Limitations: Further research is needed into more advanced replay (e.g., prioritized schemes), stability in highly non-stationary settings, and better integration of “internal consistency” shaping for complex generative domains.
• Contrast with Related Work: While some contemporaneous proposals (e.g., VSPO (Zhuang et al., 8 Dec 2025)) share the principle of using value estimates to guide training (sometimes via empirical batch statistics rather than explicit critics), classical ReVal methods uniquely pair critic-predicted value functions with explicit RL objectives and Bellman-consistent updates, yielding a dense, stable fine-tuning signal (Dai et al., 21 May 2025, Wang et al., 24 Mar 2026).

ReVal represents a robust and extensible paradigm for value-based model fine-tuning, with empirical and theoretical support for applications in diffusion models, autoregressive LLMs, and beyond.