Decoupled Policy and Value Model Optimization

• Decoupled Policy and Value Model Optimization is a reinforcement learning paradigm that separates the training of policies (actors) from value estimation (critics) to reduce instability and bias.
• It employs strategies such as frozen value models, token-level decoupling, and distinct advantage calculations to enable robust policy improvements and efficient inference.
• Empirical results demonstrate enhanced performance, reduced training time, and improved generalization in long-horizon language modeling, offline RL, and transfer learning tasks.

Decoupled Policy and Value Model Optimization refers to a family of reinforcement learning (RL) and RLHF techniques that explicitly separate (decouple) the training or application of the policy (“actor”) from the value function (“critic”)—as opposed to standard on-policy or actor-critic approaches, which tightly interleave the optimization of both. This paradigm aims to address instability, sample inefficiency, representation entanglement, bias, and poor generalization arising from actor–critic interdependence, especially in domains such as long-horizon language modeling, offline RL, and transfer learning. Below, key methodological innovations, theoretical rationales, and empirical outcomes are detailed across foundational and recent approaches.

1. Core Formulations and Theoretical Rationale

Decoupling can take several forms: training value and policy networks on disjoint data, freezing the value model during policy optimization, or introducing a third model (e.g., a generalist or global value predictor) that replaces shared optimization.

Fundamental innovations include:

• Separate Representation Learning: Policy $\pi_\theta(a|s)$ and value $V_\phi(s)$ or $Q_\phi(s,a)$ networks with non-shared parameters and/or encoders (as in IDAAC (Raileanu et al., 2021), IVG (Byravan et al., 2019)).
• Specialized Advantage Calculation: Replace standard advantage $\hat{A}(s_t, a_t) = G_t - V_\phi(s_t)$ with methods that downweight, clip, or recalibrate advantages independently for different token/action subgroups (DEPO (Tan et al., 17 Oct 2025), VC-PPO (Yuan et al., 3 Mar 2025)).
• Global or Generalist Value Models: Utilize a frozen global value model (GVM, as in DVPO (Huang et al., 24 Feb 2025)) or a context-aware value baseline ($V_0$ (Zhang et al., 3 Feb 2026)) for all policy updates, decoupled from the current actor’s parameters.
• Alignment as Decoupled Divergence Minimization: Orthogonalized Policy Optimization (OPO (Zixian, 18 Jan 2026)) frames alignment as weighted Bregman divergence minimization, decoupling sampling geometry (importance weights) from optimization geometry (penalty), e.g., via $\alpha$-divergence weights and quadratic (chi-square) regularization.

Theoretical motivations revolve around reducing bootstrap/extrapolation bias, breaking feedback loops that cause overfitting or instability, disentangling sample selection from optimization curvature, and preventing the contamination of policy features with spurious cues required only for value estimation.

2. Principal Methodologies

Several representative methodologies exemplify decoupled policy-value optimization:

Approach	Value Model Usage	Policy Optimization
DEPO (Tan et al., 17 Oct 2025)	Synchronous V, with token-level decoupling and clipping	PPO-like, advantage downweighted on redundant/inefficient tokens
VC-PPO (Yuan et al., 3 Mar 2025)	Value pretrained offline, decoupled GAE parameters	PPO with actor $\lambda_\text{actor} < 1$, critic $\lambda_\text{critic}=1$
DVPO (Huang et al., 24 Feb 2025)	Pretrained GVM, frozen during policy optimization	PPO-like, static token-level advantages from GVM, no critic update
$V_0$ (Zhang et al., 3 Feb 2026)	Generalist, context-aware value model, no updates during RL	Uses $V_0$ (zero-gradient) as baseline for PPO/GRPO; also for routing/resource allocation
IDAAC (Raileanu et al., 2021)	Fully separated networks, confounder-invariant loss	Actor learns from advantage-based auxiliary loss, no shared grads
DRO(P) (Liu et al., 2023)	Scores learned offline only on training data, fixed at test	Policy improved at deployment by maximizing fixed score, not via coupled policy iteration
IVG (Byravan et al., 2019)	Latent dynamics + value learned off-policy, held fixed	Policy backpropagates through fixed model for “imagined” value gradients

A central theme is that at policy optimization time, the value model is not updated or its gradients are not propagated into the policy representation—either it is static (DVPO, $V_0$), adversarially decoupled (IDAAC), or entirely bypassed during test-time adaptation (DROP).

3. Empirical Outcomes and Comparative Advantages

Empirical findings repeatedly confirm the efficacy of decoupled approaches:

• Performance Gains: DEPO reduces sequence length by 39% and modestly improves accuracy relative to GRPO baselines by using advantage decoupling, length penalty, and clipping (Tan et al., 17 Oct 2025). VC-PPO dramatically restores PPO efficacy for long CoT reasoning tasks, with AIME pass@1 rising from 5.6% (vanilla PPO) to 41.9% (full VC-PPO) (Yuan et al., 3 Mar 2025).
• Sample/Budget Efficiency: DVPO achieves up to 35% reduction in training time and 23–34% reduction in GPU memory for comparable or superior win rates/MT-bench scores versus PPO (Huang et al., 24 Feb 2025). $V_0$ enables ∼2% absolute gains in resource-constrained rollouts and Pareto-optimal inference routing (Zhang et al., 3 Feb 2026).
• Generalization and Robustness: IDAAC yields state-of-the-art test-time performance on Procgen and robustness under distractors (DMC) by removing task-irrelevant correlations (Raileanu et al., 2021).
• Optimization Stability: OPO maintains high, stable gradient norms throughout training and avoids gradient saturation, ensuring steady improvement even in high-confidence domains (Zixian, 18 Jan 2026).

A plausible implication is that decoupling systematically mitigates the phenomena of shortcut learning, replay bias, and the instability caused by a moving critic target.

4. Applications and Problem Domains

Decoupled policy and value optimization is now fundamental in:

• Alignment of LLMs and Large Reasoning Models (LRMs): Applied to RLHF/PPO-style fine-tuning (DEPO, DVPO, VC-PPO, $V_0$), notably for long context, chain-of-thought (CoT), and complex math/coding prompts (Tan et al., 17 Oct 2025, Yuan et al., 3 Mar 2025, Huang et al., 24 Feb 2025, Zhang et al., 3 Feb 2026).
• Offline RL and Test-Time Adaptation: DROP separates value estimation and policy extraction for conservative adaptation during deployment, avoiding error propagation typical in iterative approaches (Liu et al., 2023).
• Robot Learning and Transfer: IVG demonstrates acceleration in robot manipulation learning and robust transfer by partitioning model/value training from policy improvement (Byravan et al., 2019).
• Generalization in Procedurally Generated or OOD Environments: IDAAC shows the necessity of representation separation in visually diverse tasks (Procgen, DMC with distractors) (Raileanu et al., 2021).

The approach enables resource-constrained scheduling, efficient inference routing, and robust scaling across task families.

5. Architectural and Algorithmic Patterns

At the architectural and algorithmic level, the following practices are recurrent:

• Frozen or Global Value Baselines: During policy learning, value estimation is performed by a model not updated during policy optimization (DVPO, $V_0$), yielding consistent baselines and removing actor-critic feedback loops.
• Token- or Segment-Level Decoupling: In DEPO, token-level advantages are computed separately for efficient and inefficient output segments, with down-weighting and clipping to remove bias from overlong or spurious reasoning (Tan et al., 17 Oct 2025).
• Parameter and Representation Separation: Policy and value use distinct encoders, sometimes adversarially regularized to prevent leakage of instance-specific signal (IDAAC).
• Decoupled GAE and Pretraining: In VC-PPO, the advantage estimator for the actor ($\lambda_\text{actor}$) and critic ($\lambda_\text{critic}$) are distinct; value is pretrained on full MC return, ensuring signal propagation even with sparse rewards (Yuan et al., 3 Mar 2025).
• Test-Time Optimization over Static Value Functions: In offline RL, policies are adapted on the fly by maximizing over a frozen value/score model (DROP).
• Decoupled Sampling and Optimization Geometry: OPO allows independent tuning of exploration (via $\alpha$) and regularization (via $\mu$), as opposed to KL-based entanglement.

6. Limitations and Future Directions

Limitations of current decoupled approaches include:

• Lack of Policy-Reward Adaptivity: Static value models (DVPO, DROP, $V_0$) cannot adapt online to drift between policy and reward; this creates a ceiling on policy optimality when environmental feedback changes (Huang et al., 24 Feb 2025, Liu et al., 2023).
• Approximation Error Bound: For global value models, the accuracy of return-to-go prediction is a bottleneck on policy improvement (see Theorem 1 in (Huang et al., 24 Feb 2025)).
• Representation Mismatch: When value models are conditioned on offline or signature trajectories but not the current evolving policy, domain shift can reduce efficacy.
• Hyperparameter Sensitivity: Algorithmic stability can still depend on careful balancing of decoupling parameters, advantage clipping, or regularization strengths.
• Limited OOD Robustness: In $V_0$, transfer to unseen query IDs reduces AUC score compared to in-distribution queries (Zhang et al., 3 Feb 2026).

Suggested directions include online fine-tuning of frozen global value models, richer policy conditioning, multi-step TD and distributional-value estimation, and integration of decoupled value inference during rollout-based decoding (Huang et al., 24 Feb 2025).

7. Summary Table of Key Approaches

Method	Value Model	Policy–Value Coupling	Empirical Outcome(s)	Reference
DEPO	Token-level, online	Advantage clipped/downweighted	–39% length, +2% accuracy vs. GRPO	(Tan et al., 17 Oct 2025)
VC-PPO	Offline-pretrained	Decoupled GAE	PPO collapse fixed; +30% absolute pass@1	(Yuan et al., 3 Mar 2025)
DVPO	Frozen GVM	None, fixed Q baseline	–23–34% memory/time vs. PPO; ≥SOTA wins	(Huang et al., 24 Feb 2025)
OPO	Ratio-coordiate, quadratic	Decoupled sampling/optim.	+5% accuracy vs. GSPO, stable grads	(Zixian, 18 Jan 2026)
$V_0$	Generalist, context-aware	None (forward only)	Pareto-optimal routing, ↑calibration	(Zhang et al., 3 Feb 2026)
IDAAC	Separate, invariant	No shared parameters	+11.5 normalized test vs. PPG	(Raileanu et al., 2021)
DROP	TD Bellman, frozen	None (test-time opt.)	Matches SOTA iterative offline RL	(Liu et al., 2023)
IVG	Latent model, fixed	Alternating gradient	2–4× speedup on transfer learning	(Byravan et al., 2019)

Decoupled policy/value optimization delineates a rigorous framework for stabilizing, simplifying, and generalizing RL/RLHF across both online and offline settings, with applications spanning LLM alignment, robust robot learning, transfer, and efficient inference allocation. This paradigm integrates architectural separation, algorithmic decoupling, and theoretical divergence, yielding demonstrable gains in performance, efficiency, and stability.