---
title: Decoupled Reward Policy Optimization (DRPO)
url: https://www.emergentmind.com/topics/decoupled-reward-policy-optimization-drpo
type: topic
---

# Decoupled Reward Policy Optimization (DRPO)

Searching arXiv for recent papers on DRPO and related decoupled reward/policy optimization variants.
arxiv_search.query({"search_query":"all:\"Decoupled Reward Policy Optimization\" OR all:\"Divergence Regularized Policy Optimization\" OR all:\"Group reward-Decoupled Policy Optimization\" OR all:\"Decoupled Policy Optimization\"","start":0,"max_results":10,"sort_by":"submittedDate","sort_order":"descending"})
Decoupled Reward Policy Optimization (DRPO) denotes a class of reinforcement-learning formulations in which reward construction, normalization, or use is separated from another component that is conventionally entangled in PPO-style or GRPO-style post-training. In the most direct use of the name, DRPO was introduced for efficient reasoning in large reasoning models by decoupling the length-based learning signal of correct rollouts from incorrect ones, so that correct but long outputs are not converted into negative training signal [2510.04474]. Closely related work extends the same decoupling logic to multi-reward aggregation, value-free reward normalization, and broader reward–policy design, indicating that DRPO is better understood as a methodological family than as a single universally standardized objective [2606.16771][2506.13702][2602.19526].

## 1. Terminological scope and naming

The literature does not reserve the acronym **DRPO** for one unique algorithm. In work on efficient reasoning, **DRPO** means **Decoupled Reward Policy Optimization** [2510.04474]. In adjacent multi-reward work, the closely related acronyms **GDPO** and **GD$^2$PO** mean **Group reward-Decoupled Policy Optimization** and **Group-Dynamic reward-Decoupled Policy Optimization** [2606.16771]. In value-free single-trajectory optimization, **RPO** is explicitly presented as an instantiation of DRPO in which reward normalization is decoupled from policy learning [2506.13702]. In Deep Research agents, DRPO is used more broadly for the principle of disentangling prompt template, reward function, and policy optimizer [2602.19526].

By contrast, one 2026 paper uses **DRPO** to mean **Divergence Regularized Policy Optimization**, a trust-region method that replaces DPPO’s hard mask with a smooth quadratic regularizer on policy shift rather than a reward-decoupling mechanism [2606.09821]. This acronym collision is important because the two usages address different failure modes.

| Variant | Decoupled component | Representative paper |
|---|---|---|
| DRPO | Correct-rollout reward shaping from incorrect rollouts | [2510.04474] |
| GDPO / GD$^2$PO | Reward dimensions from one another during normalization and filtering | [2606.16771] |
| RPO | Reward normalization from learned value estimation | [2506.13702] |
| Search-R1 DRPO | Prompt, reward, and optimizer selection | [2602.19526] |
| DRPO (divergence-regularized) | Trust-region control via divergence regularization | [2606.09821] |

A useful consequence of this taxonomy is that “decoupling” in DRPO is not tied to one mathematical object. Depending on the formulation, what is decoupled may be positive versus negative samples, reward groups, reward normalization versus policy fitting, or optimizer design variables.

## 2. Core DRPO formulation for efficient reasoning

The canonical Decoupled Reward Policy Optimization formulation targets the overthinking problem in RL-trained large reasoning models. The motivating observation is that GRPO normalizes scalar rewards over an entire rollout group, so when a length penalty is added, a rollout that is correct but long can receive a negative normalized advantage and be actively discouraged. DRPO addresses this by separating the learning signal for correct rollouts from that of incorrect rollouts, and by restricting length-based shaping to the positive subset only [2510.04474].

The construction begins from the Discriminative Constrained Policy Optimization objective with scoring function
$$
s_\theta(o,q)=\frac{1}{|o|}\sum_{t=1}^{|o|}\log \pi_\theta(o_t\mid q,o_{<t}).
$$
Let $\pi_{\text{old}}^+(o\mid q)$ and $\pi_{\text{old}}^-(o\mid q)$ denote the on-policy distributions over correct and incorrect outputs. DRPO perturbs only the positive distribution by solving a KL-regularized optimization over a length reward $r_l(o)\in[0,1]$. The resulting positive weighting is
$$
\omega(o)=\frac{\exp(r_l(o)/\lambda)}{\sum_{o^*\sim\pi_{\text{old}}^+}\exp(r_l(o^*)/\lambda)}.
$$
This yields a decoupled objective in which correct samples are reweighted within the positive group, while incorrect samples remain in a separate discriminative term. A key stated property is that length-based penalties only re-scale the positive group; they never push any correct rollout’s advantage below zero [2510.04474].

The implementation is on-policy. At each RL step, a batch of questions is sampled, $G$ rollouts are drawn per question, and the outputs are partitioned into $S_q^+$ and $S_q^-$. The minibatch objective combines a weighted positive term, a log-sum-exp negative term, and a squared KL-violation penalty. Its gradient decomposes into $G_1$, which contains the positive and negative discriminative contributions, and $G_2$, which enforces the trust-region constraint through the sample KL estimate. The paper emphasizes that this formulation is general: any scalar preference reward that applies only to correct outputs can replace the length reward [2510.04474].

The limiting case $\lambda\to\infty$ is also explicit. Then $\omega(o)=1$, and DRPO reduces to vanilla DisCO. This situates DRPO as a positive-subspace reweighting mechanism rather than a wholly separate optimization paradigm.

## 3. Multi-reward decoupling: GDPO and GD$^2$PO

A second major line of DRPO-related work addresses **multi-dimensional rewards**. In GDPO, the overall score is decomposed into reward groups, and the RL loss is computed from group-relative advantages rather than from a single undifferentiated scalar. Formally, for reward dimension $i$ and rollout $n$,
$$
A_n^i=\frac{r_n^i-\operatorname{mean}(r_1^i,\dots,r_G^i)}{\operatorname{std}(r_1^i,\dots,r_G^i)+\epsilon_{\text{adv}}},
$$
and GDPO aggregates these as
$$
A_n^{\text{sum}}=\sum_{i=1}^M w^i A_n^i.
$$
The identified failure mode is **advantage cancellation**: if a rollout has positive advantage on some reward dimensions and negative advantage on others, the aggregated signal can be near zero, yielding negligible learning signal [2606.16771].

GD$^2$PO extends GDPO with two mechanisms. The first is rollout-level conflict-aware filtering. Under **hard filtering**,
$$
\delta_n^{\text{hard}}=1[\operatorname{sign}(A_n^i)=\operatorname{sign}(A_n^j)\ \forall i,j],
$$
so only sign-consistent rollouts are retained. Under **SNR-based filtering**,
$$
\operatorname{SNR}_n=\frac{\left|\sum_i w^iA_n^i\right|}{\sum_i|w^iA_n^i|+\epsilon},
\qquad
\delta_n^{\text{SNR}}=1[\operatorname{SNR}_n>\tau].
$$
The second mechanism is query-level reweighting. If $\kappa(x)=\sum_n\delta_n$ is the number of retained rollouts for prompt $x$, GD$^2$PO scales the query’s loss contribution by $\hat\kappa(x)=\kappa(x)/G$, motivated by the Appendix C argument that gradient reliability scales proportionally to $\sqrt{\kappa(x)}$ [2606.16771].

The full objective is
$$
L_{\text{GD}^2\text{PO}}(\theta)
=
-
\mathbb{E}\!\left[
\frac{1}{G}\,\hat\kappa(x)\sum_n \frac{1}{|y_n|}\sum_t
\gamma_n^t\!\left(\theta,\delta_n\sum_i w^iA_n^i\right)
\right].
$$
The central interpretation is that decoupling in the multi-reward setting is not merely per-dimension normalization; it also requires preventing destructive interference during aggregation.

An earlier applied variant appears in **GDPO-Listener**, where rewards are normalized separately over FLAME parameter groups $\{\text{expr},\text{jaw},\text{neck},\text{eyelid},\text{eyepose},\text{rot}\}$. There, per-group sequence rewards are whitened across PPO rollouts, then combined into a weighted scalar advantage for motion generation. The stated purpose is to prevent large-scale groups such as head rotation from dominating smaller but semantically important groups such as eyelid motion, and thereby to combat the “Regression-to-the-Mean” problem in listener motion generation [2603.25020].

## 4. Reward normalization, value decoupling, and policy-only objectives

Another DRPO-related direction removes or freezes the auxiliary value machinery that typically mediates reward information. In **Reward Partitioning Optimization (RPO)**, reward normalization is performed empirically from logged $(x,y,r)$ data rather than via a jointly learned value model. For prompt $x$,
$$
\widehat Z(x)=\sum_{j\in\mathcal I_x}\pi_{\rm ref}(y_j\mid x)\exp(r_j/\tau),
\qquad
\widehat V(x)=\tau\log \widehat Z(x),
$$
and the policy is trained with the pure residual objective
$$
L(\theta)=\frac1{2N}\sum_{i=1}^N
\left[
\log\frac{\pi_\theta(y_i\mid x_i)}{\pi_{\rm ref}(y_i\mid x_i)}
-
\frac{1}{\tau}\bigl(r_i-\widehat V(x_i)\bigr)
\right]^2.
$$
The paper states that no auxiliary neural value model is trained, that the gradient is an unbiased estimator of the DRO gradient under the same KL-regularized objective, and that in the infinite-data limit $\widehat V(x)\to V^*(x)$ so RPO recovers the true KL-regularized optimum [2506.13702].

A related but distinct design is **Decoupled Value Policy Optimization (DVPO)**. DVPO pretrains a frozen **global value model (GVM)** $Q_\phi(\tau,s,a)$ with TD loss on offline RLHF data, batch-normalizes the resulting $Q$-values, and then uses the normalized frozen estimate $\hat A_t=\widetilde Q_\phi(\tau,s_t,a_t)$ inside a clipped PPO objective. The principal claim is that decoupling the value model from policy updates eliminates actor–critic interdependence while retaining token-level return-to-go supervision [2502.16944].

A third adjacent line is the **Q-function Reward Model (Q-RM)**. Here reward modeling is decoupled from language generation by training a separate discriminative policy $\phi_\varphi(s,a)$ with logits $Z_\varphi(s,a)$ on preference pairs under a Bradley–Terry likelihood. Policy optimization then uses $r_\varphi(s,a)=Z_\varphi(s,a)$ as token-level reward, with stop-gradient through the reward model during RL updates. The stated theoretical result is that, in the infinite-data limit, minimizing the Q-RM loss recovers logits proportional to the optimal token-level $Q$-function [2505.23363].

Across these formulations, the common structural move is clear: reward normalization or reward inference is detached from the unstable feedback loop of simultaneous actor–critic adaptation.

## 5. Broader decoupling as a design principle

Some papers use the DRPO idea at a broader systems level rather than as a single closed-form objective. In **Search-R1**, Decoupled Reward–Policy Optimization refers to disentangling three “knobs” in Deep Research agents: prompt template, scalar reward function, and policy-gradient optimizer. The framework studies prompt selection between Slow Thinking and Fast Thinking templates, reward design among EM, F1, and the shaped reward
$$
R_{F1+}(\tau)=R_{F1}(\tau)-\alpha\,\mathbf 1[a_s=0]-\beta\,\mathbf 1[a_a=0],
$$
and optimizer choice among REINFORCE, PPO, and GRPO. The paper does not present a single DRPO surrogate; instead, DRPO is the systematic isolation and recombination of these components [2602.19526].

A structurally related decomposition appears in **HyLaR**, where **Decoupled Policy Optimization (DePO)** addresses a hybrid action space containing discrete text actions and continuous visual latent actions on the unit hypersphere. DePO partitions time steps into text positions $\mathcal Z$ and latent positions $\mathcal S$, applies separate clipped surrogates $L_{\text{tok}}$ and $L_{\text{lat}}$, and adds independent trust-region penalties: a sample-based text KL and a closed-form von Mises–Fisher latent KL
$$
D_{\text{KL}}=\kappa A_D(\kappa)\bigl(1-\mu_{\text{new}}^\top\mu_{\text{old}}\bigr).
$$
The full objective is
$$
L_{\text{total}}(\theta)=L_{\text{tok}}+\alpha L_{\text{lat}}+\beta_{\text{tok}}L_{\text{KL}}^{\text{tok}}+\beta_{\text{lat}}L_{\text{KL}}^{\text{lat}}.
$$
This is not reward decoupling in the narrow sense, but it extends the decoupling principle to heterogeneous policy geometries [2604.20328].

A plausible implication is that DRPO has evolved from a specific fix for reward interference into a more general methodology for isolating statistically mismatched subproblems inside RL post-training.

## 6. Empirical profile, limitations, and recurrent misconceptions

The empirical claims attached to DRPO-style methods are varied because the benchmark regimes differ. In the efficient-reasoning DRPO paper, a 1.5B model on GSM8K achieves **77% length reduction** with only **1.1% performance loss**, whereas the cited follow-up baseline sacrifices **4.3%** for **68% length reduction**. Averaged over four datasets, DRPO reports **AES = +0.178** for the 1.5B model and **AES = +0.249** for the 7B model, while all baselines have negative AES [2510.04474].

In multi-reward RL, GD$^2$PO reports that conflict ratio—the fraction of sign-inconsistent rollouts—is often **10–30%** during training. On two-reward tool calling with Qwen2.5-1.5B, GD$^2$PO-Hard improves “Overall” from **1.505** for GRPO and **1.509** for GDPO to **1.520**. On three-reward tool calling with Qwen2.5-3B, GD$^2$PO-SNR raises Overall from **2.604** to **2.608**. On helpfulness-safety alignment with Llama3.2-3B, GD$^2$PO-Hard raises Overall Avg from **5.756** for GRPO and **5.760** for GDPO to **5.808** [2606.16771].

In value-free scalar-feedback language modeling, RPO is reported to converge **20–30 % faster** than KTO and DRO at all model scales. On the Large model, the reported automatic metrics include **BERTScore F1 0.8706**, **ROUGE-L 0.3162**, **BLEU 0.0464**, and **Toxicity 0.0093**, each outperforming the cited KTO and DRO baselines. Under a Mistral judge, the Large-model RPO policy achieves win rates of **84.20 % ± 1.6 %** versus SFT, **86.53 % ± 1.8 %** versus KTO, and **84.86 % ± 2.1 %** versus DRO [2506.13702].

In Search-R1, the decoupled design sequence **Slow Thinking + EM + PPO $\rightarrow$ Fast Thinking + F1+ + REINFORCE** improves Search-R1 from **0.403 to 0.442** on Qwen2.5-7B and from **0.289 to 0.331** on Qwen2.5-3B, yielding the final Search-R1++ configuration [2602.19526].

Several misconceptions recur. One is that DRPO simply means “add another reward term.” The efficient-reasoning formulation is more specific: it is designed to prevent **correct but verbose** outputs from being treated as negative examples. Another is that decoupling is equivalent to reward scalarization. GD$^2$PO explicitly argues the opposite, since scalarized group-relative advantages can cancel. A third is that decoupling removes the need for trust-region control. In practice, the cited methods retain clipping, KL penalties, or explicit trust-region constraints in different forms. Finally, the acronym itself is non-unique: **Decoupled Reward Policy Optimization** and **Divergence Regularized Policy Optimization** are distinct methods that address different optimization pathologies [2606.09821].

Taken together, the DRPO literature defines a coherent research program around **interference reduction** in RL post-training. Whether the interference arises from positive/negative sample mixing, heterogeneous reward dimensions, critic–actor coupling, or mismatched action-space geometries, the central methodological move is the same: isolate the conflicting signal path, assign it its own normalization or constraint structure, and only then reintroduce it into policy optimization.

Source: https://www.emergentmind.com/topics/decoupled-reward-policy-optimization-drpo