Decoupled Reward Policy Optimization (DRPO)

• DRPO is a reinforcement learning formulation that decouples reward construction from policy optimization to mitigate interference between correct and incorrect outputs.
• It comprises variants like GDPO, RPO, and DVPO that address multi-dimensional rewards, value decoupling, and heterogeneous policy geometries.
• Empirical studies demonstrate improvements in output quality, convergence speed, and control over length penalties across diverse model scales and tasks.

Searching arXiv for papers on DRPO and related decoupled reward/policy optimization variants. arxiv_search.query({"^^^^^^^^5search_query5 Reward Policy Optimization\" OR ^^^^^^^^5all:\5 Regularized Policy Optimization\" OR ^^^^^^^^5all:\5 reward-Decoupled Policy Optimization\" OR ^^^^^^^^5all:\5 Policy Optimization\"","start":^^^^5search_query5^^^^,"max_results":^^^^5all:\5search_query5 Decoupled Reward Policy Optimization (DRPO) denotes a class of reinforcement-learning formulations in which reward construction, normalization, or use is separated from another component that is conventionally entangled in PPO-style or GRPO-style post-training. In the most direct use of the name, DRPO was introduced for efficient reasoning in large reasoning models by decoupling the length-based learning signal of correct rollouts from incorrect ones, so that correct but long outputs are not converted into negative training signal (&&&^^^^5search_query5^^^^&&&). Closely related work extends the same decoupling logic to multi-reward aggregation, value-free reward normalization, and broader reward–policy design, indicating that DRPO is better understood as a methodological family than as a single universally standardized objective (&&&^^^^5all:\5^^^^&&&, &&&^^^^5 OR all:\5^^^^&&&, &&&^^^^5 OR all:\5^^^^&&&).

^^^^5all:\5^^^^. Terminological scope and naming

The literature does not reserve the acronym DRPO for one unique algorithm. In work on efficient reasoning, DRPO means Decoupled Reward Policy Optimization (&&&^^^^5search_query5^^^^&&&). In adjacent multi-reward work, the closely related acronyms GDPO and GDPRESERVED_PLACEHOLDER_^^^^5search_query5^^^^PO mean Group reward-Decoupled Policy Optimization and Group-Dynamic reward-Decoupled Policy Optimization (&&&^^^^5all:\5^^^^&&&). In value-free single-trajectory optimization, RPO is explicitly presented as an instantiation of DRPO in which reward normalization is decoupled from policy learning (&&&^^^^5 OR all:\5^^^^&&&). In Deep Research agents, DRPO is used more broadly for the principle of disentangling prompt template, reward function, and policy optimizer (&&&^^^^5 OR all:\5^^^^&&&).

By contrast, one ^^^^5 OR all:\5search_query5 OR all:\5^^^^6 paper uses DRPO to mean Divergence Regularized Policy Optimization, a trust-region method that replaces DPPO’s hard mask with a smooth quadratic regularizer on policy shift rather than a reward-decoupling mechanism (Yao et al., 8 Jun 2026). This acronym collision is important because the two usages address different failure modes.

Variant	Decoupled component	Representative paper
DRPO	Correct-rollout reward shaping from incorrect rollouts	(&&&^^^^5search_query5^^^^&&&)
GDPO / GDPRESERVED_PLACEHOLDER_^^^^5all:\5^^^^PO	Reward dimensions from one another during normalization and filtering	(&&&^^^^5all:\5^^^^&&&)
RPO	Reward normalization from learned value estimation	(&&&^^^^5 OR all:\5^^^^&&&)
Search-R^^^^5all:\5^^^^ DRPO	Prompt, reward, and optimizer selection	(&&&^^^^5 OR all:\5^^^^&&&)
DRPO (divergence-regularized)	Trust-region control via divergence regularization	(Yao et al., 8 Jun 2026)

A useful consequence of this taxonomy is that “decoupling” in DRPO is not tied to one mathematical object. Depending on the formulation, what is decoupled may be positive versus negative samples, reward groups, reward normalization versus policy fitting, or optimizer design variables.

^^^^5 OR all:\5^^^^. Core DRPO formulation for efficient reasoning

The canonical Decoupled Reward Policy Optimization formulation targets the overthinking problem in RL-trained large reasoning models. The motivating observation is that GRPO normalizes scalar rewards over an entire rollout group, so when a length penalty is added, a rollout that is correct but long can receive a negative normalized advantage and be actively discouraged. DRPO addresses this by separating the learning signal for correct rollouts from that of incorrect rollouts, and by restricting length-based shaping to the positive subset only (&&&^^^^5search_query5^^^^&&&).

The construction begins from the Discriminative Constrained Policy Optimization objective with scoring function

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^

Let PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^ and PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^ denote the on-policy distributions over correct and incorrect outputs. DRPO perturbs only the positive distribution by solving a KL-regularized optimization over a length reward $r_l(o)\in[0,1]$. The resulting positive weighting is

$$\omega(o)=\frac{\exp(r_l(o)/\lambda)}{\sum_{o^*\sim\pi_{\text{old}}^+}\exp(r_l(o^*)/\lambda)}.$$

This yields a decoupled objective in which correct samples are reweighted within the positive group, while incorrect samples remain in a separate discriminative term. A key stated property is that length-based penalties only re-scale the positive group; they never push any correct rollout’s advantage below zero (&&&^^^^5search_query5^^^^&&&).

The implementation is on-policy. At each RL step, a batch of questions is sampled, $G$ rollouts are drawn per question, and the outputs are partitioned into $S_q^+$ and $S_q^-$. The minibatch objective combines a weighted positive term, a log-sum-exp negative term, and a squared KL-violation penalty. Its gradient decomposes into PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5^^^^, which contains the positive and negative discriminative contributions, and PRESERVED_PLACEHOLDER_^^^^5all:\5all:\5^^^^, which enforces the trust-region constraint through the sample KL estimate. The paper emphasizes that this formulation is general: any scalar preference reward that applies only to correct outputs can replace the length reward (&&&^^^^5search_query5^^^^&&&).

The limiting case PRESERVED_PLACEHOLDER_^^^^5all:\5 OR all:\5^^^^ is also explicit. Then PRESERVED_PLACEHOLDER_^^^^5all:\5 OR all:\5^^^^, and DRPO reduces to vanilla DisCO. This situates DRPO as a positive-subspace reweighting mechanism rather than a wholly separate optimization paradigm.

^^^^5 OR all:\5^^^^. Multi-reward decoupling: GDPO and GDPRESERVED_PLACEHOLDER_^^^^5all:\5 OR all:\5^^^^PO

A second major line of DRPO-related work addresses multi-dimensional rewards. In GDPO, the overall score is decomposed into reward groups, and the RL loss is computed from group-relative advantages rather than from a single undifferentiated scalar. Formally, for reward dimension PRESERVED_PLACEHOLDER_^^^^5all:\55^^^^ and rollout PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^6,

PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^7

and GDPO aggregates these as

PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^8

The identified failure mode is advantage cancellation: if a rollout has positive advantage on some reward dimensions and negative advantage on others, the aggregated signal can be near zero, yielding negligible learning signal (&&&^^^^5all:\5^^^^&&&).

GDPRESERVED_PLACEHOLDER_^^^^5all:\5^^^^9PO extends GDPO with two mechanisms. The first is rollout-level conflict-aware filtering. Under hard filtering,

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5search_query5^^^^

so only sign-consistent rollouts are retained. Under SNR-based filtering,

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5all:\5^^^^

The second mechanism is query-level reweighting. If PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ is the number of retained rollouts for prompt PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^, GDPRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^PO scales the query’s loss contribution by PRESERVED_PLACEHOLDER_^^^^5 OR all:\55^^^^, motivated by the Appendix C argument that gradient reliability scales proportionally to PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^6 (&&&^^^^5all:\5^^^^&&&).

The full objective is

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^7

The central interpretation is that decoupling in the multi-reward setting is not merely per-dimension normalization; it also requires preventing destructive interference during aggregation.

An earlier applied variant appears in GDPO-Listener, where rewards are normalized separately over FLAME parameter groups PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^8. There, per-group sequence rewards are whitened across PPO rollouts, then combined into a weighted scalar advantage for motion generation. The stated purpose is to prevent large-scale groups such as head rotation from dominating smaller but semantically important groups such as eyelid motion, and thereby to combat the “Regression-to-the-Mean” problem in listener motion generation (&&&^^^^5all:\5^^^^9&&&).

^^^^5 OR all:\5^^^^. Reward normalization, value decoupling, and policy-only objectives

Another DRPO-related direction removes or freezes the auxiliary value machinery that typically mediates reward information. In Reward Partitioning Optimization (RPO), reward normalization is performed empirically from logged PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^9 data rather than via a jointly learned value model. For prompt PRESERVED_PLACEHOLDER_^^^^5 OR all:\5search_query5^^^^,

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5all:\5^^^^

and the policy is trained with the pure residual objective

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^

The paper states that no auxiliary neural value model is trained, that the gradient is an unbiased estimator of the DRO gradient under the same KL-regularized objective, and that in the infinite-data limit PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ so RPO recovers the true KL-regularized optimum (&&&^^^^5 OR all:\5^^^^&&&).

A related but distinct design is Decoupled Value Policy Optimization (DVPO). DVPO pretrains a frozen global value model (GVM) PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ with TD loss on offline RLHF data, batch-normalizes the resulting PRESERVED_PLACEHOLDER_^^^^5 OR all:\55^^^^-values, and then uses the normalized frozen estimate PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^6 inside a clipped PPO objective. The principal claim is that decoupling the value model from policy updates eliminates actor–critic interdependence while retaining token-level return-to-go supervision (&&&^^^^5 OR all:\5all:\5^^^^&&&).

A third adjacent line is the Q-function Reward Model (Q-RM). Here reward modeling is decoupled from language generation by training a separate discriminative policy PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^7 with logits PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^8 on preference pairs under a Bradley–Terry likelihood. Policy optimization then uses PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^9 as token-level reward, with stop-gradient through the reward model during RL updates. The stated theoretical result is that, in the infinite-data limit, minimizing the Q-RM loss recovers logits proportional to the optimal token-level PRESERVED_PLACEHOLDER_^^^^5 OR all:\5search_query5^^^^-function (&&&^^^^5 OR all:\5 OR all:\5^^^^&&&).

Across these formulations, the common structural move is clear: reward normalization or reward inference is detached from the unstable feedback loop of simultaneous actor–critic adaptation.

5. Broader decoupling as a design principle

Some papers use the DRPO idea at a broader systems level rather than as a single closed-form objective. In Search-R^^^^5all:\5^^^^, Decoupled Reward–Policy Optimization refers to disentangling three “knobs” in Deep Research agents: prompt template, scalar reward function, and policy-gradient optimizer. The framework studies prompt selection between Slow Thinking and Fast Thinking templates, reward design among EM, F^^^^5all:\5^^^^, and the shaped reward

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5all:\5^^^^

and optimizer choice among REINFORCE, PPO, and GRPO. The paper does not present a single DRPO surrogate; instead, DRPO is the systematic isolation and recombination of these components (&&&^^^^5 OR all:\5^^^^&&&).

A structurally related decomposition appears in HyLaR, where Decoupled Policy Optimization (DePO) addresses a hybrid action space containing discrete text actions and continuous visual latent actions on the unit hypersphere. DePO partitions time steps into text positions PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ and latent positions PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^, applies separate clipped surrogates PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ and PRESERVED_PLACEHOLDER_^^^^5 OR all:\55^^^^, and adds independent trust-region penalties: a sample-based text KL and a closed-form von Mises–Fisher latent KL

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^6

The full objective is

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^7

This is not reward decoupling in the narrow sense, but it extends the decoupling principle to heterogeneous policy geometries (&&&^^^^5 OR all:\5 OR all:\5^^^^&&&).

A plausible implication is that DRPO has evolved from a specific fix for reward interference into a more general methodology for isolating statistically mismatched subproblems inside RL post-training.

6. Empirical profile, limitations, and recurrent misconceptions

The empirical claims attached to DRPO-style methods are varied because the benchmark regimes differ. In the efficient-reasoning DRPO paper, a ^^^^5all:\5^^^^.5B model on GSM8K achieves 77% length reduction with only ^^^^5all:\5^^^^.^^^^5all:\5 performance loss, whereas the cited follow-up baseline sacrifices ^^^^5 OR all:\5^^^^.^^^^5 OR all:\5^^^^% for 68% length reduction. Averaged over four datasets, DRPO reports AES = +^^^^5search_query5^^^^.^^^^5all:\5 for the ^^^^5all:\5^^^^.5B model and AES = +^^^^5search_query5^^^^.^^^^5 OR all:\5 OR all:\5^^^^9 for the 7B model, while all baselines have negative AES (&&&^^^^5search_query5^^^^&&&).

In multi-reward RL, GDPRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^8PO reports that conflict ratio—the fraction of sign-inconsistent rollouts—is often ^^^^5all:\5search_query5^^^^–^^^^5 OR all:\5search_query5^^^^% during training. On two-reward tool calling with Qwen^^^^5 OR all:\5^^^^.5-^^^^5all:\5^^^^.5B, GDPRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^9PO-Hard improves “Overall” from ^^^^5all:\5^^^^.5^^^^5search_query55 for GRPO and ^^^^5all:\5^^^^.5^^^^5search_query5 for GDPO to ^^^^5all:\5^^^^.5^^^^5 OR all:\5search_query5^^^^. On three-reward tool calling with Qwen^^^^5 OR all:\5^^^^.5-^^^^5 OR all:\5^^^^B, GD$r_l(o)\in[0,1]$^^^^5search_query5^^^^PO-SNR raises Overall from ^^^^5 OR all:\5^^^^.6^^^^5search_query5 OR all:\5^^^^ to ^^^^5 OR all:\5^^^^.6^^^^5search_query5^^^^8. On helpfulness-safety alignment with Llama^^^^5 OR all:\5^^^^.^^^^5 OR all:\5^^^^-^^^^5 OR all:\5^^^^B, GD$r_l(o)\in[0,1]$^^^^5all:\5^^^^PO-Hard raises Overall Avg from 5.756 for GRPO and 5.76^^^^5search_query5^^^^ for GDPO to 5.8^^^^5search_query5^^^^8 (&&&^^^^5all:\5^^^^&&&).

In value-free scalar-feedback language modeling, RPO is reported to converge ^^^^5 OR all:\5search_query5^^^^–^^^^5 OR all:\5search_query5^^^^ % faster than KTO and DRO at all model scales. On the Large model, the reported automatic metrics include BERTScore F^^^^5all:\5^^^^ ^^^^5search_query5^^^^.87^^^^5search_query5, ROUGE-L ^^^^5search_query5^^^^.^^^^5 OR all:\5all:\5start5 OR all:\5^^^^, BLEU ^^^^5search_query5^^^^.^^^^5search_query5 OR all:\5start5 OR all:\5^^^^, and Toxicity ^^^^5search_query5^^^^.^^^^5search_query5search_query5submittedDate5 OR all:\5^^^^, each outperforming the cited KTO and DRO baselines. Under a Mistral judge, the Large-model RPO policy achieves win rates of 8^^^^5 OR all:\5^^^^.^^^^5 OR all:\5search_query5^^^^ % ± ^^^^5all:\5^^^^.6 % versus SFT, 86.5^^^^5 OR all:\5^^^^ % ± ^^^^5all:\5^^^^.8 % versus KTO, and 8^^^^5 OR all:\5^^^^.86 % ± ^^^^5 OR all:\5^^^^.^^^^5all:\5^^^^ % versus DRO (&&&^^^^5 OR all:\5^^^^&&&).

In Search-R^^^^5all:\5^^^^, the decoupled design sequence Slow Thinking + EM + PPO $r_l(o)\in[0,1]$^^^^5 OR all:\5^^^^ Fast Thinking + F^^^^5all:\5^^^^+ + REINFORCE improves Search-R^^^^5all:\5^^^^ from ^^^^5search_query5^^^^.^^^^5 OR all:\5search_query5 OR all:\5^^^^ to ^^^^5search_query5^^^^.^^^^5 OR all:\5 OR all:\5 OR all:\5^^^^ on Qwen^^^^5 OR all:\5^^^^.5-7B and from ^^^^5search_query5^^^^.^^^^5 OR all:\5^^^^89 to ^^^^5search_query5^^^^.^^^^5 OR all:\5 OR all:\5all:\5^^^^ on Qwen^^^^5 OR all:\5^^^^.5-^^^^5 OR all:\5^^^^B, yielding the final Search-R^^^^5all:\5^^^^++ configuration (&&&^^^^5 OR all:\5^^^^&&&).

Several misconceptions recur. One is that DRPO simply means “add another reward term.” The efficient-reasoning formulation is more specific: it is designed to prevent correct but verbose outputs from being treated as negative examples. Another is that decoupling is equivalent to reward scalarization. GD$r_l(o)\in[0,1]$^^^^5 OR all:\5^^^^PO explicitly argues the opposite, since scalarized group-relative advantages can cancel. A third is that decoupling removes the need for trust-region control. In practice, the cited methods retain clipping, KL penalties, or explicit trust-region constraints in different forms. Finally, the acronym itself is non-unique: Decoupled Reward Policy Optimization and Divergence Regularized Policy Optimization are distinct methods that address different optimization pathologies (Yao et al., 8 Jun 2026).

Taken together, the DRPO literature defines a coherent research program around interference reduction in RL post-training. Whether the interference arises from positive/negative sample mixing, heterogeneous reward dimensions, critic–actor coupling, or mismatched action-space geometries, the central methodological move is the same: isolate the conflicting signal path, assign it its own normalization or constraint structure, and only then reintroduce it into policy optimization.