---
title: 'Dual-GRPO: Dual Group Relative Policy Optimization'
url: https://www.emergentmind.com/topics/dual-grpo
type: topic
---

# Dual-GRPO: Dual Group Relative Policy Optimization

“Dual-GRPO” is best understood as an umbrella label for Group Relative Policy Optimization variants in which the standard group-relative surrogate is augmented by a second, coupled optimization structure. In recent work, that second structure has taken several distinct forms: exemplar-driven sampling plus dual reward guidance in photorealistic portrait generation [2606.26930], dual-anchor advantages for low-dispersion verifiable rewards [2606.06058], a shared probe state that controls both clipping and temperature [2605.20722], a two-tier reward for answer correctness and reasoning consistency [2506.16141], and two-level or two-stage decompositions of reasoning and correction [2509.24494], [2506.04746]. This suggests that “Dual-GRPO” is not a single canonical algorithm, but a recurrent design pattern for extending GRPO when a single group-relative signal is too weak, too noisy, or too coarse.

## 1. Canonical GRPO substrate

All Dual-GRPO interpretations retain the basic GRPO scaffold: for each conditioning input, a policy samples a group of rollouts, assigns a scalar reward to each rollout, normalizes rewards within the group, and applies a PPO-style clipped update with those group-relative advantages. In the text-to-image formulation used by PortraitGen, the advantage is written as
\[
\hat{A}^i_t = \frac{r_i - \text{mean}(\{r_i\}_{i=1}^G)}{\text{std}(\{r_i\}_{i=1}^G)},
\]
and the policy update uses the standard clipped importance-ratio form with KL regularization to a reference policy [2606.26930].

The same critic-free structure appears in language-model and instruction-following variants. MDP-GRPO restates the standard GRPO normalization as
\[
z_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},
\]
with \(\mu_{\text{group}}\) and \(\sigma_{\text{group}}\) computed over the completions in a prompt-local group [2606.06058]. AGPO likewise starts from the GRPO surrogate
\[
J_{\text{GRPO}}(\theta) = \mathbb{E}\Biggl[ \frac{1}{G}\sum_{i=1}^{G} \Bigl( \min\bigl( \rho_i(\theta)\,A_i,\operatorname{clip}(\rho_i(\theta),1-\varepsilon,1+\varepsilon)A_i \bigr) -\beta\,D_{\mathrm{KL}\!\bigl(\pi_\theta\parallel\pi_{\text{ref}}\bigr) \Bigr) \Biggr],
\]
and changes only how \(A_i\), \(\varepsilon\), and the rollout temperature are controlled [2605.20722].

What distinguishes Dual-GRPO variants is therefore not the removal of the GRPO core, but the insertion of a second structure that supplements pure group-relative z-scoring. That second structure may act on the reward, the advantage, the sampling group, the rollout temperature, or the stage decomposition of the policy itself.

## 2. Dual reward guidance

The most direct Dual-GRPO instantiation is “PortraitGen: Exemplar-Driven GRPO with Dual-Reward Guidance for Photorealistic Portrait Generation” [2606.26930]. Its duality is explicit at the reward level. Each GRPO group contains \(G-1\) model-generated images plus one real exemplar image inverted with BELM, and rewards are computed by two complementary models: OmniReward for general image quality and AI-Portrait for human-centric fidelity. OmniReward scores each image on **Content**, **Clarity**, **Lighting and Color**, and **Composition**, while AI-Portrait performs exhaustive within-group pairwise comparisons and assigns a win-rate
\[
R_{win}(o_i) = \frac{1}{G-1} \sum_{j=1, j \neq i}^G \mathbb{I}(o_i \succ o_j),
\]
where \(\mathbb{I}(o_i \succ o_j)\) is \(1\) if \(o_i\) contains fewer synthetic artifacts than \(o_j\), and \(0\) otherwise [2606.26930]. The paper refers to this combination as a “Dual-Reward” module.

A closely related reward-level duality appears in GRPO-CARE for multimodal reasoning [2506.16141]. There, standard outcome-supervised GRPO is extended with a two-tier reward: a base reward for answer correctness and formatting, plus an adaptive consistency bonus computed from a slowly evolving EMA reference model. For a trajectory \((\tau_g,a_g)\), the reference model estimates the likelihood of the answer conditioned on the reasoning trace,
\[
p_g = \frac{1}{|a_g|} \sum_{i=1}^{|a_g|} \pi_{\phi}(a_{g,i} \mid x, \tau_g, a_{g,<i}),
\]
and a consistency bonus is awarded only to relatively high-accuracy trajectories whose clipped \(p_g\) exceeds the group baseline [2506.16141]. The total reward is
\[
R_g = r_{\text{acc},g} + r_{\text{fmt},g} + \lambda_{\text{cons}}\, r_{\text{acc},g} \cdot \mathbb{I}[\text{consistent}],
\]
and the explicit KL penalty of standard GRPO is removed in favor of this reward-side regularization [2506.16141].

In both cases, the second reward channel is not a learned critic. It is a second comparator acting on a different failure mode: OmniReward complements AI-Portrait’s anti-artifact discrimination, and answer correctness is complemented by reasoning-to-answer consistency. Dual-GRPO in this sense denotes GRPO with two simultaneously active reward semantics rather than a single scalar preference.

## 3. Dual anchors and dual statistical control

A second lineage of Dual-GRPO modifies the advantage estimator itself. MDP-GRPO addresses three pathologies of z-score normalization under discrete multi-constraint rewards—**low-variance amplification**, **mean-centering blindness**, and **zero-variance collapse**—by adding a second anchor to the standard group-relative score [2606.06058]. The first anchor is the ordinary GRPO term
\[
z_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},
\]
while the second is a goal-aware absolute anchor
\[
\delta_i = 2\sqrt{C(x)}(r_i - 0.5).
\]
After prospect-theoretic shaping
\[
\tilde{v}_i =
\begin{cases}
\lambda_{+}\,\tanh(\beta_{\text{PT}}\, v_i), & v_i \ge 0,\\[4pt]
\lambda_{-}\,\tanh(\beta_{\text{PT}}\, v_i), & v_i < 0,
\end{cases}
\]
the final advantage is
\[
A_i = (1-\alpha)\,\tilde{z}_i + \alpha\,\tilde{\delta}_i.
\]
This is a literal dual-anchor GRPO: one term preserves relative ranking within the group, the other restores an absolute notion of how far the rollout is from satisfying the constraint set [2606.06058].

AGPO introduces a different kind of duality: not two advantage anchors, but two controllers driven by one shared probe-derived state [2605.20722]. Its uncertainty score is
\[
U_t = w_r\,\hat{\sigma} \;+\; w_e\,\mathcal{E}_{\text{probe}} \;+\; w_k\,|\tilde{\kappa}_3(r)|,
\qquad
\widetilde U_t = U_t - \bar U_t,
\]
where \(\hat{\sigma}\) is reward dispersion, \(\mathcal{E}_{\text{probe}}\) is probe vote entropy, and \(|\tilde{\kappa}_3(r)|\) is safeguarded reward skewness [2605.20722]. The same state then drives two separate control laws. The rollout temperature is
\[
\tau_t = \operatorname{clip}\!\Bigl(\tau_{\text{base}}\bigl(1+\lambda\,\widetilde U_t\bigr),\; \tau_{\min},\tau_{\max}\Bigr),
\]
while the adaptive clip radius is
\[
\varepsilon_{\mathrm{adaptive}} = \operatorname{clip}\Bigl(
\varepsilon_{\mathrm{base}}\,
\frac{H(\pi_{\theta_{\mathrm{old}}})\bigl(1+\delta\,\mathcal{E}_{\mathrm{probe}}\bigr)}
{1+\alpha\,\hat{\sigma}+\zeta\,|\tilde{\kappa}_3(r)|+\gamma\,\widehat{D}_{\mathrm{KL}}^{\mathrm{step}}},
\varepsilon_{\min},\,\varepsilon_{\max}
\Bigr).
\]
AGPO therefore operationalizes Dual-GRPO as “one probe, two controllers”: one controls exploration, the other the trust region [2605.20722].

## 4. Duality in group construction

Another interpretation of Dual-GRPO changes the composition of the GRPO group itself. PortraitGen is the clearest example. Instead of sampling all \(G\) rollouts from the old policy, it generates \(G-1\) portraits through the reverse SDE and inserts one real photograph from the training set as the final member of the group. BELM inversion is used to recover the exemplar’s intermediate latents and step-wise trajectory probabilities, so that the real image can be treated as another rollout in the GRPO objective [2606.26930]. This produces a dual group: model-generated samples versus an inverted real exemplar.

At the opposite extreme, “It Takes Two: Your GRPO Is Secretly DPO” reduces the group to its minimal nontrivial size and argues that GRPO is fundamentally contrastive [2510.00977]. In the \(G=2\) case, rewards induce a positive rollout \(o^+\) and a negative rollout \(o^-\), and the resulting 2-GRPO objective is
\[
\mathcal{J}_\text{2-GRPO} = \mathbb{E}_{q,\,o^+,o^-} \left[ \frac{1}{2}\left( \pi_\theta^{\text{GRPO}}(o^+\mid q) - \pi_\theta^{\text{GRPO}}(o^-\mid q) \right) \right].
\]
Under binary rewards, the within-pair advantage effectively collapses to \(\{+1,-1,0\}\): the better rollout is pushed up, the worse rollout is pushed down, and ties yield no update [2510.00977]. Dual-GRPO in this interpretation means pairwise positive-versus-negative GRPO.

Multi-GRPO provides a broader grouping formalism and makes the “dual reward group” case explicit [2512.00743]. In its multi-objective setting, rewards are normalized independently before aggregation; with exactly two reward groups, the combined advantage is
\[
\hat{A}_i^{\text{Dual}} = w_1\hat{A}_i^{(1)} + w_2\hat{A}_i^{(2)}.
\]
The paper states that a hypothetical Dual-GRPO is therefore a constrained or simplified Multi-GRPO instance, typically with two reward groups, two branches, or both [2512.00743].

## 5. Two-level and two-stage optimization

A further family of Dual-GRPO variants factorizes the rollout into two semantic stages rather than two reward channels. GRPO-MA decomposes chain-of-thought training into a **thought policy** and an **answer policy**, both sharing the same underlying parameters [2509.24494]. For each prompt, the model samples \(K\) thoughts \(th_i\), then \(M\) answers \(ans_{i,j}\) per thought. The thought value is
\[
V(th_i) = \frac{1}{M}\sum_{j=1}^M R_{i,j},
\]
the thought advantage is normalized across thoughts,
\[
A(th_i) = \frac{V(th_i) - \text{Mean}_k[V(th_k)]}{\text{Std}_k[V(th_k)]},
\]
and the answer advantage is normalized across all answers,
\[
A(ans_{i,j}) = \frac{R_{i,j} - \text{Mean}_{k,l}[R_{k,l}]}{\text{Std}_{k,l}[R_{k,l}]}.
\]
The final loss is the sum of a GRPO-style term on thought tokens and a GRPO-style term on answer tokens [2509.24494]. Here the duality is architectural: reasoning and answer generation receive distinct relative feedback.

Multi-Layer GRPO pushes the stage decomposition further by explicitly introducing two GRPO layers [2506.04746]. The first layer uses standard GRPO to generate an initial response for the original query. The second layer receives the original query together with the first-layer response and is trained, again with GRPO, to identify and correct errors in that initial response. The same policy is shared across both layers, but the training distribution is dual-stage: \(q \mapsto o\) for solution generation, then \((q,o) \mapsto \tilde{o}\) for self-correction [2506.04746].

These methods suggest a broader definition of Dual-GRPO in which the two “sides” are not two reward heads but two factorized subproblems. One side learns to produce an initial trajectory; the other learns to refine, confirm, or correct it. The common GRPO machinery is retained, but the rollout semantics become explicitly hierarchical.

## 6. Theoretical unification and empirical status

The most formal mathematical account comes from \(f\)-GRPO and \(f\)-HAL, which recast GRPO-style alignment as variational \(f\)-divergence estimation between aligned and unaligned distributions [2602.05946]. In that framework, GRPO is generalized into an on-policy divergence estimator between reward-aligned and reward-unaligned distributions, while \(f\)-HAL interpolates between on-policy reward alignment and off-policy preference alignment. The paper states that this “provides the mathematical blueprint for a Dual‑GRPO algorithm,” either through explicit dual critics, dual multipliers, or simultaneous optimization of reward-based and preference-based divergences [2602.05946]. This does not define a single Dual-GRPO loss, but it does supply a unifying interpretation: duality may refer to two distributions, two objectives, or a primal–dual variational structure.

Empirically, the reported gains are heterogeneous but consistently favorable. PortraitGen, the paper that explicitly presents itself as a “Dual-GRPO” interpretation, reaches **OmniReward Content 0.97**, **UnifiedReward Coherence 3.83**, and **PickScore 22.77** on PortraitBench [2606.26930]. GRPO-CARE reports a **6.7%** gain on the hardest SEED-Bench-R1 evaluation level together with a **24.5%** improvement in reasoning–answer consistency [2506.16141]. MDP-GRPO improves strict constraint satisfaction by **up to 5.0% on Llama-3.2-3B** under discrete multi-constraint rewards [2606.06058]. AGPO, which uses dual statistical feedback rather than dual rewards, reaches **67.3% on GSM8K** and **40.5% on MATH** with Qwen2.5-14B under the same generated-token budget as PPO and GRPO [2605.20722]. At the minimal pairwise end, 2-GRPO is reported to achieve performance **on par with 16-GRPO**, using only **1/8 of the rollouts** and reducing training time by **over 70%** [2510.00977].

A recurring misconception is that Dual-GRPO names one specific algorithm. The surveyed literature suggests otherwise. In current usage, the label can denote dual rewards, dual anchors, dual controllers, dual groups, dual stages, or dual variational objectives. Another misconception is that duality necessarily introduces a learned critic; in nearly all of these variants, the method remains critic-free and keeps GRPO’s group-relative backbone intact. The more precise reading is therefore structural: Dual-GRPO denotes GRPO with two coupled sources of relative guidance, introduced to overcome specific failure modes of single-signal group normalization.

Source: https://www.emergentmind.com/topics/dual-grpo