---
title: '2-GRPO: Two-Sample GRPO in Reinforcement Learning'
url: https://www.emergentmind.com/topics/two-sample-grpo-2-grpo
type: topic
---

# 2-GRPO: Two-Sample GRPO in Reinforcement Learning

Two-Sample Group Relative Policy Optimization (2-GRPO) is a specialization of GRPO, a reinforcement learning (RL) algorithm employed for post-training large language models (LLMs) in the verifiable-reward regime. Unlike conventional wisdom that prescribes large group sizes to ensure stability and precise estimation, 2-GRPO sets the group size $G=2$ and achieves training and empirical results on par with standard “large-group” settings, with substantially reduced computational overhead. This configuration is underpinned by connections to contrastive learning and Direct Preference Optimization (DPO), and yields unbiased, low-variance, pairwise policy updates. 

## 1. GRPO Framework and the Two-Sample Specialization

Group Relative Policy Optimization operates by sampling $G$ trajectories (rollouts) for each prompt $q$ from the policy $\pi_\theta(\cdot\mid q)$, observing the corresponding binary verifiable rewards $r_i\in\{0,1\}$, and forming a group-normalized advantage:
\[
A_i = \frac{r_i - \frac1G\sum_{j=1}^G r_j}{\sqrt{\frac1G\sum_{j=1}^G(r_j-\bar r)^2}+\varepsilon},\qquad\bar r=\frac{1}{G}\sum_{j} r_j
\]
The GRPO objective (with omitted clipping for clarity) is given by:
\[
\mathcal{J}_{\rm GRPO}(\theta) = \mathbb{E}_{q,\{\tau_i\}_{i=1}^G}\;\frac{1}{G} 
\sum_{i=1}^G A_i \sum_{t=1}^{|o_i|} \log \pi_\theta(o_{i,t}\mid o_{i,<t},q)
\]
For $G=2$, the advantages reduce to $A_1=\mathrm{sign}(r_1-r_2)$ and $A_2=-A_1$, corresponding to a pure pairwise comparison. The loss per prompt collapses to:
\[
\mathcal{L}_{\rm 2\text{-}GRPO}(q) = -\frac12\Bigl(\sum_t \log\pi_\theta(o^+_t\mid o^+_{<t},q) - \sum_t \log\pi_\theta(o^-_t\mid o^-_{<t},q)\Bigr)
\]
where $(o^+,r^+=1)$ is the preferred and $(o^-,r^-=0)$ the non-preferred trajectory.

## 2. Contrastive-Learning Interpretation and DPO Equivalence

A central insight is that the GRPO policy-gradient objective, in both finite-sample and population forms, is a contrastive loss:
\[
\nabla_\theta \mathcal{L} = -\mathbb{E}\left[a\,\nabla_\theta\log\pi_\theta(y^+|x) - b\,\nabla_\theta\log\pi_\theta(y^-|x)\right]
\]
For $G=2$, the grouping matches the DPO (Direct Preference Optimization) form, establishing 2-GRPO as algebraically equivalent (up to a constant scaling) to a DPO-style pairwise update:
\[
\mathcal{L}_{\rm DPO}(\theta) = -\mathbb{E}_{(q,o_+,o_-)}\,\log\sigma\Bigl(\beta\left[\log\frac{\pi_\theta(o_+\mid q)}{\pi_{\rm ref}(o_+\mid q)} 
- \log\frac{\pi_\theta(o_-\mid q)}{\pi_{\rm ref}(o_-\mid q)}\right]\Bigr)
\]
Thus, both GRPO (with $G=2$) and DPO operate via contrastive, pairwise preference aggregation over sampled outputs [2510.00977].

## 3. Theoretical Properties and Statistical Implications

Several results establish the soundness of this two-sample regime:
- **Advantage Consistency:** The sign and relative scaling of the 2-GRPO advantage estimator matches large-$G$ GRPO, up to a constant factor, in the $N\to\infty$ limit.
- **Gradient Variance:** For a fixed total rollout budget, increasing $Q$ (number of prompts per minibatch) by reducing $G$ results in variance per update scaling inversely in $Q$. Consequently, 2-GRPO possesses $8\times$ lower per-update gradient variance than 16-GRPO when both use equal total rollouts per update.
- **Exploration on Hard Prompts:** Distributing the rollout budget over more, smaller groups (i.e., using $G=2$ with larger $Q$) increases the probability of generating successful rollouts on hard prompts, especially as the policy improves over time.

## 4. Alignment Objective, Pairwise Preferences, and Regularization

The 2-GRPO stationary objective integrates both reward and regularization terms:
\[
\mathcal{J}_{2\text{-}{\rm GRPO}}\bigl(\pi_\theta(\cdot\mid q)\bigr)
= \mathbb{E}_{o,o'\sim\pi_\theta}\left[\mathcal{P}(o\succ o'\mid q)-\mathcal{P}(o'\succ o\mid q)\right]
- \beta\, \KL\left(\pi_{\rm ref}(\cdot\mid q)\|\pi_\theta(\cdot\mid q)\right)
\]
The first term aggregates expected pairwise preferences, while the second enforces closeness to a reference policy via a reverse-KL penalty. In the binary case (answers $a,b$ with preference margin $\gamma_{a,b}$, see below), the stationary policy is analytically tractable:
\[
\pi_\theta(a\mid q) 
= \frac{1}{2}\left(
1-\frac{\beta}{\gamma_{a,b}} + \sqrt{\left(1-\frac{\beta}{\gamma_{a,b}}\right)^2 + 4\frac{\beta}{\gamma_{a,b}}p_{\rm ref}}
\right)
\]
Here, $\beta$ modulates the trade-off between exploiting pairwise margin $\gamma_{a,b}$ and adhering to $\pi_{\rm ref}$ [2502.18548].

## 5. Sample Complexity, Computational Overhead, and Batch Statistics

For fixed rollout budget $R=QGT$, both 16-GRPO and 2-GRPO perform the same number of updates. However, wall-clock time per step is dramatically lower for small $G$ due to parallelization and memory efficiency:
| Method      | Group Size ($G$) | Prompts ($Q$) | Rollouts/Step | Relative Training Time |
|-------------|------------------|--------------|---------------|-----------------------|
| 16-GRPO     | 16               | 32           | 512           | 100% (baseline)       |
| 8-GRPO      | 8                | 64           | 512           | ∼85%                  |
| 4-GRPO      | 4                | 128          | 512           | ∼75%                  |
| 2-GRPO      | 2                | 256          | 512           | ∼30%                  |

Generating 16 rollouts per prompt is up to 70% slower than 2, reflecting substantial real-world speedups for 2-GRPO [2510.00977].

## 6. Empirical Validation

Experimental campaigns on mathematical reasoning tasks confirm that 2-GRPO attains parity with, and occasionally surpasses, standard multi-rollout GRPO:
| Method           | Rollouts | Time | Mean@32 | Pass@32 |
|------------------|----------|------|---------|---------|
| Baseline w/o RL  | –        | –    | 31.83   | 81.92   |
| 16-GRPO          | 1.2M     | 100h | 70.24   | 87.24   |
| 2-GRPO           | 0.15M    | ∼30h | 69.28   | 87.43   |
| Δ%               | –        | –70% | –0.96   | +0.19   |

The savings are consistent across LLM architectures and datasets (Qwen-1.5B, Qwen-7B, DeepSeek-1.5B; MATH, DAPO-Math-Sub), demonstrating that 2-GRPO achieves comparable or superior policy quality using only 1/8 of the rollouts and less than 30% of the training time [2510.00977].

## 7. Practical Considerations and Research Directions

2-GRPO highlights that large group normalization is non-essential: unbiased, pairwise normalization yields both stable training and lower variance. The compute–statistical trade-off, as realized in 2-GRPO, enables frequent, low-variance policy updates with minimal loss in data-efficiency. Zero-advantage rollouts, though contributing to normalization, can be omitted from the backward pass while maintained in normalization statistics for computational savings. Adaptive group-sizing heuristics and more efficient rollout-generation schemes represent plausible future advancements. The equivalence of GRPO (with $G=2$) and DPO also invites research into unified, pairwise-preference-based post-training for reinforcement learning from verifiable rewards [2510.00977] [2502.18548].

Source: https://www.emergentmind.com/topics/two-sample-grpo-2-grpo