2-GRPO: Two-Sample GRPO in Reinforcement Learning

• 2-GRPO is a reinforcement learning algorithm that employs pairwise (G=2) rollouts for unbiased policy updates, matching large-group performance with lower cost.
• It leverages contrastive learning principles and DPO equivalence to achieve stable policy optimization and reduced gradient variance.
• Empirical results show that 2-GRPO attains comparable accuracy to standard methods while reducing training time by up to 70% and rollout requirements by 1/8.

Two-Sample Group Relative Policy Optimization (2-GRPO) is a specialization of GRPO, a reinforcement learning (RL) algorithm employed for post-training LLMs in the verifiable-reward regime. Unlike conventional wisdom that prescribes large group sizes to ensure stability and precise estimation, 2-GRPO sets the group size $G=2$ and achieves training and empirical results on par with standard “large-group” settings, with substantially reduced computational overhead. This configuration is underpinned by connections to contrastive learning and Direct Preference Optimization (DPO), and yields unbiased, low-variance, pairwise policy updates.

1. GRPO Framework and the Two-Sample Specialization

Group Relative Policy Optimization operates by sampling $G$ trajectories (rollouts) for each prompt $q$ from the policy $\pi_\theta(\cdot\mid q)$, observing the corresponding binary verifiable rewards $r_i\in\{0,1\}$, and forming a group-normalized advantage: $$A_i = \frac{r_i - \frac1G\sum_{j=1}^G r_j}{\sqrt{\frac1G\sum_{j=1}^G(r_j-\bar r)^2}+\varepsilon},\qquad\bar r=\frac{1}{G}\sum_{j} r_j$$ The GRPO objective (with omitted clipping for clarity) is given by: $$\mathcal{J}_{\rm GRPO}(\theta) = \mathbb{E}_{q,\{\tau_i\}_{i=1}^G}\;\frac{1}{G} \sum_{i=1}^G A_i \sum_{t=1}^{|o_i|} \log \pi_\theta(o_{i,t}\mid o_{i,<t},q)$$ For $G=2$, the advantages reduce to $A_1=\mathrm{sign}(r_1-r_2)$ and $A_2=-A_1$, corresponding to a pure pairwise comparison. The loss per prompt collapses to: $G$0 where $G$1 is the preferred and $G$2 the non-preferred trajectory.

2. Contrastive-Learning Interpretation and DPO Equivalence

A central insight is that the GRPO policy-gradient objective, in both finite-sample and population forms, is a contrastive loss: $G$3 For $G$4, the grouping matches the DPO (Direct Preference Optimization) form, establishing 2-GRPO as algebraically equivalent (up to a constant scaling) to a DPO-style pairwise update: $G$5 Thus, both GRPO (with $G$6) and DPO operate via contrastive, pairwise preference aggregation over sampled outputs (Wu et al., 1 Oct 2025).

3. Theoretical Properties and Statistical Implications

Several results establish the soundness of this two-sample regime:

• Advantage Consistency: The sign and relative scaling of the 2-GRPO advantage estimator matches large-$G$7 GRPO, up to a constant factor, in the $G$8 limit.
• Gradient Variance: For a fixed total rollout budget, increasing $G$9 (number of prompts per minibatch) by reducing $q$0 results in variance per update scaling inversely in $q$1. Consequently, 2-GRPO possesses $q$2 lower per-update gradient variance than 16-GRPO when both use equal total rollouts per update.
• Exploration on Hard Prompts: Distributing the rollout budget over more, smaller groups (i.e., using $q$3 with larger $q$4) increases the probability of generating successful rollouts on hard prompts, especially as the policy improves over time.

4. Alignment Objective, Pairwise Preferences, and Regularization

The 2-GRPO stationary objective integrates both reward and regularization terms: $q$5 The first term aggregates expected pairwise preferences, while the second enforces closeness to a reference policy via a reverse-KL penalty. In the binary case (answers $q$6 with preference margin $q$7, see below), the stationary policy is analytically tractable: $q$8 Here, $q$9 modulates the trade-off between exploiting pairwise margin $\pi_\theta(\cdot\mid q)$0 and adhering to $\pi_\theta(\cdot\mid q)$1 (Vojnovic et al., 25 Feb 2025).

5. Sample Complexity, Computational Overhead, and Batch Statistics

For fixed rollout budget $\pi_\theta(\cdot\mid q)$2, both 16-GRPO and 2-GRPO perform the same number of updates. However, wall-clock time per step is dramatically lower for small $\pi_\theta(\cdot\mid q)$3 due to parallelization and memory efficiency: | Method | Group Size ($\pi_\theta(\cdot\mid q)$4) | Prompts ($\pi_\theta(\cdot\mid q)$5) | Rollouts/Step | Relative Training Time | |-------------|------------------|--------------|---------------|-----------------------| | 16-GRPO | 16 | 32 | 512 | 100% (baseline) | | 8-GRPO | 8 | 64 | 512 | ∼85% | | 4-GRPO | 4 | 128 | 512 | ∼75% | | 2-GRPO | 2 | 256 | 512 | ∼30% |

Generating 16 rollouts per prompt is up to 70% slower than 2, reflecting substantial real-world speedups for 2-GRPO (Wu et al., 1 Oct 2025).

6. Empirical Validation

Experimental campaigns on mathematical reasoning tasks confirm that 2-GRPO attains parity with, and occasionally surpasses, standard multi-rollout GRPO: | Method | Rollouts | Time | Mean@32 | Pass@32 | |------------------|----------|------|---------|---------| | Baseline w/o RL | – | – | 31.83 | 81.92 | | 16-GRPO | 1.2M | 100h | 70.24 | 87.24 | | 2-GRPO | 0.15M | ∼30h | 69.28 | 87.43 | | Δ% | – | –70% | –0.96 | +0.19 |

The savings are consistent across LLM architectures and datasets (Qwen-1.5B, Qwen-7B, DeepSeek-1.5B; MATH, DAPO-Math-Sub), demonstrating that 2-GRPO achieves comparable or superior policy quality using only 1/8 of the rollouts and less than 30% of the training time (Wu et al., 1 Oct 2025).

7. Practical Considerations and Research Directions

2-GRPO highlights that large group normalization is non-essential: unbiased, pairwise normalization yields both stable training and lower variance. The compute–statistical trade-off, as realized in 2-GRPO, enables frequent, low-variance policy updates with minimal loss in data-efficiency. Zero-advantage rollouts, though contributing to normalization, can be omitted from the backward pass while maintained in normalization statistics for computational savings. Adaptive group-sizing heuristics and more efficient rollout-generation schemes represent plausible future advancements. The equivalence of GRPO (with $\pi_\theta(\cdot\mid q)$6) and DPO also invites research into unified, pairwise-preference-based post-training for reinforcement learning from verifiable rewards (Wu et al., 1 Oct 2025, Vojnovic et al., 25 Feb 2025).