RefGRPO: Efficient Critic-Free RL

• RefGRPO is a family of unbiased, critic-free reinforcement learning algorithms that uses trajectory-level importance weighting to correct biases in standard GRPO.
• It introduces TIC-GRPO and 2-GRPO variants, applying trajectory-level correction and pairwise contrastive losses to enhance stability, efficiency, and convergence guarantees.
• The algorithm demonstrates practical benefits including improved sample efficiency, up to 70% reduction in computational overhead, and robust performance in LLM fine-tuning and federated learning.

The RefGRPO algorithm (Reference Group Relative Policy Optimization) encompasses a family of recent methods that reframe and generalize Group Relative Policy Optimization (GRPO), a critic-free policy gradient technique widely adopted for large-scale reinforcement learning from verifiable or binary rewards. RefGRPO has been developed to address known statistical, theoretical, and practical limitations of GRPO by introducing unbiased trajectory-level importance weighting, robust advantage normalization, and computational efficiency in both standard and specialized settings. It subsumes variants such as TIC-GRPO (Trajectory-level Importance-Corrected GRPO) and 2-GRPO (two-sample GRPO/DPO-equivalent), providing provable guarantees for convergence, sample efficiency, and policy improvement. These methods have shown efficacy in finetuning LLMs, agentic RL, robust federated learning, and high-variance optimization domains.

1. Foundations and Motivation

RefGRPO is rooted in the critic-free paradigm introduced by GRPO, which eschews learned baseline value networks in favor of group-wise reward normalization. The motivation for the RefGRPO variants is twofold: (i) to correct the bias of standard GRPO, whose gradient targets the stale (old) policy unless properly importance-weighted, and (ii) to balance variance, stability, and computational cost as group size and sample scaling become limiting in high-throughput applications.

GRPO computes, for each prompt or environment state, a group of $G$ trajectories under a behavior (old) policy $\pi_{\text{old}}$, yielding scalar rewards $\{r_i\}$. The group advantage for rollout $i$ is $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$, where $\mu_G, \sigma_G$ are the group mean and standard deviation. This advantage is used per-token in a PPO-style clipped surrogate loss, updated with a KL-penalty toward a reference policy. However, original GRPO applies per-token importance reweighting but does not fundamentally correct for the distributional shift between $\pi_{\theta_{\text{old}}}$ and current $\pi_{\theta}$, inducing a small but statistically meaningful bias (Pang et al., 4 Aug 2025).

2. Core Methodology: TIC-GRPO and 2-GRPO

RefGRPO formalizes two main algorithmic corrections, each targeting bias reduction, unbiased policy improvement, and computational efficiency.

Trajectory-level Importance Correction (TIC-GRPO):

• The main innovation is to compute a single trajectory-level importance ratio

$$w_i = \frac{\prod_{t=1}^T \pi_\theta(a_t^{(i)}|s_{t-1}^{(i)})}{\prod_{t=1}^T \pi_{\theta_{\text{old}}}(a_t^{(i)}|s_{t-1}^{(i)})}$$

and to weight the entire group-normalized advantage for trajectory $i$ by $\pi_{\text{old}}$0 in the loss function.

• The updated surrogate loss is

$\pi_{\text{old}}$1

• This correction makes the policy gradient an unbiased estimator of the desired objective at the current $\pi_{\text{old}}$2 (Pang et al., 4 Aug 2025).

Minimal Group Contrastive RefGRPO (2-GRPO):

• Recognizing that group size $\pi_{\text{old}}$3 is sufficient for unbiased pairwise preference policy gradients, 2-GRPO (equivalent to a DPO step) samples two rollouts per prompt and assigns $\pi_{\text{old}}$4/$\pi_{\text{old}}$5 advantages based on reward comparison:
  • If only one is correct, assign $\pi_{\text{old}}$6 to the winner and $\pi_{\text{old}}$7 to the loser.
  • If both are correct/incorrect, assign $\pi_{\text{old}}$8 to both.
• The loss reduces to a pure contrastive loss between correct and incorrect samples:

$\pi_{\text{old}}$9

where $\{r_i\}$0 is the token log-probability sum (Wu et al., 1 Oct 2025).

Both approaches admit implementation as a drop-in replacement for standard GRPO in any PPO-like RL codebase, requiring only changes to the advantage weighting.

3. Theoretical Guarantees and Statistical Properties

The trajectory-corrected RefGRPO algorithms inherit the consistency and optimality properties detailed in recent theory (Zhou et al., 1 Mar 2026, Pang et al., 4 Aug 2025):

• Bias and Unbiasedness: Trajectory-level (not token-level) importance weighting ensures unbiased estimation of the gradient of the current policy objective. In standard GRPO, the estimator targets the old policy, with a bias of order $\{r_i\}$1 where $\{r_i\}$2 is the number of inner steps between policy refreshes and $\{r_i\}$3 is the step size (Pang et al., 4 Aug 2025).
• Variance and Scaling: The variance can be controlled systematically by group size $\{r_i\}$4 and sample batch size $\{r_i\}$5. For fixed total rollouts, $\{r_i\}$6 offers nearly equivalent performance and exploration as $\{r_i\}$7, provided the overall batch budget is maintained [(Wu et al., 1 Oct 2025); see also scaling results in (Zhou et al., 1 Mar 2026)].
• Convergence: Both standard GRPO and TIC-GRPO converge at rate $\{r_i\}$8 in the squared gradient norm under conventional RL regularity assumptions (Pang et al., 4 Aug 2025). For binary verifiable rewards, the RefGRPO closed-form policy update yields a provable amplification of the success rate above the initial reference policy, regardless of the initialization, provided suitable KL-regularization weight $\{r_i\}$9 is chosen (Mroueh, 9 Mar 2025).
• U-statistics framing: The GRPO/RefGRPO estimator is a symmetric U-statistic, achieving asymptotically minimal mean-squared error among all baselines using only prompt-level information (Zhou et al., 1 Mar 2026).

4. Algorithmic Implementation and Pseudocode

RefGRPO variants are designed for minimal overhead and highest practical utility:

• TIC-GRPO (trajectory importance correction): See step-by-step pseudocode in [(Pang et al., 4 Aug 2025), Sec. 1], involving group sampling, reward aggregation, single-trajectory IS weighting, PPO-style clipping, KL-penalty, and optimizer update.
• 2-GRPO: See practical PyTorch-style code in (Wu et al., 1 Oct 2025), batch-sampling two rollouts per prompt, computing pairwise advantages ($i$0), and backpropagating via sum-over-tokens of log-probs. $\mu_G, \sigma_G$4
• Hyperparameters: Group size $i$1 (2-GRPO), $i$2–$i$3 (TIC-GRPO); PPO clip $i$4 (asymmetric clipping possible); KL-weight $i$5; learning rate $i$6–$i$7 depending on model and batch size; refresh $i$8 every $i$9–$A_i = (r_i-\mu_G)/(\sigma_G+\delta)$0 inner steps to balance bias and computation (Pang et al., 4 Aug 2025).

5. Practical Performance, Trade-offs, and Applications

RefGRPO delivers both practical and theoretical benefits:

• Computational Efficiency: 2-GRPO achieves $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$1 70% reduction in rollout FLOPs and wall-clock time over full-group GRPO at equivalent performance. Rollout costs scale $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$2; setting $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$3 or $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$4 and increasing $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$5 as needed preserves gradient variance (Wu et al., 1 Oct 2025, Pang et al., 4 Aug 2025).
• Stability and Exploration: 2-GRPO maintains exploration on hard prompts due to sequential coverage: more prompts at $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$6 yields as many or more “at least one correct” events as a large group $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$7 with fewer prompts (Wu et al., 1 Oct 2025).
• Empirical Results: On LLM math and reasoning tasks, 2-GRPO and full-group GRPO deliver nearly identical final accuracies (within $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$8 points) across all tested architectures and tasks, validating the theoretical scaling (Wu et al., 1 Oct 2025). Convergence rates and stability are further corroborated by ablation studies with and without importance weighting (Pang et al., 4 Aug 2025).
• Domains of Use: RefGRPO has been successfully applied in LLM post-training for mathematical reasoning, agentic reinforcement learning (e.g., reflection calibration (Zhu, 12 Jun 2026)), robust federated RL, molecular property optimization, and neural combinatorial optimization.

6. Theoretical Underpinnings and Interpretations

• Relation to Contrastive Learning and DPO: 2-GRPO is mathematically equivalent to a contrastive/pairwise-preference loss (i.e., Direct Preference Optimization) in the case of binary rewards and group size 2 (Wu et al., 1 Oct 2025). This reframing explains its unbiasedness and variance properties.
• Policy Stationarity and Preference Pooling: In the population limit, RefGRPO stationary policies solve a reverse-KL-regularized version of rational pooling, not log-pooling as in RLHF. Preference aggregation depends on group normalization and KL weight, with closed-form solution in binary and pairwise settings (Vojnovic et al., 25 Feb 2025).
• Amplification Dynamics: For verifiable binary rewards, RefGRPO’s fixed-point on the success probability $A_i = (r_i-\mu_G)/(\sigma_G+\delta)$9 always exceeds the initial policy probability $\mu_G, \sigma_G$0. Sufficiently large $\mu_G, \sigma_G$1 ensures both stability and improvement (Mroueh, 9 Mar 2025).

7. Limitations and Open Directions

While RefGRPO presents strong guarantees and empirical advantages, open challenges remain:

• Applicability Beyond Binary/Verifiable Rewards: Extensions to continuous-valued rewards and non-verifiable preference signals require further validation.
• Fine-tuning KL-regularization: Sharply controlling the policy drift via $\mu_G, \sigma_G$2 and PPO clipping remains delicate in heterogeneous or highly multi-modal domains.
• Group Size Trade-offs: Although $\mu_G, \sigma_G$3 is theoretically and empirically sufficient for many settings, certain structured domains may benefit from larger or adaptively chosen group sizes (Zhou et al., 1 Mar 2026).

The RefGRPO framework—spanning TIC-GRPO, 2-GRPO, and related unbiased trajectory-level correction algorithms—represents the current best practice for scalable, sample-efficient, and provably robust critic-free reinforcement learning in large-scale model alignment and complex sequential decision-making (Pang et al., 4 Aug 2025, Wu et al., 1 Oct 2025).