Negative-Enhanced GRPO

Updated 6 April 2026

Negative-Enhanced GRPO is a reinforcement learning method that utilizes virtual samples and advantage calibration to learn from groups with uniformly negative rewards.
It incorporates techniques like asymmetric clipping, confidence-reweighing, and token-level negative gradient modulation to stabilize learning under challenging reward landscapes.
Empirical results show that NGRPO improves sample efficiency and accuracy on math reasoning and LLM alignment tasks by addressing gradient misalignment issues.

Negative-Enhanced GRPO (NGRPO) refers to a set of recent algorithmic advances that extend Group Relative Policy Optimization (GRPO) in reinforcement learning with verifiable rewards (RLVR) for LLM alignment and mathematical reasoning. NGRPO methods overcome critical deficiencies of vanilla GRPO—specifically, the inability to learn from "negative groups" (where all group samples are incorrect) and the oscillatory or misaligned effects of negative gradient updates. By introducing virtual samples, advantage calibration, confidence-reweighting, contrastive multi-label objectives, or token-level negative gradient modulation, NGRPO consistently yields stronger learning signals and improved sample efficiency, especially on hard tasks or when correct demonstrations are rare.

1. Limitations of GRPO and Motivation for Negative-Enhancement

GRPO operates by generating a group of $G$ trajectories for a prompt, computing their verifiable rewards $r_1, \ldots, r_G$ , and applying a standardized intra-group advantage:

$A_i = \frac{r_i - \mu_R}{\sigma_R}$

where $\mu_R$ and $\sigma_R$ are the mean and standard deviation across the group. This construction yields zero gradients in homogeneous groups (all $r_i$ identical)—rendering GRPO incapable of learning from groups in which all samples are failures, a scenario prevalent in challenging domains such as mathematical reasoning (Nan et al., 23 Sep 2025, Feng et al., 9 Oct 2025). Moreover, the penalization of incorrect responses is undifferentiated in GRPO and can suppress the log-likelihood of correct answers, a phenomenon termed Lazy Likelihood Displacement (LLD) (Deng et al., 24 May 2025). These pathologies motivate the development of negative-enhanced variants that re-inject learning signal into negative groups and correct gradient misalignment.

2. Core Negative-Enhanced GRPO Methods

Numerous, independently introduced NGRPO variants share the explicit goal of extracting gradient utility from negative groups and managing gradient interactions:

Advantage Calibration via Virtual Maximum-Reward Samples: Augments each group with a hypothetical perfect sample ( $r_{\max}$ , typically 1.0) before computing the mean and variance for normalization. This yields nonzero, typically negative, calibrated advantages for all real samples in previously "degenerate" groups, thus allowing the policy to learn even when all sampled outputs are wrong (Nan et al., 23 Sep 2025).
Asymmetric Clipping: Applies stricter clipping bounds on negative-advantage samples than on positive ones ( $\varepsilon_{\mathrm{neg}} < \varepsilon_{\mathrm{pos}}$ ). This strategy prevents destabilizing gradients without inhibiting exploration from positive samples (Nan et al., 23 Sep 2025).
Confidence-Reweighted Negative Groups (LENS): Assigns nonzero, confidence-weighted penalties to incorrect responses. Overconfident mistakes (high $\pi_\theta(a|q)$ ) are penalized more, while low-confidence errors incur milder penalties. This mechanism is derived from a maximum likelihood estimation perspective, yielding the per-sample reward

$\widetilde r(a) = r(a) - (1 - r(a)) \frac{1}{G} \frac{c(a)}{\hat D(q) - c(a)}$

(with $r_1, \ldots, r_G$ 0 the length-normalized probability and $r_1, \ldots, r_G$ 1 an empirical normalizer) (Feng et al., 9 Oct 2025).

Token-level Negative Gradient Modulation (NTHR): Identifies tokens in negative samples whose penalization would most suppress correct answers. Their negative-advantage terms are selectively downweighted, mitigating the LLD effect (Deng et al., 24 May 2025).
Contrastive Partitioned Loss (ReNCE/NGRPO): Employs explicit positive-vs-negative partitioning with a multi-label noise contrastive estimation (NCE) objective. Top-reward samples are contrasted against all negatives with normalized log-softmax scores, coupled with an adaptive margin and KL trust region penalty for stability (Zhang et al., 30 Jan 2026).

3. Formal Objectives and Algorithmic Structure

The algorithmic realizations of NGRPO extend the GRPO surrogate objective with calibrated advantages, contrastive normalization, or selective reweighting:

Advantage Calibration and Asymmetric Clipping (Nan et al., 23 Sep 2025):

For group $r_1, \ldots, r_G$ 2, augment $r_1, \ldots, r_G$ 3.
Compute

$r_1, \ldots, r_G$ 4

Calibrated advantage for each real sample: $r_1, \ldots, r_G$ 5.
Clipped surrogate loss:

$r_1, \ldots, r_G$ 6

with typical values $r_1, \ldots, r_G$ 7, $r_1, \ldots, r_G$ 8 (Nan et al., 23 Sep 2025).

Confidence-Reweighting (LENS) (Feng et al., 9 Oct 2025):

For each group, compute length-normalized confidences $r_1, \ldots, r_G$ 9 and group-wise normalizer $A_i = \frac{r_i - \mu_R}{\sigma_R}$ 0.
Adjusted rewards:

$A_i = \frac{r_i - \mu_R}{\sigma_R}$ 1

Standardized to zero mean/unit variance across group, yielding per-sample advantages for policy gradient updates.

Multi-Label Contrastive Objective (ReNCE/NGRPO) (Zhang et al., 30 Jan 2026):

Partition $A_i = \frac{r_i - \mu_R}{\sigma_R}$ 2 samples into positive and negative sets by reward.
For each prompt $A_i = \frac{r_i - \mu_R}{\sigma_R}$ 3, use the loss:

$A_i = \frac{r_i - \mu_R}{\sigma_R}$ 4

with $A_i = \frac{r_i - \mu_R}{\sigma_R}$ 5, $A_i = \frac{r_i - \mu_R}{\sigma_R}$ 6.

4. Empirical Performance and Benchmark Results

NGRPO methods have been systematically evaluated on math reasoning datasets (MATH-500, AIME2025, AMC23, Minerva, OlympiadBench):

Advantage calibration plus asymmetric clipping yields consistent improvement in Pass@ $A_i = \frac{r_i - \mu_R}{\sigma_R}$ 7 Area Under Curve (AUC) and exact match metrics over PPO, GRPO, DAPO, and related baselines. Example result on AIME2025: NGRPO achieves 31.28% vs. DAPO's 30.27% and GRPO's 28.33%, with similar margins on AMC23 and MATH500 (Nan et al., 23 Sep 2025).
Confidence-reweighted LENS-variant further boosts Pass@ $A_i = \frac{r_i - \mu_R}{\sigma_R}$ 8 accuracy, especially on hard problem subsets, and allows otherwise wasted negative groups (≈35–45% of groups) to produce meaningful updates (Feng et al., 9 Oct 2025).
Token-level NTHR reduces Lazy Likelihood Displacement and yields 0.8–2.4 point gains across multiple LLM sizes and task-specific models (Deng et al., 24 May 2025).
Explicit contrastive losses (ReNCE) surpass DAPO and GRPO on a six-benchmark mean pass@1 score: 63.0% (ReNCE) vs. 61.8% (DAPO) vs. 60.3% (GRPO) (Zhang et al., 30 Jan 2026).

Across all approaches, controlled updates to negative samples substantially improve sample efficiency, learning stability, and final accuracy, especially when correct generations are sparse.

5. Algorithmic Workflows and Implementation

NGRPO is modular and can be implemented as follows (variant-dependent):

Sample a group of $A_i = \frac{r_i - \mu_R}{\sigma_R}$ 9 responses per prompt using the current or reference policy.
Evaluate verifiable rewards for all group members.
Augment the group (if using calibrated virtual sample) and compute normalized advantages or confidence-based penalties.
Partition (for contrastive/NCE objectives) or compute per-token influence scores (for NTHR).
Apply asymmetric clipping or other stability constraints during the surrogate objective computation.
Aggregate losses across prompts/groups, optionally apply KL regularization for trust-region control.
Optimize policy parameters via batched gradient steps.

Typical implementations use batch size 512–1024, group size $\mu_R$ 0 of 8–16, and AdamW as optimizer. Exact pseudocode for each method is provided in (Nan et al., 23 Sep 2025, Feng et al., 9 Oct 2025, Deng et al., 24 May 2025, Zhang et al., 30 Jan 2026).

6. Statistical and Learning-Theoretic Considerations

NGRPO methods transform the geometry of policy gradients in positive and negative groups. Advantage calibration with virtual rewards ensures that degenerate cases (identical rewards) do not produce null gradients. Confidence-weighted penalties scale the influence of negative examples in proportion to model uncertainty or overconfidence. Token-level downweighting of negative gradients specifically targets those tokens in incorrect responses that “drag” the probability of correct sequences downward, directly addressing the LLD phenomenon (Deng et al., 24 May 2025). Contrastive losses yield bounded, normalized updates, robustly separating correct from incorrect samples while continuing to provide gradients from all samples in the group (Zhang et al., 30 Jan 2026).

7. Broader Implications and Extensions

NGRPO principles generalize beyond mathematical reasoning and can be directly applied to code generation, retrieval-augmented reasoning, or any RLVR domain with significant numbers of negative or low-quality groups. The key mechanisms—virtual sample augmentation, penalty reweighting, token-level masking, and contrastive normalization—are orthogonal and, in some cases, combinable. These methods prevent gradient starvation, promote more global exploration, and adapt the strength of negative updates to local difficulty and model confidence (Nan et al., 23 Sep 2025, Feng et al., 9 Oct 2025, Deng et al., 24 May 2025, Zhang et al., 30 Jan 2026).

A plausible implication is that future research will extend NGRPO to multi-stage, hierarchical, or curriculum RLVR, as well as adversarial evaluation or self-paced discovery of new virtual positives. Adaptive schemes for hyperparameters such as margin scale, asymmetric clip ratios, and penalty weights are also prospective directions. In all cases, the central insight remains: leveraging and regulating negative samples is critical for stable and efficient reinforcement learning in settings where correct trajectories are rare or the reward landscape is highly nonconvex.