Negative Reward Training Methods

• Negative reward training is a set of methods that penalize undesirable behavior by applying explicit negative scalar rewards, pairwise rejection labels, or reshaped reward signals.
• Techniques such as signed trajectory weighting, hard-negative mining, and pessimistic reward fine-tuning are used to mitigate reward hacking and confounding under distribution shift.
• These approaches reframe the learning process to discourage unsafe outputs while maintaining policy stability and enhancing robustness in safety-critical applications.

Negative reward training denotes a family of learning procedures in which optimization is shaped by signals associated with undesirable behavior, rejected outputs, failures, unsafe trajectories, or low-quality generations. In the recent literature, the term is not used for a single canonical objective. Some methods use literal signed scalar rewards on incorrect trajectories, as in Reward Informed Fine-Tuning, where responses are paired with $r(x,y)\in\mathbb{R}$ and incorrect responses receive $r=-0.2$ in the main setup (Liu et al., 14 Jan 2026). Others deliver negative supervision indirectly, for example by relabeling adversarially discovered high-reward but low-quality outputs as rejected preference pairs in reward-model training (Bukharin et al., 8 Apr 2025). Still others suppress undesirable behavior without explicit negative scalar rewards, using non-negative latent factorization, pessimistic reward estimation, candidate-level credit reallocation, or environment-mediated elimination (Duan et al., 11 Feb 2026, Xu et al., 26 May 2025, Ai et al., 31 Mar 2026, Dodgson et al., 18 Jan 2026). Across these formulations, the central technical problem is constant: to discourage exploitable, unsafe, spurious, or low-quality behavior without inducing reward hacking, trivial pessimism, or failure under distribution shift (Pang et al., 2022, McKinney et al., 2023).

1. Formal scope and canonical setting

In RLHF, negative reward training is usually defined relative to a standard preference-learning pipeline. A reward model $R_\theta(x,y)$ is trained from prompt/response preference triples

$D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$

with Bradley–Terry loss

$$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$

A downstream policy is then optimized against the learned reward while staying close to an SFT model through a KL term. In this setting, “negative” training signal can enter either as literal negative scalar reward, as pairwise rejection labels, or as pessimistic reshaping of the learned reward landscape (Bukharin et al., 8 Apr 2025).

The papers surveyed here therefore use the phrase across several distinct but related mechanisms.

Formulation	Mechanism of negative signal	Representative source
Signed trajectory weighting	Correct and incorrect self-generated outputs receive $r=+1.0$ and $r=-0.2$	RIFT (Liu et al., 14 Jan 2026)
Hard-negative reward-model training	High-reward adversarial outputs are relabeled as rejected preference pairs	Adv-RM (Bukharin et al., 8 Apr 2025)
Artifact-neutral preference augmentation	Non-contextual comparisons and tie labels suppress spurious cues	RRM (Liu et al., 2024)
Pessimistic reward fine-tuning	The reward is trained to lower scores of optimizer-selected outputs	PET (Xu et al., 26 May 2025)
Candidate-level blame assignment	Set-level reward is decomposed into candidate-specific contributions	ShapE-GRPO (Ai et al., 31 Mar 2026)
Survival-based elimination	Only behaviors with $\Delta R(\tau)>0$ are retained	NSL (Dodgson et al., 18 Jan 2026)

This heterogeneity matters because superficially similar phrases can denote different update locations. In some methods, the policy itself is optimized with a signed reward. In others, the reward model is the main object being trained, and the policy benefits only indirectly through a reshaped reward function. A further distinction is whether “negative” means negative-valued scalar reward, nonpositive advantage, rejection in pairwise ranking, or elimination from the training distribution.

2. Explicit signed rewards and stabilized negative updates

The clearest explicit formulation appears in Reward Informed Fine-Tuning. RIFT defines a dataset of triplets

$\mathcal{D}=\{(x,y,r)\},$

partitions it into

$$\mathcal{D}^+=\{(x,y)\mid r(x,y)>0\},\qquad \mathcal{D}^-=\{(x,y)\mid r(x,y)<0\},$$

and in the main setup uses $r=-0.2$0 and $r=-0.2$1. The naive signed objective,

$r=-0.2$2

is unstable because for $r=-0.2$3, the loss is unbounded below as $r=-0.2$4. RIFT therefore keeps the log-likelihood term for positives but replaces the negative-sample term with a linear penalty in probability: $r=-0.2$5 The paper reports that this stabilized signed-reward objective consistently outperforms RFT on mathematical benchmarks, and that performance is fairly stable for $r=-0.2$6 (Liu et al., 14 Jan 2026).

A different literature argues that explicit negative scalar rewards are not always necessary because outcome-based training can already induce negative expected advantage. In the GRPO analysis of risky intermediate actions, if an action is chosen with probability $r=-0.2$7 and increases failure probability by $r=-0.2$8, then

$r=-0.2$9

which is negative whenever $R_\theta(x,y)$0. On that view, “punishment” is delivered through negative advantage rather than through an explicit $R_\theta(x,y)$1-style reward label. The same paper argues that persistent bad actions are better explained by gradient coupling between similar samples than by the absence of negative learning signal (Liu et al., 28 Sep 2025).

Diffusion-model safety post-training exposes the same distinction. SafeDiffusion-R1 evaluates a pure negative-only variant, “$R_\theta(x,y)$2 CLIP penalty (neg-only),” and reports MeanUnsafe $R_\theta(x,y)$3, CLIP-T $R_\theta(x,y)$4, and FID $R_\theta(x,y)$5. Its steering reward, which combines positive and negative anchors through a directional objective in embedding space, reports MeanUnsafe $R_\theta(x,y)$6 and CLIP-T $R_\theta(x,y)$7. The paper’s practical conclusion is that pure negative reward can suppress the target behavior but may collapse utility, whereas a shaped reward that points toward a positive alternative is more stable (Kumar et al., 18 May 2026).

3. Hard negatives, rejected pairs, and bad-behavior data

A large fraction of work closest to “negative reward training” does not optimize the policy with a literal negative scalar reward. Instead, it manufactures negative supervision for the reward model. Adv-RM is the clearest example. It begins from the standard preference-learning setup and trains an adversarial policy to produce responses that receive high reward from a target reward model but are out-of-distribution and likely low quality. Its operational criterion combines a prompt-dependent threshold $R_\theta(x,y)$8 with a disagreement score

$R_\theta(x,y)$9

and filtered adversarial samples satisfy both $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$0 and high z-scored disagreement. These responses are then inserted as the rejected side of new preference pairs, while a reasonably strong SFT response is used as the chosen side, even when the old reward model originally preferred the adversarial response. The method adds roughly 1000 such pairs, retrains from scratch with the same Bradley–Terry loss, and stops after 2 rounds when the attack is no longer successful. In this sense, Adv-RM is a systematic method for relabeling “bad but high-scoring” outputs as negatives (Bukharin et al., 8 Apr 2025).

The same paper gives unusually direct evidence that hard negatives alter the downstream reward landscape rather than merely the training set. In synthetic experiments, Adv-RM finds adversarial examples with $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$1 standard and $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$2 strict success on train, and $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$3 standard and strict success on test. In real reward models, human-evaluated attack success reaches $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$4 on Skywork-Gemma 27B, $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$5 on Llama-Nemotron 70B, and $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$6 on Nemotron-4 340B. After retraining, downstream RLHF remains beneficial for “2–3 times more steps than RLHF” before reward hacking appears, and the authors report about $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$7 the cost of standard RLHF training (Bukharin et al., 8 Apr 2025).

Robust Reward Model Training Mitigates Reward Hacking pursues the same goal through pair construction rather than adversarial search. It augments ordinary contextual preference pairs with “Non-contextuals,” in which the contextual response must win, and “Neutrals,” in which two non-contextual responses are labeled Tie. The causal intent is to remove context-free artifacts such as length, markdown, boldface, bullet points, and stock prefixes from the preference signal. This is not explicit scalar punishment, but it acts as an anti-artifact negative signal because artifact-bearing responses can no longer win merely by carrying those features. On RewardBench, the resulting robust reward model improves accuracy from $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$8 to $D=\{x_i,y_{w,i},y_{\ell,i}\}_{i=1}^N,$9; DPO policies trained from it improve MT-Bench from $$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$0 to $$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$1 and AlpacaEval-2 length-controlled win rate from $$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$2 to $$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$3 (Liu et al., 2024).

In embodied AI, the same issue appears as a data problem. The position paper on embodied reward models argues that reward models trained primarily on successful behavior systematically over-reward unsafe interactions, poor execution, and shortcut strategies. On 723 valid RoboArena tasks, preference-ordering accuracy is about $$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$4–$$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$5 on Pick/Place but falls to $$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$6–$$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$7 on Tool Use. The intervention study shows little or no gain from text-only negatives, about $$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$8 improvement from text plus real failure videos on simpler categories, and about $$\mathcal{L}(\theta)=-\mathbb{E}_{x,y_w,y_\ell\sim D}\left[\log\left(\sigma(R_\theta(x,y_w)-R_\theta(x,y_\ell))\right)\right].$$9 improvement from dense negative reward examples on harder categories such as Tool Use and Pour Liquid. The paper’s broader claim is that realistic bad robot data are not optional calibration polish but necessary supervision for defining what should receive low reward (Tian et al., 31 May 2026).

4. Pessimistic, non-negative, and contribution-based reward models

One important response to negative reward training is to retain the goal of suppressing undesirable behavior while changing the reward representation so that explicit negative scalar values are unnecessary or tightly controlled. Bayesian Non-Negative Reward Model is the most explicit case. It keeps Bradley–Terry preference learning but represents reward through non-negative latent activations and non-negative global weights: $r=+1.0$0 Because $r=+1.0$1 and $r=+1.0$2, suppression of undesirable behavior occurs through sparse weighting and comparative ranking rather than through signed negative components. The paper reports that under $r=+1.0$3 label noise, BNRM improves over BT by up to 16.7 percentage points, and on RM-Bench Hard its Pearson correlation between response length and reward drops from $r=+1.0$4 for vanilla BT to $r=+1.0$5. The cost claim is also noteworthy: latent-factor overhead is described as negligible relative to the LLM forward pass (Duan et al., 11 Feb 2026).

PET moves in a different direction by making the reward deliberately pessimistic against optimizer-selected outputs. Its core objective is

$r=+1.0$6

so the reward is trained not only to fit preference data but also to lower the relative score of the rejection-sampling policy that would be obtained by optimizing against it. This is a direct formalization of conservative anti-overestimation. On TL;DR summarization, unregularized PPO on the pessimistic reward obtains win rate $r=+1.0$7 with KL $r=+1.0$8, while unregularized PPO on the proxy reward collapses to win rate $r=+1.0$9 with KL $r=-0.2$0. PET therefore locates the “negative” intervention at the reward-model level rather than in an external KL penalty (Xu et al., 26 May 2025).

ShapE-GRPO introduces a further variant in which negative signal arises from fair blame assignment inside a set. For candidate set $r=-0.2$1, it defines the Shapley value

$r=-0.2$2

and broadcasts $r=-0.2$3 to tokens belonging to candidate $r=-0.2$4. In the binary reward specialization, if there are $r=-0.2$5 correct candidates then each correct candidate receives $r=-0.2$6 and each incorrect candidate receives $r=-0.2$7. Under the paper’s main setting, Proposition 2 states that $r=-0.2$8 for tokens in incorrect candidates, so free-riding candidates stop inheriting positive updates from a single strong peer (Ai et al., 31 Mar 2026).

Positive-Unlabeled Reward Learning adds a cautionary complement to these methods. In both imitation learning and supervised reward learning, it argues that agent data are often unlabeled rather than negative. Standard positive-negative discrimination can therefore create a reward model that produces low reward signal regardless of the input state, while pure supervised regression can overestimate reward in unsupported regions. PURL replaces PN classification with a non-negative PU estimator and, in the supervised case, uses a masked reward

$r=-0.2$9

The central lesson is that unjustified negativity is itself a reward-model pathology, not a reliable form of conservatism (Xu et al., 2019).

5. Failure modes, diagnostics, and evaluation criteria

Negative reward training is fragile because the reward mechanism itself is often misspecified. Reward Gaming in Conditional Text Generation identifies three recurrent causes of failure: noise-induced spurious correlation, naturally occurring spurious correlation, and covariate shift. In a synthetic Sudoku example, only $\Delta R(\tau)>0$0 of all reward-model training examples are false positives, yet RL drives mean reward above $\Delta R(\tau)>0$1 while actual validity remains below $\Delta R(\tau)>0$2. In a machine-translation example, the token “...” is annotated as no-error in $\Delta R(\tau)>0$3 of its occurrences, and its frequency rises to more than three quarters of sampled generations under RL, while BLEU does not improve. The paper’s broader point is that optimizing a learned penalty or reward detector amplifies its errors rather than merely inheriting them (Pang et al., 2022).

A related warning is that a learned reward can look successful during joint training and still fail as a reusable objective. “On The Fragility of Learned Reward Functions” argues that evaluation by the final co-trained policy is insufficient, because a frozen learned reward can fail to train a new policy from scratch. The paper therefore recommends retraining-based evaluation: freeze the reward model, optimize a fresh policy against it, and evaluate under the true reward. It treats EPIC distance as supplementary rather than decisive. This perspective is directly relevant to negative reward training, since a penalty-bearing reward that only works along the original optimization path is not a robust training signal (McKinney et al., 2023).

Offline preference learning introduces another failure mode: reward confusion under spurious correlations. The Confusing Minigrid benchmark shows that fixed offline preference data can make the learned reward attach value to the wrong feature even when failures are present in the dataset. IMPEC addresses this by building a global preference chain with active learning and transitivity. In the 25-seed ablation, baseline failure count is $\Delta R(\tau)>0$4, while IMPEC reduces it to $\Delta R(\tau)>0$5. The lesson for negative reward training is that penalty assignment is an attribution problem: if the preference graph is confounded, the model may learn to penalize proxies rather than the intended bad behavior (Chen et al., 2024).

Inverse Reward Design expresses the same issue in a hand-specified reward language. It treats the designed reward $\Delta R(\tau)>0$6 not as the literal objective but as an observation about intended reward $\Delta R(\tau)>0$7 in a training MDP $\Delta R(\tau)>0$8: $\Delta R(\tau)>0$9 Its central implication for negative reward training is that the absence of a negative penalty on a novel state is not evidence that the state is acceptable; it may only mean the training environment never required that penalty to be specified (Hadfield-Menell et al., 2017).

Finally, some settings fail because negative or sparse signals never differentiate trajectories at all. The zero-reward barrier paper studies a graph-search task where the base model never samples a correct solution. Dr. GRPO, VinePPO, Progress Rewards, and Best-of-$\mathcal{D}=\{(x,y,r)\},$0 all fail on Degree-10-Path-10 under this condition. The paper’s analysis shows that if every sampled rollout has the same reward, group-relative advantages are identically zero. Mixing in easier samples such as Degree-5-Path-5 creates nonzero reward support and eventually enables the model to solve the original hard task. This suggests that a uniform negative penalty would not help unless it induced meaningful relative signal (Prakash et al., 4 Oct 2025).

6. Domain-specific instantiations and unresolved issues

Negative reward training now spans language-model alignment, diffusion safety, embodied reward models, and self-training systems. In diffusion post-training, SafeDiffusion-R1 replaces a learned safe/unsafe reward model with a steering reward defined in HPSv2/CLIP embedding space. For unsafe prompts, it steers the text representation toward a safe direction before computing image-text similarity, and trains online with GRPO on both negative and positive text prompts. On I2P nudity detection, Stable Diffusion v1.4 yields 646 detections, SafeDiffusion-R1 yields 31, and the more aggressive unsafe-anchor variant yields 15; on GenEval, compositional accuracy rises from $\mathcal{D}=\{(x,y,r)\},$1 for SD1.4 to $\mathcal{D}=\{(x,y,r)\},$2 for SafeDiffusion-R1 trained with GenEval+Nudity prompts (Kumar et al., 18 May 2026).

Embodied settings expose the cost of missing negative data even more directly. The RoboArena analysis argues for release of failed, suboptimal, unsafe, and hazardous trajectories, for synthetic bad-data generation engines, and for decentralized physical evaluation systems. Its intervention results further suggest that the most useful negatives are realistic embodied failures that remain close to the success manifold rather than generic corruptions. This implies a practical asymmetry: collecting positive demonstrations is not enough to calibrate false positives in a reward model (Tian et al., 31 May 2026).

The literature also identifies substantial tradeoffs. Adv-RM depends on ensemble disagreement heuristics and may miss failures when all reward models make the same mistake; too many adversarial samples can hurt performance, suggesting overfitting to a narrow attack family (Bukharin et al., 8 Apr 2025). BNRM’s non-negativity improves interpretability and debiasing but does not naturally express signed additive penalties (Duan et al., 11 Feb 2026). PET demonstrates that pessimistic rewards can replace KL regularization in some offline RLHF settings, but its behavior still depends on the pessimistic coefficient $\mathcal{D}=\{(x,y,r)\},$3 and the choice of inner optimizer (Xu et al., 26 May 2025). ShapE-GRPO is exact and polynomial-time for permutation-invariant set utilities, but its main theory is specialized to max-style candidate-set rewards (Ai et al., 31 Mar 2026).

A final contrast is provided by environment-mediated self-training. “Survival is the Only Reward” proposes negative-space learning, in which the environment does not provide semantic feedback, dense rewards, or task-specific supervision. Instead, trajectories are filtered by

$\mathcal{D}=\{(x,y,r)\},$4

and only survivors are propagated into future training. The paper explicitly states that $\mathcal{D}=\{(x,y,r)\},$5 is not a reward signal but a direct consequence of execution under real constraints. This framework is strongly analogous to negative reward training in spirit, since nonviable behavior is eliminated, but it is ultimately a different paradigm: failure acts by exclusion from the lineage rather than by scalar punishment in an optimization objective (Dodgson et al., 18 Jan 2026).

Taken together, the recent literature suggests that negative reward training is best understood not as a single algorithmic recipe but as a design space. At one end are explicit signed rewards with stabilized objectives; at another are hard-negative mining, pessimistic reward models, and candidate-level blame assignment; at a third are methods that avoid signed negatives and instead restructure comparisons, uncertainty, or survival criteria. The common denominator is not the presence of a negative number. It is the attempt to make undesirable behavior legible to optimization without allowing the optimizer to exploit the representation of undesirability itself.