---
title: Reward Bias Substitution
url: https://www.emergentmind.com/topics/reward-bias-substitution
type: topic
---

# Reward Bias Substitution

to=arxiv_search.search  бызшәа:  пүтjson
{"query":"all:\"reward bias substitution\" OR ti:\"Reward Bias Substitution\"","max_results":10,"sort_by":"relevance"}{"query":"id:2605.27996 OR id:2409.17407 OR id:2103.12656 OR id:2603.12795 OR id:2604.27495 OR id:2512.23461","max_results":10,"sort_by":"relevance"}
to=arxiv_search.search ￣亚洲្មី  微信上的天天中彩票 to=arxiv_search.search 大发展有限公司官网  天天中彩票双色球json
{"query":"id:2605.27996","max_results":5,"sort_by":"relevance"}{"query":"id:2409.17407","max_results":5,"sort_by":"relevance"}{"query":"id:2103.12656","max_results":5,"sort_by":"relevance"}{"query":"id:2603.12795","max_results":5,"sort_by":"relevance"}{"query":"id:2604.27495","max_results":5,"sort_by":"relevance"}{"query":"id:2512.23461","max_results":5,"sort_by":"relevance"}
Reward Bias Substitution denotes a family of interventions in reinforcement learning, imitation learning, and RLHF that replace a conventional, biased, or otherwise problematic reward signal with an alternative object that is intended to better track the target objective. Depending on context, the substituted object may be a distribution of successful outcomes, a calibrated scalar reward, a factorized latent reward representation, a group-relative optimization signal, or a fairness-aware incentive function. In recent work, the same phrase also names a failure mode: single-axis debiasing can remove one measured proxy while redirecting optimization pressure onto correlated proxies, so that apparent mitigation under an audit distribution does not imply mitigation under policy optimization [2103.12656][2409.17407][2605.27996].

## 1. Conceptual scope and formalization

A common formalization writes the learned reward as an additive decomposition
\[
r_\theta(x)=r_\theta^*(x)+b_c^\theta(c(x)),
\]
where \(r_\theta^*(x)\) is the calibrated or intended component and \(b_c^\theta(c(x))\) is a bias term depending only on a measurable characteristic \(c(x)\), such as response length, markdown structure, or style [2409.17407]. In this formulation, reward bias substitution means estimating \(b_c^\theta\) and replacing the biased reward or margin with a calibrated surrogate that approximates the latent, bias-free quantity.

Other papers formulate the same idea at different representational levels. Example-based control replaces a handwritten reward with a success-event probability \(p^\pi(e_{t+}=1\mid s_t,a_t)\), learned directly from transitions and success examples rather than from an explicit reward function [2103.12656]. Bayesian non-negative reward modeling decomposes scalar reward as
\[
r(x,y)=\theta(x,y)^\top \Phi,
\]
with instance-specific latent factors \(\theta\) and global non-negative factors \(\Phi\), so that bias-related factors can be downweighted rather than the whole reward discarded [2602.10623]. Representation-learning approaches instead posit latent variables \(Z=(Z_C,Z_S)\), where \(Z_C\) is bias-free and independent of a spurious surrogate \(S\), and then train the downstream reward only on the recovered \(\hat Z_C\) [2510.23751]. Information-theoretic formulations maximize mutual information between reward predictions and preference labels while minimizing mutual information with explicit bias attributes such as length, sycophancy, or format [2512.23461].

A broader implication is that reward bias substitution is not one method but a design pattern. It may act on the task specification, the scalar reward, internal representations, optimization baselines, or even the incentive system around model training. The critical 2026 formulation sharpens this further by arguing that single-axis mitigation can rotate optimization pressure onto correlated proxies rather than eliminate it [2605.27996].

## 2. Replacing explicit reward specification

One direct form of reward substitution removes the need to specify a reward at all. In example-based control, the environment is treated as an MDP without reward, with a binary event variable \(e_t\in\{0,1\}\) indicating task success. The objective is to maximize the discounted probability of eventual success,
\[
\arg\max_\pi \; p^\pi(e_{t+}=1),
\]
using only a distribution of success examples \(p(s_t\mid e_t=1)\) and a transition dataset [2103.12656]. The key technical object is a future-success classifier \(C^\pi(s_t,a_t)\) whose odds ratio equals the desired success probability,
\[
\frac{C^\pi(s_t,a_t)}{1-C^\pi(s_t,a_t)} = p^\pi(e_{t+}=1\mid s_t,a_t).
\]
This quantity satisfies a data-driven Bellman equation in which the usual reward term is replaced by \((1-\gamma)p(e_t=1\mid s_t)\), even though that scalar is never explicitly estimated [2103.12656].

The resulting algorithm, Recursive Classification of Examples, alternates between learning the classifier \(C_\theta^\pi(s,a)\) from success examples and transitions, and updating the policy to maximize classifier output. In the tabular setting, its expected update is equivalent to value iteration with reward \(r(s_t)=(1-\gamma)p(e_t=1\mid s_t)\), and tabular RCE converges [2103.12656]. Empirically, it outperforms prior methods that first learn an explicit reward and then optimize it.

A related substitution appears in adversarial imitation learning. Standard GAIL commonly derives either always-positive rewards
\[
r^+(s,a)=-\log(1-D(s,a))
\]
or always-negative rewards
\[
r^-(s,a)=\log D(s,a),
\]
and the paper on neutral reward functions argues that these induce survival bias or termination bias depending on environment structure [2009.09467]. The proposed replacement is a neutral log-odds reward,
\[
r^{\text{neutral}}(s,a)=\log D(s,a)-\log(1-D(s,a)).
\]
Under oracle-style analysis, this removes both survival bias and termination bias. In Minigrid multiple-terminal tasks, the neutral reward achieves the highest success rates, including \(0.97\pm 0.15\) on GoToDoor and DistShift1, outperforming negative-reward GAIL and DAC in the reported comparisons [2009.09467].

## 3. Post-hoc calibration and inference-time intervention

A second major line of work keeps the reward model fixed and substitutes its outputs at inference time. Post-hoc reward calibration assumes the additive decomposition
\[
r_\theta(x)=r_\theta^*(x)+b_c^\theta(c(x))
\]
and estimates the bias term from already scored examples, then subtracts it without retraining the RM [2409.17407]. The basic RC-Mean estimator uses local averaging in feature space, while RC-LWR uses LOWESS-style locally weighted regression with adaptive bandwidth and robustification. For pairwise comparison, the calibrated margin becomes
\[
\hat{\Delta}^*_{r_\theta}(x_1,x_2)
= \Delta_{r_\theta}(x_1,x_2)
-\big(\hat b_c(c(x_1))-\hat b_c(c(x_2))\big),
\]
optionally scaled by a calibration constant \(\gamma\) [2409.17407].

In the length-bias case, this procedure yields an average gain of about \(+3.11\) RewardBench points across 33 reward models with RC-LWR-Penalty, reduces average absolute reward-length correlation from \(0.293 \pm 0.184\) to about \(0.023 \pm 0.022\), and lowers gameability from roughly \(20\)–\(38\%\) to roughly \(9\)–\(12\%\) on strong BT reward models [2409.17407]. The same framework also extends to markdown features and to two-dimensional calibration over length and markdown count.

FiMi-RM makes the same substitution explicit for length bias, but by fitting a non-linear bias model \(model_f(\mathrm{len}(y))\) and then training the reward model to be decorrelated from it [2505.12843]. The fitting stage uses a sinusoidal length encoding, a two-layer ResNet, and a combined Pearson-plus-MSE objective. The debiasing stage minimizes
\[
\mathcal{L}'_{\text{pearson}} = |\rho(r,\hat r_{\text{detach}})| + \mathcal{L}_{\text{BT}},
\]
so that reward outputs are driven toward low correlation with the learned length-bias component [2505.12843]. On Qwen2.5-7B, this produces far more balanced reward accuracy between cases where the chosen answer is longer and cases where the rejected answer is longer, and it gives the best length-controlled win rate in both Best-of-\(N\) and DPO settings reported in the paper.

Inference-time interventions can also act inside the reward model rather than on its final scalar. CIRM identifies neurons whose activations are strongly correlated with predefined bias attributes such as length, paragraph count, overlap, exclamation marks, and bold text, and then clamps those neurons to median values at inference time [2604.27495]. On the reported setups, this edits less than \(2\%\) of all neurons in the reward models and allows small 2B and 7B RMs, when used for preference annotation, to achieve alignment performance comparable to a 70B RM on AlpacaEval and MT-Bench [2604.27495]. The same paper reports that bias signals are concentrated in early layers.

SteerRM replaces neuron-level editing with sparse autoencoder features. It identifies bias-related SAE features from contrastive pairs and suppresses them at inference time, improving RM-Bench Hard-split accuracy by \(7.3\) points on average across six reward models while preserving overall performance [2603.12795]. Mechanistic reward shaping applies a related null-space projection to low-complexity bias directions such as length, uncertainty markers, and answer position; across RewardBench-2, all single-bias and combined interventions are reported as non-inferior to baseline reward quality under a \(5\)-point margin, while substantially reducing the targeted biases [2603.03291].

## 4. Factorized, causal, and information-theoretic substitutions

A more structural approach replaces an opaque scalar reward with a disentangled internal representation. Bayesian Non-negative Reward Modeling factorizes reward as \(r(x,y)=\theta(x,y)^\top\Phi\), where \(\theta\in\mathbb{R}_+^K\) is sparse and instance-specific and \(\Phi\in\mathbb{R}_+^K\) is a sparse global factor vector [2602.10623]. The intended effect is “disentanglement-then-debiasing”: local sparsity encourages factor specialization, while global sparsity downweights factors that do not consistently improve likelihood. On RM-Bench Hard, the paper reports that a vanilla BT RM has Pearson correlation between reward and length of about \(0.488\), while BT-BNRM reduces this to about \(0.123\) without any explicit length penalty [2602.10623]. This makes reward bias substitution concrete at the factor level: bias-related factors remain represented but can be suppressed or assigned near-zero weight.

Information-theoretic debiasing replaces correlation penalties with mutual-information control. DIR maximizes mutual information between reward predictions and preference labels while minimizing mutual information with bias attributes such as response length, sycophancy, and format [2512.23461]. The practical loss keeps the standard Bradley–Terry preference term and adds a CLUB-based upper bound for the bias MI. For length bias, DIR reduces the Pearson correlation between reward and length on RM-Bench from \(0.533\) to \(0.468\); for format bias, it moves bold and list win-rates from \(89.0\%\) and \(92.5\%\) under BT to \(51.2\%\) and \(52.0\%\), while also improving difficult RewardBench-Filtered subsets such as Chat Hard, Safety, and Reasoning [2512.23461]. In PPO experiments, it also yields higher average benchmark accuracy than the reported baselines.

A causally motivated representation-learning framework provides stronger identifiability claims. It assumes observed text \(T\) is generated from latent variables \(Z=(Z_C,Z_S)\), where \(Z_C\) is bias-free and independent of a surrogate \(S\), while \(Z_S\) carries the spurious dependence [2510.23751]. Under the stated assumptions, the paper shows that the non-spurious latent subspace is theoretically identifiable from data, and then trains a VAE-like model so that reward is learned only from \(\hat Z_C\). On synthetic data, the recovered latent variables achieve \(R^2\approx 0.83\pm 0.03\); on the sycophancy benchmark, worst-case accuracy rises to \(0.63\) from \(0.47\) for vanilla; on the concept-bias benchmark, worst-case accuracy rises to \(0.60\) from \(0.31\) [2510.23751]. Here reward bias substitution is literal: the raw textual representation is replaced by a theoretically identified bias-free latent subspace.

A related substitution happens in preference likelihood rather than reward representation. In the cognitive-bias RLHF paper, the standard global rationality parameter \(\beta\) is replaced by an instance-dependent
\[
\beta_{\text{new}}=\operatorname{logistic}(k(\theta-D_f)),
\]
where \(D_f\) is an LLM-judged probability that the observed feedback is influenced by cognitive bias [2605.06895]. This downweights comparisons likely to reflect conjunction fallacy, anchoring, base-rate neglect, or related biases. On CogBias, the downstream debiased model reaches \(67.6\%\) ground-truth accuracy versus \(42.4\%\) for the vanilla LLM, and under the 1:3 biased-to-ground-truth training ratio it yields \(67.6\%\) versus \(1.8\%\) for the fixed-\(\beta\) baseline [2605.06895].

## 5. Optimization-time substitution and system-level reward design

Some methods substitute not the reward itself but the optimization machinery around it. BiasGRPO is a critic-free RLHF method for social-bias mitigation that replaces the PPO value-function baseline with a group-relative baseline computed from a group of sampled completions for the same prompt [2606.04807]. For rewards \(r_1,\dots,r_G\), the advantage is
\[
\hat A_i = \frac{r_i-\mu_r}{\sigma_r},
\quad
\mu_r=\frac{1}{G}\sum_{j=1}^G r_j.
\]
The substitution is explicit: the critic \(V(s)\) is discarded, and a centered, normalized group-relative signal is used instead. Reported training statistics show reward standard deviation \(0.0668\) for BiasGRPO versus \(0.1434\) for PPO, alongside better performance on BOLD, RealToxicityPrompts, and BBQ [2606.04807].

Reward-Biased Maximum Likelihood Estimation substitutes a pure likelihood objective with a reward-augmented estimator. In finite MDPs, the estimator maximizes a criterion of the form
\[
\alpha(\tau_k)J(\theta,\pi)
-\sum_{(x,u)}n_k(x,u)\,KL(\hat p_k(x,u),\theta(x,u)),
\]
and the resulting RBMLE policy attains \(\mathcal{O}(\log T)\) regret under the stated assumptions [2011.07738]. In neural contextual bandits, NeuralRBMLE adds a bias term \(\alpha(t)\max_a f(x_{t,a};\theta)\) to the regularized log-likelihood, yielding two variants with \(\widetilde{\mathcal{O}}(\sqrt{T})\) regret [2203.04192]. In both cases, the reward bias term is deliberately inserted into estimation to enforce exploration.

Outside RLHF, substitution also appears in reward engineering for fairness and incentives. In RL-based feature selection, the naive performance reward is replaced by a multi-component reward
\[
R(s_t,a_t)
= W
-\sum_{f\in S\cap B}\psi
-\sum \mathbf{1}\{P(f_b,f_s)\}\frac{w(f_b,f_s)\lambda}{\ell(f_b,f_s)}
+\phi(|S|)
+\sum_{f\in S\cap R}\rho,
\]
combining AUC, direct-bias penalties, indirect proxy-bias penalties, size regularization, and bonuses for preferred features [2510.09705]. In medical federated learning, Shapley-based reward systems are designed separately for predictive performance and for subgroup bias, and the combined reward system is explicitly intended to prevent a performance-only scheme from transferring model bias against patients to the institutional level [2205.00470].

These optimization-level examples suggest that reward bias substitution is not limited to correcting reward-model outputs. It also includes replacing exploitable optimization targets, baselines, or institutional payoffs with objectives that internalize robustness, exploration, or fairness.

## 6. Audit gaps, failure modes, and unresolved issues

The strongest recent critique is that single-axis mitigation can create the appearance of success while merely redirecting optimization pressure. The 2026 paper formalizes mitigation outcomes into a regime taxonomy and proves that successful mitigation, bias substitution, and overcorrection produce identical observables under any audit-distribution scoring, including ranking accuracy and win-rate, even when granted oracle access to the true reward [2605.27996]. The reason is a measurement-versus-optimization gap: mitigation is usually audited on a fixed distribution \(\mu_{\text{diag}}\), whereas policy training shifts to a policy-induced distribution \(\mu_\pi\).

The paper gives several concrete demonstrations. In language-model RLHF, adding a length penalty during GRPO compresses responses as intended but redirects optimization pressure onto confidence calibration, driving the policy into overconfidence while factual free-form accuracy falls [2605.27996]. A published length-debiasing operator can zero reward-length correlation on the audit distribution yet reintroduce bias under best-of-\(N\) selection on three of four SOTA reward models [2605.27996]. The same work also reports a length-sycophancy coupling whose direction reverses under human–LLM judge disagreement.

This critique reframes earlier successes. Reported gains such as \(+3.11\) RewardBench points from post-hoc calibration, \(+7.3\) RM-Bench Hard points from SteerRM, or reduced reward–length correlation after FiMi-RM or DIR show that bias proxies can be suppressed on the measured distribution [2409.17407][2603.12795][2512.23461]. A plausible implication is that such results are necessary but not sufficient for certifying that optimization pressure has been removed rather than displaced. The 2026 analysis therefore prescribes augmenting evaluation with policy-induced distributions and tracking multiple biases jointly, rather than relying on single-axis audit scores alone [2605.27996].

Several limitations recur across the literature. Calibration methods assume that the true reward is approximately independent of the chosen characteristic in expectation; if longer or more detailed answers are genuinely better, full removal can under-reward good outputs [2409.17407]. Representation-editing methods currently work best for low-complexity biases and have difficulty with entangled effects such as sycophancy or model-style sensitivity [2603.03291]. Causal and information-theoretic approaches require either explicit surrogates, diverse labelers, or carefully specified bias attributes, and they remain limited to observed or modeled biases [2510.23751][2512.23461].

Taken together, the literature treats reward bias substitution both as a remedy and as a diagnostic warning. As a remedy, it replaces flawed reward signals with examples, calibrated scores, factorized latents, invariant representations, group-relative baselines, or fairness-aware objectives. As a warning, it highlights that removing a measured proxy on an audit set does not by itself establish that optimization no longer exploits bias. The topic therefore sits at the intersection of reward modeling, causal representation learning, mechanistic interpretability, and evaluation under distribution shift [2103.12656][2605.27996].

Source: https://www.emergentmind.com/topics/reward-bias-substitution