Reward Bias Substitution

• Reward Bias Substitution is a method to replace biased reward signals with calibrated surrogates that better reflect intended objectives in reinforcement learning.
• Techniques involve additive reward decomposition, example-based control, and inference-time interventions to mitigate bias redirection.
• Empirical evaluations demonstrate reduced bias correlations and improved fairness, though challenges remain in addressing underlying optimization pressure shifts.

to=arxiv_search.search бызшәа: пүтjson {"^^^^^^^^3query3 bias substitution\"^^^^^^^^3 OR ti:\3^^^^^^^^"Reward Bias Substitution\"","max_results":^^^^3all:\3query3^^^^,"sort_by":"relevance"}{" OR id:(&&&^^^^3all:\3^^^^&&&) OR id:(&&&^^^^3 OR ti:\3^^^^&&&) OR id:(Sun et al., 13 Mar 2026) OR id:(Shinoda et al., 30 Apr 2026) OR id:(Li et al., 29 Dec 2025)","max_results":^^^^3all:\3query3^^^^,"sort_by":"relevance"} to=arxiv_search.search ￣亚洲្មី 微信上的天天中彩票 to=arxiv_search.search 大发展有限公司官网 天天中彩票双色球json {"^^^^^^^^3query3 OR ti:\3^^^^&&&)","max_results":5,"sort_by":"relevance"}{"^^^^^^^^3query3 Reward Bias Substitution denotes a family of interventions in reinforcement learning, imitation learning, and RLHF that replace a conventional, biased, or otherwise problematic reward signal with an alternative object that is intended to better track the target objective. Depending on context, the substituted object may be a distribution of successful outcomes, a calibrated scalar reward, a factorized latent reward representation, a group-relative optimization signal, or a fairness-aware incentive function. In recent work, the same phrase also names a failure mode: single-axis debiasing can remove one measured proxy while redirecting optimization pressure onto correlated proxies, so that apparent mitigation under an audit distribution does not imply mitigation under policy optimization (&&&^^^^3 OR ti:\3^^^^&&&, &&&^^^^3all:\3^^^^&&&, &&&^^^^3query3^^^^&&&).

^^^^3all:\3^^^^. Conceptual scope and formalization

A common formalization writes the learned reward as an additive decomposition

PRESERVED_PLACEHOLDER_^^^^3query3^^^^

where PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^ is the calibrated or intended component and PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^ is a bias term depending only on a measurable characteristic $c(x)$, such as response length, markdown structure, or style (&&&^^^^3all:\3^^^^&&&). In this formulation, reward bias substitution means estimating $b_c^\theta$ and replacing the biased reward or margin with a calibrated surrogate that approximates the latent, bias-free quantity.

Other papers formulate the same idea at different representational levels. Example-based control replaces a handwritten reward with a success-event probability $p^\pi(e_{t+}=1\mid s_t,a_t)$, learned directly from transitions and success examples rather than from an explicit reward function (&&&^^^^3 OR ti:\3^^^^&&&). Bayesian non-negative reward modeling decomposes scalar reward as

$r(x,y)=\theta(x,y)^\top \Phi,$

with instance-specific latent factors $\theta$ and global non-negative factors $\Phi$, so that bias-related factors can be downweighted rather than the whole reward discarded (&&&^^^^3all:\3^^^^7&&&). Representation-learning approaches instead posit latent variables $Z=(Z_C,Z_S)$, where PRESERVED_PLACEHOLDER_^^^^3all:\3query3^^^^ is bias-free and independent of a spurious surrogate PRESERVED_PLACEHOLDER_^^^^3all:\3all:\3^^^^, and then train the downstream reward only on the recovered PRESERVED_PLACEHOLDER_^^^^3all:\3 OR ti:\3^^^^ (&&&^^^^3all:\3^^^^8&&&). Information-theoretic formulations maximize mutual information between reward predictions and preference labels while minimizing mutual information with explicit bias attributes such as length, sycophancy, or format (Li et al., 29 Dec 2025).

A broader implication is that reward bias substitution is not one method but a design pattern. It may act on the task specification, the scalar reward, internal representations, optimization baselines, or even the incentive system around model training. The critical ^^^^3 OR ti:\3query3 OR ti:\3^^^^6 formulation sharpens this further by arguing that single-axis mitigation can rotate optimization pressure onto correlated proxies rather than eliminate it (&&&^^^^3query3^^^^&&&).

^^^^3 OR ti:\3^^^^. Replacing explicit reward specification

One direct form of reward substitution removes the need to specify a reward at all. In example-based control, the environment is treated as an MDP without reward, with a binary event variable PRESERVED_PLACEHOLDER_^^^^3all:\33^^^^ indicating task success. The objective is to maximize the discounted probability of eventual success,

PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^4

using only a distribution of success examples PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^5 and a transition dataset (&&&^^^^3 OR ti:\3^^^^&&&). The key technical object is a future-success classifier PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^6 whose odds ratio equals the desired success probability,

PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^7

This quantity satisfies a data-driven Bellman equation in which the usual reward term is replaced by PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^8, even though that scalar is never explicitly estimated (&&&^^^^3 OR ti:\3^^^^&&&).

The resulting algorithm, Recursive Classification of Examples, alternates between learning the classifier PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^9 from success examples and transitions, and updating the policy to maximize classifier output. In the tabular setting, its expected update is equivalent to value iteration with reward PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3query3^^^^, and tabular RCE converges (&&&^^^^3 OR ti:\3^^^^&&&). Empirically, it outperforms prior methods that first learn an explicit reward and then optimize it.

A related substitution appears in adversarial imitation learning. Standard GAIL commonly derives either always-positive rewards

PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3all:\3^^^^

or always-negative rewards

PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3 OR ti:\3^^^^

and the paper on neutral reward functions argues that these induce survival bias or termination bias depending on environment structure (&&&^^^^3 OR ti:\3^^^^4&&&). The proposed replacement is a neutral log-odds reward,

PRESERVED_PLACEHOLDER_^^^^3 OR ti:\33^^^^

Under oracle-style analysis, this removes both survival bias and termination bias. In Minigrid multiple-terminal tasks, the neutral reward achieves the highest success rates, including PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^4 on GoToDoor and DistShift^^^^3all:\3^^^^, outperforming negative-reward GAIL and DAC in the reported comparisons (&&&^^^^3 OR ti:\3^^^^4&&&).

3. Post-hoc calibration and inference-time intervention

A second major line of work keeps the reward model fixed and substitutes its outputs at inference time. Post-hoc reward calibration assumes the additive decomposition

PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^5

and estimates the bias term from already scored examples, then subtracts it without retraining the RM (&&&^^^^3all:\3^^^^&&&). The basic RC-Mean estimator uses local averaging in feature space, while RC-LWR uses LOWESS-style locally weighted regression with adaptive bandwidth and robustification. For pairwise comparison, the calibrated margin becomes

PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^6

optionally scaled by a calibration constant PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^7 (&&&^^^^3all:\3^^^^&&&).

In the length-bias case, this procedure yields an average gain of about PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^8 RewardBench points across 33 reward models with RC-LWR-Penalty, reduces average absolute reward-length correlation from PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^9 to about $c(x)$^^^^3query3^^^^, and lowers gameability from roughly $c(x)$^^^^3all:\3^^^^–c(x)c(x)^^^^3 OR ti:\3^^^^ to roughly $c(x)$3–$c(x)$4 on strong BT reward models (&&&^^^^3all:\3^^^^&&&). The same framework also extends to markdown features and to two-dimensional calibration over length and markdown count.

FiMi-RM makes the same substitution explicit for length bias, but by fitting a non-linear bias model $c(x)$5 and then training the reward model to be decorrelated from it (&&&^^^^3 OR ti:\3^^^^9&&&). The fitting stage uses a sinusoidal length encoding, a two-layer ResNet, and a combined Pearson-plus-MSE objective. The debiasing stage minimizes

$c(x)$6

so that reward outputs are driven toward low correlation with the learned length-bias component (&&&^^^^3 OR ti:\3^^^^9&&&). On Qwen^^^^3 OR ti:\3^^^^.5-7B, this produces far more balanced reward accuracy between cases where the chosen answer is longer and cases where the rejected answer is longer, and it gives the best length-controlled win rate in both Best-of-$c(x)$7 and DPO settings reported in the paper.

Inference-time interventions can also act inside the reward model rather than on its final scalar. CIRM identifies neurons whose activations are strongly correlated with predefined bias attributes such as length, paragraph count, overlap, exclamation marks, and bold text, and then clamps those neurons to median values at inference time (Shinoda et al., 30 Apr 2026). On the reported setups, this edits less than $c(x)$8 of all neurons in the reward models and allows small ^^^^3 OR ti:\3^^^^B and 7B RMs, when used for preference annotation, to achieve alignment performance comparable to a 7^^^^3query3^^^^B RM on AlpacaEval and MT-Bench (Shinoda et al., 30 Apr 2026). The same paper reports that bias signals are concentrated in early layers.

SteerRM replaces neuron-level editing with sparse autoencoder features. It identifies bias-related SAE features from contrastive pairs and suppresses them at inference time, improving RM-Bench Hard-split accuracy by $c(x)$9 points on average across six reward models while preserving overall performance (Sun et al., 13 Mar 2026). Mechanistic reward shaping applies a related null-space projection to low-complexity bias directions such as length, uncertainty markers, and answer position; across RewardBench-^^^^3 OR ti:\3^^^^, all single-bias and combined interventions are reported as non-inferior to baseline reward quality under a $b_c^\theta$^^^^3query3^^^^-point margin, while substantially reducing the targeted biases (Fein et al., 6 Feb 2026).

4. Factorized, causal, and information-theoretic substitutions

A more structural approach replaces an opaque scalar reward with a disentangled internal representation. Bayesian Non-negative Reward Modeling factorizes reward as $b_c^\theta$^^^^3all:\3^^^^, where $b_c^\theta$^^^^3 OR ti:\3^^^^ is sparse and instance-specific and $b_c^\theta$3 is a sparse global factor vector (&&&^^^^3all:\3^^^^7&&&). The intended effect is “disentanglement-then-debiasing”: local sparsity encourages factor specialization, while global sparsity downweights factors that do not consistently improve likelihood. On RM-Bench Hard, the paper reports that a vanilla BT RM has Pearson correlation between reward and length of about $b_c^\theta$4, while BT-BNRM reduces this to about $b_c^\theta$5 without any explicit length penalty (&&&^^^^3all:\3^^^^7&&&). This makes reward bias substitution concrete at the factor level: bias-related factors remain represented but can be suppressed or assigned near-zero weight.

Information-theoretic debiasing replaces correlation penalties with mutual-information control. DIR maximizes mutual information between reward predictions and preference labels while minimizing mutual information with bias attributes such as response length, sycophancy, and format (Li et al., 29 Dec 2025). The practical loss keeps the standard Bradley–Terry preference term and adds a CLUB-based upper bound for the bias MI. For length bias, DIR reduces the Pearson correlation between reward and length on RM-Bench from $b_c^\theta$6 to $b_c^\theta$7; for format bias, it moves bold and list win-rates from $b_c^\theta$8 and $b_c^\theta$9 under BT to $p^\pi(e_{t+}=1\mid s_t,a_t)$^^^^3query3^^^^ and $p^\pi(e_{t+}=1\mid s_t,a_t)$^^^^3all:\3^^^^, while also improving difficult RewardBench-Filtered subsets such as Chat Hard, Safety, and Reasoning (Li et al., 29 Dec 2025). In PPO experiments, it also yields higher average benchmark accuracy than the reported baselines.

A causally motivated representation-learning framework provides stronger identifiability claims. It assumes observed text $p^\pi(e_{t+}=1\mid s_t,a_t)$^^^^3 OR ti:\3^^^^ is generated from latent variables $p^\pi(e_{t+}=1\mid s_t,a_t)$3, where $p^\pi(e_{t+}=1\mid s_t,a_t)$4 is bias-free and independent of a surrogate $p^\pi(e_{t+}=1\mid s_t,a_t)$5, while $p^\pi(e_{t+}=1\mid s_t,a_t)$6 carries the spurious dependence (&&&^^^^3all:\3^^^^8&&&). Under the stated assumptions, the paper shows that the non-spurious latent subspace is theoretically identifiable from data, and then trains a VAE-like model so that reward is learned only from $p^\pi(e_{t+}=1\mid s_t,a_t)$7. On synthetic data, the recovered latent variables achieve $p^\pi(e_{t+}=1\mid s_t,a_t)$8; on the sycophancy benchmark, worst-case accuracy rises to $p^\pi(e_{t+}=1\mid s_t,a_t)$9 from $r(x,y)=\theta(x,y)^\top \Phi,$^^^^3query3^^^^ for vanilla; on the concept-bias benchmark, worst-case accuracy rises to $r(x,y)=\theta(x,y)^\top \Phi,$^^^^3all:\3^^^^ from $r(x,y)=\theta(x,y)^\top \Phi,$^^^^3 OR ti:\3^^^^ (&&&^^^^3all:\3^^^^8&&&). Here reward bias substitution is literal: the raw textual representation is replaced by a theoretically identified bias-free latent subspace.

A related substitution happens in preference likelihood rather than reward representation. In the cognitive-bias RLHF paper, the standard global rationality parameter $r(x,y)=\theta(x,y)^\top \Phi,$3 is replaced by an instance-dependent

$r(x,y)=\theta(x,y)^\top \Phi,$4

where $r(x,y)=\theta(x,y)^\top \Phi,$5 is an LLM-judged probability that the observed feedback is influenced by cognitive bias (&&&4^^^^3all:\3^^^^&&&). This downweights comparisons likely to reflect conjunction fallacy, anchoring, base-rate neglect, or related biases. On CogBias, the downstream debiased model reaches $r(x,y)=\theta(x,y)^\top \Phi,$6 ground-truth accuracy versus $r(x,y)=\theta(x,y)^\top \Phi,$7 for the vanilla LLM, and under the ^^^^3all:\3^^^^:3 biased-to-ground-truth training ratio it yields $r(x,y)=\theta(x,y)^\top \Phi,$8 versus $r(x,y)=\theta(x,y)^\top \Phi,$9 for the fixed-$\theta$^^^^3query3^^^^ baseline (&&&4^^^^3all:\3^^^^&&&).

5. Optimization-time substitution and system-level reward design

Some methods substitute not the reward itself but the optimization machinery around it. BiasGRPO is a critic-free RLHF method for social-bias mitigation that replaces the PPO value-function baseline with a group-relative baseline computed from a group of sampled completions for the same prompt (Reddy et al., 3 Jun 2026). For rewards $\theta$^^^^3all:\3^^^^, the advantage is

$\theta$^^^^3 OR ti:\3^^^^

The substitution is explicit: the critic $\theta$3 is discarded, and a centered, normalized group-relative signal is used instead. Reported training statistics show reward standard deviation $\theta$4 for BiasGRPO versus $\theta$5 for PPO, alongside better performance on BOLD, RealToxicityPrompts, and BBQ (Reddy et al., 3 Jun 2026).

Reward-Biased Maximum Likelihood Estimation substitutes a pure likelihood objective with a reward-augmented estimator. In finite MDPs, the estimator maximizes a criterion of the form

$\theta$6

and the resulting RBMLE policy attains $\theta$7 regret under the stated assumptions (Mete et al., 2020). In neural contextual bandits, NeuralRBMLE adds a bias term $\theta$8 to the regularized log-likelihood, yielding two variants with $\theta$9 regret (Hung et al., 2022). In both cases, the reward bias term is deliberately inserted into estimation to enforce exploration.

Outside RLHF, substitution also appears in reward engineering for fairness and incentives. In RL-based feature selection, the naive performance reward is replaced by a multi-component reward

$\Phi$^^^^3query3^^^^

combining AUC, direct-bias penalties, indirect proxy-bias penalties, size regularization, and bonuses for preferred features (Khadka et al., 9 Oct 2025). In medical federated learning, Shapley-based reward systems are designed separately for predictive performance and for subgroup bias, and the combined reward system is explicitly intended to prevent a performance-only scheme from transferring model bias against patients to the institutional level (Pandl et al., 2022).

These optimization-level examples suggest that reward bias substitution is not limited to correcting reward-model outputs. It also includes replacing exploitable optimization targets, baselines, or institutional payoffs with objectives that internalize robustness, exploration, or fairness.

6. Audit gaps, failure modes, and unresolved issues

The strongest recent critique is that single-axis mitigation can create the appearance of success while merely redirecting optimization pressure. The ^^^^3 OR ti:\3query3 OR ti:\3^^^^6 paper formalizes mitigation outcomes into a regime taxonomy and proves that successful mitigation, bias substitution, and overcorrection produce identical observables under any audit-distribution scoring, including ranking accuracy and win-rate, even when granted oracle access to the true reward (&&&^^^^3query3^^^^&&&). The reason is a measurement-versus-optimization gap: mitigation is usually audited on a fixed distribution $\Phi$^^^^3all:\3^^^^, whereas policy training shifts to a policy-induced distribution $\Phi$^^^^3 OR ti:\3^^^^.

The paper gives several concrete demonstrations. In language-model RLHF, adding a length penalty during GRPO compresses responses as intended but redirects optimization pressure onto confidence calibration, driving the policy into overconfidence while factual free-form accuracy falls (&&&^^^^3query3^^^^&&&). A published length-debiasing operator can zero reward-length correlation on the audit distribution yet reintroduce bias under best-of-$\Phi$3 selection on three of four SOTA reward models (&&&^^^^3query3^^^^&&&). The same work also reports a length-sycophancy coupling whose direction reverses under human–LLM judge disagreement.

This critique reframes earlier successes. Reported gains such as $\Phi$4 RewardBench points from post-hoc calibration, $\Phi$5 RM-Bench Hard points from SteerRM, or reduced reward–length correlation after FiMi-RM or DIR show that bias proxies can be suppressed on the measured distribution (&&&^^^^3all:\3^^^^&&&, Sun et al., 13 Mar 2026, Li et al., 29 Dec 2025). A plausible implication is that such results are necessary but not sufficient for certifying that optimization pressure has been removed rather than displaced. The ^^^^3 OR ti:\3query3 OR ti:\3^^^^6 analysis therefore prescribes augmenting evaluation with policy-induced distributions and tracking multiple biases jointly, rather than relying on single-axis audit scores alone (&&&^^^^3query3^^^^&&&).

Several limitations recur across the literature. Calibration methods assume that the true reward is approximately independent of the chosen characteristic in expectation; if longer or more detailed answers are genuinely better, full removal can under-reward good outputs (&&&^^^^3all:\3^^^^&&&). Representation-editing methods currently work best for low-complexity biases and have difficulty with entangled effects such as sycophancy or model-style sensitivity (Fein et al., 6 Feb 2026). Causal and information-theoretic approaches require either explicit surrogates, diverse labelers, or carefully specified bias attributes, and they remain limited to observed or modeled biases (&&&^^^^3all:\3^^^^8&&&, Li et al., 29 Dec 2025).

Taken together, the literature treats reward bias substitution both as a remedy and as a diagnostic warning. As a remedy, it replaces flawed reward signals with examples, calibrated scores, factorized latents, invariant representations, group-relative baselines, or fairness-aware objectives. As a warning, it highlights that removing a measured proxy on an audit set does not by itself establish that optimization no longer exploits bias. The topic therefore sits at the intersection of reward modeling, causal representation learning, mechanistic interpretability, and evaluation under distribution shift (&&&^^^^3 OR ti:\3^^^^&&&, &&&^^^^3query3^^^^&&&).