Counterfactual Reward Model

• Counterfactual Reward Model is a framework that defines reward signals via alternative what-if scenarios to improve reliability and reduce bias.
• It employs techniques such as autoencoders, hindsight critics, and invariance regularization to assign rewards and improve credit assignment.
• The approach enhances off-policy evaluation and safe policy development by pairing synthetic counterfactual transitions with real experience.

Counterfactual reward model denotes a class of reward-learning, reward-assignment, and reward-evaluation methods in which reward signals are defined, regularized, or audited through explicit “what-if” alternatives to observed actions, states, responses, or environments. In current literature, the term spans several distinct constructions: reward assignment for generated counterfactual transitions in off-policy reinforcement learning, hindsight-conditioned critics for counterfactual credit assignment, counterfactual-invariance constraints for preference and reward modeling in RLHF, and off-policy estimators that combine factual rewards with counterfactual annotations or trajectories for safer policy evaluation (Lee et al., 18 Mar 2025, Mesnard et al., 2020, Wang et al., 16 Jan 2025, Mandyam et al., 2024). This breadth suggests that a counterfactual reward model is not a single standardized object, but a family of mechanisms that use counterfactual structure to make reward signals more informative, less biased, or more robust.

1. Conceptual scope and recurring structure

Across these works, a recurring pattern is to start from a factual datum—such as a transition $(s_t,a_t,r_t,s_{t+1})$, a preference tuple $(x_i,y_i^+,y_i^-)$, or an observed trajectory $\tau$—and then define a counterfactual alternative that changes some variable while attempting to preserve the relevant causal context. The counterfactual is then used in one of four ways: to assign reward to synthetic experience, to estimate the contribution of actions to future rewards, to regularize a learned reward model against spurious correlations, or to evaluate a target policy without deployment.

The central technical difficulty is validity. In some settings, validity is guaranteed only under a structural assumption. In Counterfactual Experience Augmentation, reward copying is justified under the bisimulation assumption; in Counterfactual Credit Assignment, hindsight information must be conditionally independent of the action to avoid bias; in causal reward modeling for RLHF, reward predictions are regularized toward counterfactual invariance with respect to nuisance variables; and in off-policy evaluation, the usefulness of counterfactual annotations depends on where they enter the estimator (Lee et al., 18 Mar 2025, Mesnard et al., 2020, Wang et al., 16 Jan 2025, Mandyam et al., 2024).

A common misconception is that any generated “what-if” trajectory is necessarily individualized. Work on counterfactual inference in MDPs shows that once the counterfactual path diverges far enough from the observation, the result can cease to be tailored to the observed episode and instead become interventional rather than counterfactual (Kazemi et al., 2024). A related result establishes that many structural causal models can match the observational and interventional distributions of an MDP while disagreeing on counterfactual predictions, so counterfactual reward estimates are generally not identifiable without additional assumptions or robustness criteria (Lally et al., 19 Feb 2025).

2. Reward construction for counterfactual transitions

A direct construction appears in “Counterfactual experience augmented off-policy reinforcement learning,” where the Counterfactual Experience Augmentation algorithm generates synthetic transitions and then assigns them rewards from real experience (Lee et al., 18 Mar 2025). Its core model is the State Transition Autoencoder, implemented as a Conditional Variational Autoencoder that learns the transition difference

$d_t = s_{t+1}-s_t$

conditioned on state and action. The conditional likelihood is written as

$\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$

with ELBO

$$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$

The total loss is $\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$, where $\mathcal{L}_r$ is a squared reconstruction term and $\mathcal{L}_{kl}$ is the usual Gaussian KL term.

After training, the method samples counterfactual actions $\hat a$. In discrete action spaces, all untried actions are counterfactuals. In continuous spaces, actions are selected by maximum entropy sampling using Gaussian kernel density estimation so that unexplored regions receive coverage. The counterfactual next state is then generated as

$(x_i,y_i^+,y_i^-)$0

The reward-model step is Closest Transition Pair matching. For each generated transition $(x_i,y_i^+,y_i^-)$1, the algorithm finds the real transition whose next state is closest to $(x_i,y_i^+,y_i^-)$2, using Euclidean or Manhattan distance: $(x_i,y_i^+,y_i^-)$3 The resulting augmented experience is $(x_i,y_i^+,y_i^-)$4. In discrete or strongly bisimilar environments this is described as precise; in continuous settings the method relies on approximate matches, clusters, or a tunable threshold for retaining only closely matching counterfactuals (Lee et al., 18 Mar 2025).

The significance of this construction is operational rather than purely semantic. The paper explicitly ties the enlarged replay buffer to learning efficiency via

$(x_i,y_i^+,y_i^-)$5

where $(x_i,y_i^+,y_i^-)$6 is the effective number of unique experiences, factual or counterfactual. It also frames counterfactual reward grounding as an out-of-distribution mitigation mechanism: the agent avoids overfitting to a narrow on-policy distribution because rewards for synthetic transitions are anchored in nearby real transitions rather than invented ex nihilo (Lee et al., 18 Mar 2025).

An earlier but conceptually related line appears in “Counterfactual Control for Free from Generative Models,” which learns the joint distribution $(x_i,y_i^+,y_i^-)$7 without reference to any specific reward function during training and then optimizes arbitrary rewards by gradient descent in latent space at inference time (Guttenberg et al., 2017). This uses counterfactual futures as a substrate for reward evaluation, but the reward itself remains externally specified rather than learned.

3. Counterfactual critics and long-term credit assignment

A second major interpretation of counterfactual reward modeling concerns credit assignment rather than reward assignment. “Counterfactual Credit Assignment in Model-Free Reinforcement Learning” conditions value functions on future events while constraining the hindsight information $(x_i,y_i^+,y_i^-)$8 to be conditionally independent of the action $(x_i,y_i^+,y_i^-)$9 given the state $\tau$0 (Mesnard et al., 2020). Under this condition, the policy-gradient estimator becomes

$\tau$1

and the paper states the variance guarantee

$\tau$2

The practical mechanism is a future-conditional baseline or critic that explains away exogenous trajectory-specific randomness—“luck”—without leaking action information and thereby introducing bias. The independence constraint is monitored through

$\tau$3

when $\tau$4 carries no additional information about the action.

“Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis” sharpens this idea by measuring contributions with respect to rewards or learned representations of rewarding objects rather than future states (Meulemans et al., 2023). Its contribution coefficient is

$\tau$5

The paper argues that measuring contributions with respect to future states, as in HCA, produces spurious estimates and can degrade toward REINFORCE in many relevant environments. By centering the counterfactual question on rewarding outcomes—“Would the agent still have reached this reward if it had taken another action?”—COCOA obtains lower-bias and lower-variance gradients than HCA and common baselines (Meulemans et al., 2023).

Counterfactual reward reasoning also appears in competitive multi-agent RL. In “Competitive Multi-Agent Deep Reinforcement Learning with Counterfactual Thinking,” the agent evaluates $\tau$6 intent actions counterfactually through

$\tau$7

defines regret

$\tau$8

and can use an adjusted competitive reward

$\tau$9

Here the reward model is not a separate preference model; it is a counterfactual evaluation mechanism embedded in the critic and regret computation (Wang et al., 2019).

4. Preference reward models, invariance, and bias mitigation

In LLM alignment and RLHF, counterfactual reward models are primarily used to suppress shortcut learning and spurious correlations. “Beyond Reward Hacking: Causal Rewards for LLM Alignment” defines a causal reward model in which a nuisance variable $d_t = s_{t+1}-s_t$0 should not affect reward predictions under intervention (Wang et al., 16 Jan 2025). Counterfactual invariance is stated as

$d_t = s_{t+1}-s_t$1

for all realizations $d_t = s_{t+1}-s_t$2, and training augments the standard Bradley–Terry preference loss with an MMD regularizer across bins of $d_t = s_{t+1}-s_t$3: $d_t = s_{t+1}-s_t$4 The method is presented as a drop-in replacement for standard reward modeling in RLHF. Reported results include a reduction in the sycophancy-bias metric from $d_t = s_{t+1}-s_t$5 for Vanilla RM to $d_t = s_{t+1}-s_t$6 for Conditional CRM, and a reduction in overall discrimination from $d_t = s_{t+1}-s_t$7 for Vanilla RM to $d_t = s_{t+1}-s_t$8 for Conditional CRM and $d_t = s_{t+1}-s_t$9 for Unconditional CRM (Wang et al., 16 Jan 2025).

“DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity” keeps the original preference labels fixed and instead uses semantics-preserving counterfactual perturbations as diagnostic probes during optimization (Liu et al., 8 Jun 2026). For a preference tuple $\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$0, it measures margin shifts and preference flips under counterfactual variants of the chosen response: $\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$1 These are aggregated into a shortcut sensitivity score $\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$2, and the Bradley–Terry objective is reweighted online by

$\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$3

The paper reports gains on RM-Bench “Hard” splits, including Qwen3-4B $\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$4 points $\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$5, Qwen3-8B $\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$6, and Qwen3-4B $\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$7 points on Safety evaluations, with computational overhead of about $\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$8 compared to vanilla Bradley–Terry training (Liu et al., 8 Jun 2026).

“Fairness Aware Reward Optimization” extends this line by incorporating demographic parity, equalized odds, and counterfactual fairness directly into reward-model training (Choi et al., 8 Feb 2026). Its counterfactual fairness violation is defined by

$\log p(d|a)=\log\int p(d|z,a)p(z|a)dz,$9

with Lagrangian optimization over fairness constraints. The paper states reward-level fairness certificates, a formal characterization of the accuracy-fairness trade-off under KL-regularized fine-tuning, fairness transfer from reward to policy, and a non-empty Pareto frontier. On ACS Employment, Faro-cf reduces $$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$0 from $$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$1 to $$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$2 and $$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$3 from $$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$4 to $$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$5 relative to Fair-Bayes; on HolisticBias, Gemma base toxicity drops from $$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$6 to $$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$7 for Faro-cf (Choi et al., 8 Feb 2026).

Multimodal variants make the counterfactual signal itself part of the reward. “Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning” proposes a Counterfactual Trust Score

$$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$8

combining counterfactual shifts, reconstruction uncertainty, fairness rule violations, temporal reward shifts, and counterfactual consistency penalties; the reported fake-news detection accuracy is $$\log p(d|a)\geq \mathbb{E}_{q(z|d,a)}[\log p(d|z,a)]-\mathbb{D}_{KL}[q(z|d,a)||p(z|a)].$$9 (Mathew et al., 27 Aug 2025). In video RL post-training, “Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learning” introduces Counterfactual Relation Reward between original and transformed videos so that answers change for dynamic questions and remain unchanged for static questions; on Qwen3-VL-8B, CRPO improves DyBench P-Acc by $\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$0 and TimeBlind I-Acc by $\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$1 over the base model (Du et al., 21 May 2026).

5. Off-policy evaluation, contextual bandits, and robust counterfactual policy analysis

Counterfactual reward models also play a central role in off-policy evaluation. “CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation” studies contextual bandits with expert-annotated counterfactual rewards $\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$2 and asks where these annotations should enter a doubly robust estimator (Mandyam et al., 2024). The paper proposes three variants: DM$\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$3-IS, DM-DMIS, and DM$\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$4-DMIS. Its theoretical result is precise: when annotations are imperfect, using them only in the direct-method reward model is best; using them in the importance-sampling part introduces bias more readily. The practical recommendation is therefore to use counterfactual annotations only to train the reward model within the DM component unless annotation bias is demonstrably absent (Mandyam et al., 2024).

“Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models” formulates POMDP trajectories as structural causal models, infers posterior exogenous noise from an observed episode, and replays the episode under a target policy with that noise fixed to obtain paired counterfactual trajectories (Oberst et al., 2019). This yields an episode-level decomposition of expected reward differences that is explicitly intended for off-policy “debugging,” with a sepsis-management case study showing how trajectory-level counterfactual analysis can surface dangerous policy behavior hidden by aggregate OPE.

Sequential recommendation introduces another reward-model complication: reward interactions across positions in a slate. “Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions” proposes Reward Interaction Inverse Propensity Scoring, an asymptotically unbiased estimator that exploits a causal graph with Markovian reward dependencies (McInerney et al., 2020). In a live music-recommendation A/B setting, RIPS achieves lower RMSE than IIPS and standard IPS; for random logging, the reported RMSE values are $\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$5 for RIPS, $\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$6 for IIPS, and $\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$7 for IPS (McInerney et al., 2020).

In offline contextual bandits, “Offline Contextual Bandit with Counterfactual Sample Identification” reframes direct reward modeling as a discriminative task: given a positive outcome, identify which action led to it against a counterfactual action sampled from the logging policy (Gilotte et al., 3 Sep 2025). Under uniform logging, the paper derives

$\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$8

so the learned classifier preserves the action ranking induced by expected reward. In synthetic experiments with $\mathcal{L}=\mathcal{L}_r+\mathcal{L}_{kl}$9 samples, CSI-expect reaches $\mathcal{L}_r$0 normalized reward versus $\mathcal{L}_r$1 for DM and $\mathcal{L}_r$2 for LS-IPS; in a banner-design production system, CSI with all features achieves $\mathcal{L}_r$3 clicks in IPS evaluation versus $\mathcal{L}_r$4 for DM (Gilotte et al., 3 Sep 2025).

Robust policy analysis pushes further by treating the counterfactual reward evaluation problem itself as uncertain. “Robust Counterfactual Inference in Markov Decision Processes” computes closed-form lower and upper bounds on counterfactual transition probabilities across all compatible causal models and then solves for policies that maximize worst-case expected reward in the resulting interval counterfactual MDP (Lally et al., 19 Feb 2025). This converts counterfactual reward estimation from a point prediction problem into a maximin optimization problem.

6. Interpretability, influence constraints, and adjacent formalisms

Counterfactual reward models are not only training objects; they are also explanatory objects. “Explaining Learned Reward Functions with Counterfactual Trajectories” defines Counterfactual Trajectory Explanations as pairs $\mathcal{L}_r$5 that contrast an original and a counterfactual partial trajectory under a learned reward function (Wehner et al., 2024). The paper derives six quality criteria—Validity, Proximity, Diversity, State Importance, Realisticness, and Sparsity—and combines them in a scalar score

$\mathcal{L}_r$6

A Monte-Carlo-based algorithm then searches for explanations optimizing these criteria. The reported result is that CTEs are informative for a proxy-human model, improving similarity between its predictions and the reward function on unseen and out-of-distribution trajectories (Wehner et al., 2024).

A second interpretive issue is whether the counterfactual remains genuinely tied to the observation. “Counterfactual Influence in Markov Decision Processes” formalizes influence by comparing counterfactual and interventional transition distributions and shows that disjoint support implies loss of individualized influence (Kazemi et al., 2024). The paper then constructs influence-constrained counterfactual MDPs by identifying $\mathcal{L}_r$7-step influenced states, extending them to $\mathcal{L}_r$8-influenced states with reverse BFS, pruning the counterfactual MDP, and solving for $\mathcal{L}_r$9-CF policies. This provides an explicit optimality–influence trade-off: low $\mathcal{L}_{kl}$0 preserves stronger personalization but restricts achievable reward; high $\mathcal{L}_{kl}$1 weakens influence while allowing more optimal policies (Kazemi et al., 2024).

Adjacent formalisms extend counterfactual reward reasoning beyond ordinary reward predictors. “Counterfactual Planning in AGI Systems” uses a learning world and a distinct planning world, computes the action that maximizes expected utility in the planning world, and executes that action in the real world; this is used to design an emergency stop button, a safety interlock, an input terminal for iterative reward-function improvement, and a counterfactual oracle (Holtman, 2021). “About Time: Model-free Reinforcement Learning with Timed Reward Machines” augments Q-learning with counterfactual-imagining heuristics over delays, timed automaton states, and clock valuations, and reports that counterfactual imagining improves discounted reward and episode completion time in Taxi and Frozen Lake benchmarks (Majumdar et al., 19 Dec 2025). These works indicate that the notion of a counterfactual reward model can extend from a scalar predictor to a broader reward-bearing structure in planning or automata-based control.

Taken together, the literature presents counterfactual reward modeling as a technical response to three recurring failures of conventional reward learning: narrow data support, weak or noisy credit assignment, and spurious reward correlations. The main open questions identified by the surveyed work are not whether counterfactuals are useful, but which counterfactuals are identifiable, which remain under the influence of the observation, and which assumptions—bisimulation, action-independent hindsight, fairness constraints, or robust interval bounds—are appropriate for a given domain (Lee et al., 18 Mar 2025, Mesnard et al., 2020, Lally et al., 19 Feb 2025).