Fairness Reward Model (FRM)

• FRM is a reward model that explicitly conditions outputs on fairness and robustness criteria to prevent bias amplification.
• It incorporates techniques like counterfactual fairness, fairness regularization, and group parity to align reward signals with equitable decision-making.
• Key architectures include trust-scored models, process-level verifiers, and constrained optimization approaches that balance accuracy with fairness.

Searching arXiv for papers on Fairness Reward Models and related reward-fairness frameworks. A Fairness Reward Model (FRM) is a reward model whose outputs are explicitly conditioned on fairness and robustness criteria, instead of only predictive accuracy. In contemporary work, the term covers several closely related constructions: a reward model whose scalar outputs are modulated by a fairness or trust layer in RLHF; a reward model trained under explicit demographic parity, equalized odds, or counterfactual fairness constraints; and a process-level verifier that assigns fairness scores to intermediate reasoning steps and then reweights decision trajectories accordingly (Mathew et al., 27 Aug 2025). Across these formulations, the common purpose is to prevent reward signals from encoding or amplifying spurious correlations, demographic skews, or stereotype-based reasoning, thereby altering either reward estimation, policy optimization, or inference-time aggregation in a fairness-aware direction (Ouyang et al., 29 May 2025).

1. Conceptual scope and development

The modern FRM literature emerged from two lines of work. One line concerns fair reward design in sequential decision-making, where fairness is enforced by modifying the objective or the reward allocation itself. Examples include fairness-aware PPO with retrospective and prospective penalty terms derived from demographic parity, counterfactual fairness, and conditional statistical parity (Malfa et al., 6 Feb 2025); minimum reward guarantee fairness in multi-agent bandits (Manupriya et al., 21 Feb 2025); and RL-based recommender rewards that combine accuracy, fairness, and new-item exploration (Guo et al., 30 Apr 2025). The other line concerns reward-model fairness in RLHF and LLM alignment, where bias is treated as a property of the learned reward function and is addressed during reward-model training, reward calibration, or inference-time reward use (Ouyang et al., 29 May 2025).

Earlier precursors did not use the LLM-era terminology of reward models, but they already treated fairness as a property of reward assignment. FaRM introduced selective fairness and cumulative fairness for information aggregation in spontaneous localized settings, combining report strength, consistency, reliability, and location robustness in a fair reward mechanism (Moti et al., 2019). REFORM introduced gamma-fairness, qualitative fairness, and the Temporal Reputation Model (TERM), then used trustworthy agents’ additional chances of pairing to improve fairness in peer-based mechanisms (Kanaparthy et al., 2021). Fair incentives for repeated engagement imposed fairness within a period and group fairness over time, then showed that the optimal static solution to a deterministic variant is asymptotically optimal for the dynamic problem under fairness constraints (Freund et al., 2021). These systems are not FRMs in the RLHF sense, but they established the now-familiar pattern of coupling reward with explicit fairness structure.

A useful synthesis of the recent literature is that FRMs now appear in at least four recurring forms.

Formulation	Core mechanism	Representative paper
Counterfactual trust-scored reward model	Raw reward is filtered or penalized by drift, uncertainty, fairness violations, error, and counterfactual inconsistency	(Mathew et al., 27 Aug 2025)
Process-level fairness verifier	Step-level fairness scores are aggregated into chain-level weights for decision-making	(Hall et al., 15 Jul 2025)
Resource-allocation fairness reward model	Reward learning maximizes utility together with Fairness Regularization or Fairness Coefficient	(Ouyang et al., 29 May 2025)
In-processing constrained reward model	Reward model is trained under demographic parity, equalized odds, or counterfactual fairness constraints	(Choi et al., 8 Feb 2026)

This multiplicity of meanings is not accidental. It reflects the fact that “fairness reward model” names a function rather than a single architecture: a mechanism that makes reward estimation, reward usage, or reward optimization sensitive to a fairness criterion.

2. Formal objectives and fairness notions

A recurrent starting point is the standard pairwise reward-modeling view from RLHF. With reward model $r_\phi(x,y)$, preference learning uses Bradley–Terry style probabilities and optimizes pairwise correctness on preferred and dispreferred responses (Ouyang et al., 29 May 2025). FRM work departs from this baseline by asserting that local pairwise correctness is insufficient: a reward model may rank preferred answers above dispreferred ones within each pair, yet still allocate reward unfairly across categories, lengths, demographic groups, or other entities (Ouyang et al., 29 May 2025).

One influential formalization recasts reward learning as a resource allocation problem. Let $a_i = r_\phi(y_w) - r_\phi(y_l)$ denote the reward difference allocated to preference pair $i$. Utility is

$$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$

and fairness is measured by the unified fairness functional

$$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$

This yields two FRM objectives: $$\max_{\mathbf{a}} \ U(\mathbf{a}) + \alpha F(\mathbf{a})$$ for Fairness Regularization, and

$$\max_{\mathbf{a}} \ U(\mathbf{a}) \cdot F(\mathbf{a})^{\gamma}$$

for Fairness Coefficient (Ouyang et al., 29 May 2025). With $\tau=-1$, the fairness measure recovers $n\cdot J(\mathbf{a})$, where $J(\mathbf{a})$ is Jain’s fairness index (Ouyang et al., 29 May 2025).

A second formalization uses group fairness of reward models as parity in mean reward across groups. If $a_i = r_\phi(y_w) - r_\phi(y_l)$0 is a reward model and $a_i = r_\phi(y_w) - r_\phi(y_l)$1 is the prompt distribution for group $a_i = r_\phi(y_w) - r_\phi(y_l)$2, group fairness requires

$a_i = r_\phi(y_w) - r_\phi(y_l)$3

This is a demographic parity–style requirement adapted to continuous reward scores, motivated by a benchmark in which all groups’ prompts are expert-written arXiv abstracts and are therefore assumed to have equal ground-truth quality (Song et al., 10 Mar 2025).

A third formalization moves from group-level parity to counterfactual fairness. In the counterfactual reward model of multimodal RLHF, the core counterfactual consistency term is

$a_i = r_\phi(y_w) - r_\phi(y_l)$4

where $a_i = r_\phi(y_w) - r_\phi(y_l)$5 is obtained by perturbing protected attributes while keeping other content fixed (Mathew et al., 27 Aug 2025). High $a_i = r_\phi(y_w) - r_\phi(y_l)$6 indicates that the reward changes significantly when only a protected attribute changes. This makes fairness a property of local invariance under counterfactual perturbation rather than only a property of average group statistics.

Faro integrates these strands into an in-processing FRM with explicit demographic parity (DP), equalized odds (EO), or counterfactual fairness (CF) constraints imposed directly on reward-level preference probabilities $a_i = r_\phi(y_w) - r_\phi(y_l)$7. It defines $a_i = r_\phi(y_w) - r_\phi(y_l)$8-fairness by requiring

$a_i = r_\phi(y_w) - r_\phi(y_l)$9

with $i$0 instantiated by group-wise expected preference probabilities for DP, by group-and-label-conditioned expectations for EO, and by group-and-unrestricted-attribute-conditioned expectations for CF (Choi et al., 8 Feb 2026). This yields a reward model that the paper describes as simultaneously ordinal, cardinal, and fair (Choi et al., 8 Feb 2026).

3. Architectural patterns

The most explicit FRM architecture in multimodal RLHF is the counterfactual reward model with Counterfactual Trust Score (CTS). In that system, a CatBoost classifier plays the role of the base reward model and outputs a scalar reward-like signal $i$1 for multimodal input $i$2 (Mathew et al., 27 Aug 2025). The raw reward is then not trusted as-is. Instead, it is evaluated and modulated by a trust layer whose central quantity is

$i$3

Here $i$4 is a normalized drift score from PSI, JSD, and autoencoder or transformer-autoencoder reconstruction error; $i$5 is average prediction uncertainty; $i$6 is fairness rule violation rate; $i$7 is classification error; and $i$8 is the counterfactual consistency penalty (Mathew et al., 27 Aug 2025). The resulting fairness-aware reward can be supplied to RLHF as

$i$9

or, more generally,

$$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$0

This architecture embeds drift detection, anomaly detection, group fairness, temporal robustness, and counterfactual invariance into a single meta-reward layer.

A different architectural pattern appears in decision-oriented LLMs. There, the FRM is a process-level reward model over reasoning text rather than over final outputs. A chain of thought is segmented into reasoning steps $$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$1, and the model assigns a fairness score $$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$2 to each step (Hall et al., 15 Jul 2025). These are averaged into a chain-level fairness score

$$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$3

converted into weights

$$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$4

and then used for fairness-weighted answer aggregation: $$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$5 The FRM therefore does not alter the base reasoning model’s parameters. It instead functions as an inference-time verifier that down-weights biased trajectories and favors equitable ones (Hall et al., 15 Jul 2025).

A third pattern is purely optimization-centric. Faro keeps the ordinary reward-model architecture—an instruction-tuned LLM with a scalar reward head—but changes its training objective to

$$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$6

where $$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$7 collects differentiable proxy violations for DP, EO, or CF (Choi et al., 8 Feb 2026). In this formulation, the fairness structure is not added as a separate trust layer or inference-time filter; it is internal to reward optimization itself.

These designs suggest three broad engineering choices. Fairness may be attached after reward prediction as a gating scalar, attached during reasoning as a verifier over intermediate steps, or attached within reward learning as a constrained optimization problem. A plausible implication is that FRM is best understood as a family of interfaces between reward and fairness rather than a single model class.

4. Training and deployment paradigms

Recent FRM training methods differ primarily in the supervision signal used to define fairness. In the process-level LLM framework, fairness labels are weakly supervised and assigned to reasoning steps by GPT-4o-mini acting as an LLM judge (Hall et al., 15 Jul 2025). The training set contains about 255,000 individual reasoning steps from 4,395 BBQ questions, with 201,500 steps labeled UNBIASED and 53,500 labeled BIASED (Hall et al., 15 Jul 2025). The FRM, based on LLaMA-3.2-1B-Instruct, is then trained with binary cross-entropy on those step-level labels. This produces a generalizable fairness verifier that transfers across tasks, domains, and model families without additional fine-tuning (Hall et al., 15 Jul 2025).

In the resource-allocation FRM, supervision remains ordinary human preference data, but fairness is injected through the objective. For reward models, the two main losses are

$$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$8

and

$$U(\mathbf{a}) = \mathbb{E}_{a_i \in \mathbf{a}} \left[ \log \sigma(a_i) \right],$$9

The same idea extends to Direct Preference Optimization by defining the allocation vector through policy-reference log-ratios and then applying Fairness Regularization or Fairness Coefficient to the implicit reward (Ouyang et al., 29 May 2025). This makes the FRM a drop-in modification of BT-style reward modeling and DPO-style policy learning.

Faro adopts a Lagrangian constrained approach and optimizes the reward model with ProxyGDA: an inner gradient descent loop on reward parameters $$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$0 and an outer projected gradient ascent loop on dual variables $$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$1 (Choi et al., 8 Feb 2026). Its theoretical contribution is unusually explicit. The framework provides reward-level fairness certificates with controllable slack, proves a bound on fairness drift under KL-regularized fine-tuning, proves fairness transfer from reward to policy, and establishes the existence of a non-empty Pareto frontier between error and fairness (Choi et al., 8 Feb 2026).

Objective-level FRMs also appear in multi-agent RL. Fair-PPO augments the PPO objective with fairness penalties derived from demographic parity, counterfactual fairness, or conditional statistical parity: $$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$2 The fairness term is decomposed into retrospective components based on realized rewards and prospective components based on value estimates (Malfa et al., 6 Feb 2025). This is not a reward model in the narrow RLHF sense, but it implements the same FRM principle: the effective reward is shaped by a fairness scalar before policy improvement.

5. Evaluation methodologies and empirical findings

The clearest evidence that FRMs address a real failure mode comes from direct measurement of reward-model unfairness. A benchmark of eight single-input reward models on 16,000 arXiv title–abstract pairs—2,000 examples in each of eight occupational disciplines—found that all evaluated reward models exhibit statistically significant group unfairness (Song et al., 10 Mar 2025). Group fairness was defined as parity in mean reward across groups, measured with the Normalized Maximum Group Difference

$$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$3

alongside one-way ANOVA and Tukey HSD tests (Song et al., 10 Mar 2025). All eight models had $$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$4 and $$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$5-values $$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$6; every model showed at least 23 of 28 significant pairwise group differences; and the observed NMGD magnitudes ranged from 9.78% for ArmoRM-Llama3-8B-v0.1 to 262.89% for tulu-v2.5-13b-preference-mix-rm (Song et al., 10 Mar 2025). This benchmark situates FRMs as a response to a measurable and systematic property of current reward models rather than a merely hypothetical concern.

The counterfactual reward model for multimodal RLHF provides one of the most detailed end-to-end demonstrations. Evaluated on a multimodal fake-versus-true news dataset with framing bias, class imbalance, and distributional drift, it achieved an accuracy of 89.12% in fake news detection while reducing spurious correlations and unfair reinforcement signals relative to baseline reward models (Mathew et al., 27 Aug 2025). The framework also injects synthetic bias into later sequential batches through bi-subject distribution shifts, framing disturbance, and temporal drift, then shows that $$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$7, $$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$8, $$f_\tau(\mathbf{a}) = \operatorname{sign}(1-\tau)\cdot\left[\sum_{i=1}^n\left(\frac{a_i}{\sum_{j} a_j}\right)^{1-\tau}\right]^{\frac{1}{\tau}}.$$9, and $$\max_{\mathbf{a}} \ U(\mathbf{a}) + \alpha F(\mathbf{a})$$0 rise while $$\max_{\mathbf{a}} \ U(\mathbf{a}) + \alpha F(\mathbf{a})$$1 falls in biased batches (Mathew et al., 27 Aug 2025). This is important because FRM performance is evaluated not only by downstream accuracy but by whether low-trust, biased reward regimes become visible and suppressible.

In LLM decision-making, the process-level FRM was trained on BBQ-derived weak supervision and then deployed without additional fine-tuning on COMPAS, Civil Comments, and Bias in Bios (Hall et al., 15 Jul 2025). On held-out BBQ, it yielded more than 25 percentage points higher accuracy than majority voting across bias categories (Hall et al., 15 Jul 2025). On downstream tasks, it consistently reduced equalized opportunity and equalized odds gaps while matching, or even surpassing, baseline accuracy (Hall et al., 15 Jul 2025). The paper also reports that, for Civil Comments, FRM improved accuracy by about 4 percentage points compared with majority voting, and for Mistral-7B on COMPAS it improved accuracy by more than 10 percentage points over CoT baselines (Hall et al., 15 Jul 2025). These results make the narrower claim that process-level fairness verification can preserve the accuracy benefits of test-time chain-of-thought sampling while reducing stereotype-driven disparities.

Resource-allocation FRMs show a different pattern. On reward modeling, Fairness Regularization and Fairness Coefficient maintain or slightly improve RewardBench and HH-RLHF accuracy while reducing category bias, length bias, and social bias as measured by reward distributions across helpful versus harmless prompts, length bins, and CrowS-Pairs stereotype pairs (Ouyang et al., 29 May 2025). On policy optimization, DPO + FR and DPO + FC improve AlpacaEval2 and MT-Bench relative to plain DPO for both LLaMA3-SFT and Qwen2.5-SFT bases (Ouyang et al., 29 May 2025). Faro reports a parallel outcome at the reward and policy levels: significant reductions in bias and harmful generations, maintenance or improvement of model quality, and theoretical fairness certificates not present in earlier pre- or post-processing approaches (Choi et al., 8 Feb 2026).

6. Limitations, controversies, and terminological ambiguities

FRMs do not resolve the long-standing difficulty that fairness is not uniquely defined. The literature uses demographic parity, equalized odds, counterfactual fairness, minimum reward guarantees, personalized exposure fairness, and process-level stereotype detection, each with a distinct target and failure mode (Choi et al., 8 Feb 2026). This implies that FRM design is inseparable from normative choice. A model that is $$\max_{\mathbf{a}} \ U(\mathbf{a}) + \alpha F(\mathbf{a})$$2-DP fair need not be $$\max_{\mathbf{a}} \ U(\mathbf{a}) + \alpha F(\mathbf{a})$$3-EO fair, and a system that equalizes mean rewards across groups may still violate counterfactual invariance or individual fairness.

Weak supervision is another central limitation. The process-level FRM relies on GPT-4o-mini labels for “BIASED” versus “UNBIASED” reasoning steps, and the paper reports human–human agreement of about 88% but LLM–human agreement of about 75% overall (Hall et al., 15 Jul 2025). The authors also note over-labeling of benign group mentions, under-labeling of subtle bias, and confusion on hallucinated or incoherent reasoning (Hall et al., 15 Jul 2025). This suggests that current FRMs may be strongest at detecting explicit stereotype use and weaker at structural or implicit bias.

The trade-off between fairness and utility remains active rather than resolved. Fair-PPO explicitly observes the Price of Fairness, with fairness improvements accompanied by reduced rewards, even though agents with and without the sensitive attribute renounce comparable amounts of rewards (Malfa et al., 6 Feb 2025). The counterfactual trust-scored FRM describes its weights $$\max_{\mathbf{a}} \ U(\mathbf{a}) + \alpha F(\mathbf{a})$$4 and $$\max_{\mathbf{a}} \ U(\mathbf{a}) + \alpha F(\mathbf{a})$$5 as tunable bias reduction thresholds, implying that stronger fairness enforcement may come at some potential cost in raw accuracy or reward model sharpness (Mathew et al., 27 Aug 2025). Faro formalizes this by proving the existence of a non-empty Pareto frontier between error and fairness (Choi et al., 8 Feb 2026). A plausible implication is that future FRM work will be judged increasingly by how explicitly it characterizes this frontier rather than by fairness gains alone.

Finally, the acronym itself is not stable. In CoRe, FRM denotes Formal Reward Module, a code-based reward component paired with a Residual Reward Module, and the paper is not about fairness per se (Ni et al., 2 Jul 2026). This ambiguity is terminological rather than substantive, but it matters for literature search and interpretation. In the fairness literature proper, FRM most often denotes a reward model whose training, outputs, or deployment are conditioned by fairness criteria; outside that literature, the same acronym may refer to unrelated reward decompositions.

In current usage, then, a Fairness Reward Model is best understood not as a single canonical model, but as a family of fairness-aware reward constructions. Whether implemented as a trust-gated counterfactual reward, a fairness-constrained preference model, or a verifier over reasoning trajectories, its defining characteristic is that reward is treated as a site of fairness intervention rather than a neutral proxy for preference.