BaseReward in Reward Modeling

• BaseReward is a context-dependent concept that defines both learned reward signals in LLM alignment and implicit rewards from model log-probabilities.
• Recent approaches like CARP refine BaseReward by integrating semantic alignment scores to ensure rewards depend primarily on prompt intent.
• BaseReward spans multiple domains, serving as a benchmark in multimodal reward models and as a primitive scalar in RL theory and incentive mechanisms.

Searching arXiv for the cited BaseReward-related papers and closely related reward-modeling work. Searching (Lu et al., 15 Apr 2026). BaseReward is a context-dependent term used for several related but non-identical objects in reward-centered machine learning and optimization. In LLM alignment, it commonly denotes the learned reward $r_\theta(x,y)$ used to score candidate responses for preference modeling, reinforcement learning from human feedback, or Best-of-$N$ selection, and recent work studies how to refine this signal so that it depends on prompt intent rather than prompt-independent artifacts (Lu et al., 15 Apr 2026). In a different but closely related formulation, BaseReward can denote an implicit reward read directly from a base model’s log-probabilities, such as $r_{\text{base}}(x,y)=\log \pi(y\mid x)$, which exposes value-laden structure inherited from pretraining (Christian et al., 28 Jan 2026). The same label also appears as the proper name of a specific multimodal reward model, as a primitive scalar reward in formal reinforcement-learning theory, and as a baseline component in decentralized incentive schemes (Zhang et al., 19 Sep 2025).

1. Terminological scope

The literature uses “BaseReward” in several technically distinct senses rather than as a single canonical definition. In reward modeling for LLM alignment, it refers to the learned reward $r_\theta(x,y)$ used to train policy models or to select responses during inference; in this usage, the term identifies the core scalar preference signal before auxiliary corrections are applied (Lu et al., 15 Apr 2026). In the implicit-reward formulation, BaseReward is the reward intrinsic to a base policy’s next-token distribution, surfaced directly from logits or log-probabilities rather than from a separately trained reward head (Christian et al., 28 Jan 2026). In multimodal alignment, “BaseReward” is also the name of a concrete Multimodal Reward Model built on a Qwen2.5-VL backbone with an optimized two-layer reward head and a curated mixture of multimodal and text-only preference data (Zhang et al., 19 Sep 2025).

A concise way to organize the term is to separate “base reward as an object” from “BaseReward as a model name.” The first includes primitive or learned scalar objectives that govern optimization, ranking, or incentive allocation. The second denotes a specific benchmark-oriented MRM system. This suggests that the term is best interpreted relationally: its meaning is fixed by the surrounding optimization framework rather than by a universal notation.

Domain	Meaning of BaseReward	Representative source
LLM alignment	Learned reward $r_\theta(x,y)$ for scoring responses	(Lu et al., 15 Apr 2026)
Implicit reward analysis	Log-probability-based reward from a base policy	(Christian et al., 28 Jan 2026)
Multimodal reward modeling	Name of a specific MRM baseline	(Zhang et al., 19 Sep 2025)
RL theory	Primitive scalar transition reward $r(t)$	(Bowling et al., 2022)
Bayesian RL planning	Posterior-mean immediate reward $R_b(s,a)$	(Sorg et al., 2012)
Incentive mechanisms	Constant baseline offset $-\lambda\beta$ in payment	(Witt et al., 2021)

2. BaseReward in LLM alignment and preference modeling

In current alignment pipelines, BaseReward often denotes the learned reward model $r_\theta(x,y)$ used to score candidate completions for policy optimization or decoding. The basic pairwise training objective is Bradley–Terry ranking, with

$$L_{\mathrm{RM}}=-\log \sigma\!\big(r_\theta(x,y^+)-r_\theta(x,y^-)\big),$$

where $N$0 and $N$1 are chosen and rejected responses (Lu et al., 15 Apr 2026). In this setting, BaseReward is the scalar that downstream RLHF or Best-of-$N$2 procedures attempt to maximize.

The difficulty is that a reward model trained only on pairwise labels can exploit shortcuts in the training distribution. The documented failure modes include length bias, sycophancy, formatting artifacts, generic politeness preambles, and other prompt-independent cues that correlate with human preference labels but are not causally tied to the prompt’s intent (Lu et al., 15 Apr 2026). When such cues contaminate the base reward signal, reward hacking becomes possible: responses can obtain higher scores by increasing verbosity or agreeable tone rather than by better satisfying the prompt.

The same paper formalizes this problem through a causal decomposition in which a response combines prompt intention $N$3 and artifact $N$4, with $N$5, $N$6, and $N$7 (Lu et al., 15 Apr 2026). The intended reward should depend on $N$8, not on $N$9. A plausible implication is that BaseReward quality is not exhausted by held-out pairwise accuracy: robustness to prompt-independent perturbations is a structural property of the reward signal itself.

3. Causal refinement of the base reward signal

A recent proposal for refining BaseReward is CARP, which introduces an intent decoder that reconstructs a latent prompt embedding from a candidate answer and uses the reconstruction error as a regularizer during reward-model training (Lu et al., 15 Apr 2026). The decoder predicts $r_{\text{base}}(x,y)=\log \pi(y\mid x)$0, where $r_{\text{base}}(x,y)=\log \pi(y\mid x)$1 is the last-token embedding from the 14th hidden layer of LLaMA-3-8B, from sparse response features $r_{\text{base}}(x,y)=\log \pi(y\mid x)$2 obtained with a pretrained sparse autoencoder and $r_{\text{base}}(x,y)=\log \pi(y\mid x)$3 (Lu et al., 15 Apr 2026). The reconstruction loss is

$r_{\text{base}}(x,y)=\log \pi(y\mid x)$4

The resulting Semantic Alignment Score is

$r_{\text{base}}(x,y)=\log \pi(y\mid x)$5

with lower SAS indicating better alignment with prompt intent (Lu et al., 15 Apr 2026). CARP injects SAS into Bradley–Terry training through

$r_{\text{base}}(x,y)=\log \pi(y\mid x)$6

where $r_{\text{base}}(x,y)=\log \pi(y\mid x)$7 and $r_{\text{base}}(x,y)=\log \pi(y\mid x)$8 (Lu et al., 15 Apr 2026). An equivalent view is

$r_{\text{base}}(x,y)=\log \pi(y\mid x)$9

so BaseReward is refined by subtracting a penalty for deviation between reconstructed intent and prompt embedding (Lu et al., 15 Apr 2026).

The theoretical motivation is that SAS emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. The paper states a high-probability artifacts suppression theorem for the learned decoder parameters $r_\theta(x,y)$0, with bounds involving $r_\theta(x,y)$1 and $r_\theta(x,y)$2, and a prediction artifacts suppression theorem showing that decoder predictions depend primarily on prompt intent $r_\theta(x,y)$3 and are nearly independent from $r_\theta(x,y)$4 (Lu et al., 15 Apr 2026). Empirically, the decoder achieves $r_\theta(x,y)$5 average accuracy on chosen-vs-rewrite discrimination across helpfulness, math, and safety, while remaining near chance on chosen-vs-reject; RewardBench weighted average accuracy on Gemma-2-9B-it rises from $r_\theta(x,y)$6 to $r_\theta(x,y)$7, and Best-of-$r_\theta(x,y)$8 length-controlled win rate on AlpacaEval-2 rises from $r_\theta(x,y)$9 to $r_\theta(x,y)$0 while average output length drops from 1667 to 1611 characters (Lu et al., 15 Apr 2026). The same study reports negligible correlation between SAS and response length, with Pearson $r_\theta(x,y)$1 and $r_\theta(x,y)$2, and robustness to controlled rewrites involving lengthening and mild off-topic drift (Lu et al., 15 Apr 2026).

4. Implicit BaseReward from logits and inherited value structure

A distinct line of work defines BaseReward as an implicit reward that can be read directly from base-model log-probabilities. Starting from the KL-regularized RLHF objective, the paper derives that for any pair of policies $r_\theta(x,y)$3 and $r_\theta(x,y)$4, the second can be interpreted as the reward-finetuned version of the first with

$r_\theta(x,y)$5

and, in the single-model case,

$r_\theta(x,y)$6

with the corresponding pairwise margin

$r_\theta(x,y)$7

(Christian et al., 28 Jan 2026). The paper further proposes the mixture-weighted log-ratio

$r_\theta(x,y)$8

which suppresses large log-ratio values for tokens that neither model would realistically produce (Christian et al., 28 Jan 2026).

This implicit BaseReward is used to study inherited value biases in reward models. Across 10 leading open-weight RMs, the paper reports that Llama-based RMs show a robust preference for agency and Gemma-based RMs a robust preference for communion on positive prompts, with reversal on negative prompts, even when preference data and finetuning are identical (Christian et al., 28 Jan 2026). The statistical evidence includes a three-way interaction between Big Two category, base model family, and prompt valence with $r_\theta(x,y)$9, follow-up permutation-based $r(t)$0-tests with all $r(t)$1 after Bonferroni correction, medium effect sizes with Cohen’s $r(t)$2–$r(t)$3, and representational similarity analysis showing that base-model choice explains $r(t)$4 of variance in token-rank differences (Christian et al., 28 Jan 2026).

The same pattern appears directly in base LLMs and in implicit reward scores. For instruction-tuned Gemma 2 2B versus Llama 3.2 3B, the three-way ANOVA interaction is $r(t)$5, $r(t)$6; for the pre-trained comparison it is $r(t)$7 (Christian et al., 28 Jan 2026). MWLR ranks “Freedom” at the top and “Love” at the bottom for the prompt “What, in one word, is the greatest thing ever?” in the Llama-over-Gemma direction, and across all 21 combinations of instruction-tuned Llama-3 and Gemma-2 models, $r(t)$8, with the gap increasing with model size (Christian et al., 28 Jan 2026). This suggests that BaseReward, when instantiated as base-model logits, is not value-neutral: it encodes pretraining-dependent preferences that persist into downstream reward models.

5. BaseReward as a multimodal reward model

“BaseReward” is also the name of a specific multimodal reward model introduced as a baseline for MLLM alignment (Zhang et al., 19 Sep 2025). In this usage, BaseReward is not merely a generic scalar objective but a concrete MRM architecture. The model adopts Qwen2.5-VL-7B as backbone, replaces the language head with a scalar reward head, and trains on pairwise preference data with the Bradley–Terry loss

$r(t)$9

(Zhang et al., 19 Sep 2025). The reward head is an optimized two-layer MLP with SiLU activation; deeper heads or other activations were inferior in the reported ablations (Zhang et al., 19 Sep 2025).

The training recipe emphasizes data mixture and simplicity. The final system uses seven non-grey datasets totaling about $R_b(s,a)$0M preference pairs, a learning rate selected from $R_b(s,a)$1 with $R_b(s,a)$2 chosen, batch size 128, and 64 Nvidia H100 GPUs (Zhang et al., 19 Sep 2025). Two tested regularizations were not retained: zero-coefficient regularization decreased performance as $R_b(s,a)$3 increased, and length normalization did not help (Zhang et al., 19 Sep 2025). Ensemble variants use

$R_b(s,a)$4

with validation-based weighting or uniform averaging, and validation-based methods did not clearly outperform simple averaging (Zhang et al., 19 Sep 2025).

On MM-RLHF-Reward Bench, BaseReward with Qwen2.5-VL-7B reaches Acc $R_b(s,a)$5 and Acc$R_b(s,a)$6 $R_b(s,a)$7, while the ensemble of Qwen2-VL-7B and Qwen2.5-VL-7B reaches Acc $R_b(s,a)$8 and Acc$R_b(s,a)$9 $-\lambda\beta$0 (Zhang et al., 19 Sep 2025). On VL-Reward Bench, BaseReward with Qwen2.5-VL-7B reaches overall $-\lambda\beta$1, macro $-\lambda\beta$2, and hallucination $-\lambda\beta$3, while the ensemble reaches overall $-\lambda\beta$4 and macro $-\lambda\beta$5 (Zhang et al., 19 Sep 2025). In reinforcement learning with GRPO on a Qwen2.5-VL-3B policy, BaseReward scoring improves multiple downstream metrics, and a hybrid scheme

$-\lambda\beta$6

produces the strongest aggregate gains across perception, reasoning, and conversation (Zhang et al., 19 Sep 2025). The same study reports that BaseReward consistently outperforms R1-Reward in RL due to lower overhead and reliable scalar rewards (Zhang et al., 19 Sep 2025).

6. Formal, algorithmic, and non-LLM usages

Outside LLM alignment, BaseReward appears as the primitive scalar signal in formal analyses of the reward hypothesis. In that framework, the base reward is $-\lambda\beta$7 on transitions $-\lambda\beta$8, and under the Markov Reward Theorem there exist a deterministic Markov reward $-\lambda\beta$9 and a transition-dependent discount $r_\theta(x,y)$0 such that

$r_\theta(x,y)$1

with $r_\theta(x,y)$2 unique up to a positive scale factor (Bowling et al., 2022). This use treats BaseReward as the minimal scalar object that represents a preference relation over distributions of histories, provided Completeness, Transitivity, Independence, Continuity, and Temporal $r_\theta(x,y)$3-Indifference hold (Bowling et al., 2022).

In approximate Bayesian reinforcement learning, planning in the mean MDP uses the posterior-mean immediate reward $r_\theta(x,y)$4 as the base reward (Sorg et al., 2012). A variance-based bonus is then added,

$r_\theta(x,y)$5

and the augmented reward $r_\theta(x,y)$6 yields optimism and polynomial sample-complexity guarantees (Sorg et al., 2012). Here, BaseReward is the exploitation term before exploration bonuses.

A different usage occurs in decentralized federated distillation on blockchain, where the paper does not define a variable named “BaseReward” but identifies the constant payment offset $r_\theta(x,y)$7 as the natural baseline component in each per-sample payment (Witt et al., 2021). The per-sample payment is

$r_\theta(x,y)$8

so the baseline component functions as a floor that deters low-effort participation (Witt et al., 2021). In cryptocurrency mining, an analogous role is played by the total per-block base reward $r_\theta(x,y)$9 in HaPPY-Mine, which is pegged to system hashrate and decreases once total hashrate exceeds a threshold $$L_{\mathrm{RM}}=-\log \sigma\!\big(r_\theta(x,y^+)-r_\theta(x,y^-)\big),$$0 (Kiffer et al., 2021). These uses are structurally different from RLHF, but they preserve the common idea of a primitive scalar incentive from which broader system behavior is induced.

7. Conceptual implications and open technical issues

Across these formulations, BaseReward functions as the scalar interface between preference, policy, and optimization. In reward modeling for LLMs, the central question is whether this scalar depends on prompt intent or on prompt-independent shortcuts; CARP addresses this by constructing a causally grounded refinement $$L_{\mathrm{RM}}=-\log \sigma\!\big(r_\theta(x,y^+)-r_\theta(x,y^-)\big),$$1 and reporting robustness to verbosity and mild off-topic drift (Lu et al., 15 Apr 2026). In implicit-reward analysis, the core issue is that the scalar already present in base logits can encode inherited value biases that preference finetuning only partially mitigates, even with roughly $$L_{\mathrm{RM}}=-\log \sigma\!\big(r_\theta(x,y^+)-r_\theta(x,y^-)\big),$$2k pairs in the reported ablations (Christian et al., 28 Jan 2026).

In multimodal alignment, BaseReward as a named MRM highlights an engineering conclusion: a tuned Naive-RM head, curated multimodal and text-only data, and simple pairwise training can outperform more elaborate paradigms on MM-RLHF-Reward Bench and VL-Reward Bench, while coding-heavy dimensions remain a weakness without dedicated preference data (Zhang et al., 19 Sep 2025). In formal RL theory, the term retains a more foundational meaning as the scalar quantity required to represent goals and purposes under specific axioms (Bowling et al., 2022).

A plausible implication is that debates about “the” base reward often conflate at least three levels: the primitive reward signal supplied by a formal objective, the learned reward model trained from comparisons, and the latent reward already encoded in a base model’s distribution. The cited work indicates that these levels can diverge substantially. When they do, robustness, value inheritance, and downstream policy behavior become properties of the reward substrate rather than mere implementation details.