Reward in Reinforcement Learning

• Reward is a scalar signal used in reinforcement learning to induce preferences over actions and policies, acting as both task specification and optimization target.
• Reward design and shaping balance signal density, interpretability, and granularity through methods like internal predictive models and dynamic decomposition.
• Advanced reward learning employs preference modeling, backward inference, and ensemble techniques to tackle mis-specification, reward hacking, and alignment challenges.

Reward, in reinforcement learning and adjacent alignment literatures, denotes a scalar signal used to induce preferences over actions, trajectories, or complete policies; in contemporary language-model alignment it also denotes a learned surrogate $R_\phi(x,y)$ that maps a prompt $x$ and candidate completion $y$ to a scalar score approximating human judgments. Across these settings, reward functions serve simultaneously as task specification, optimization target, and source of inductive bias, which is why current work treats reward not as a mere engineering detail but as a formal object with representational limits, algorithmic design choices, and failure modes under misspecification and distribution shift (Bowling et al., 2022, Zhong et al., 12 Apr 2025).

1. Formal status of reward

The classical formulation studied in reinforcement learning is the expected cumulative sum of scalar rewards. In the formalization given in “Settling the Reward Hypothesis,” for observations $O$, actions $A$, policy $\pi$, environment $e$, reward $r:O\times A\to\mathbb{R}$, and transition-dependent discount $\gamma:O\times A\to[0,1]$, the $n$-step value is

$x$0

The same work states the formal reward hypothesis as the claim that any binary preference relation over policies that captures “goals and purposes” can be represented by some pair $x$1 through eventual comparison of these values (Bowling et al., 2022).

That representation is not unconditional. The paper’s Markov Reward Theorem shows that a preference relation over lotteries of histories is representable by a Markov reward and discount iff it satisfies vNM axioms 1–4 together with Temporal $x$2-Indifference; equivalently, there exist $x$3, $x$4, and $x$5 such that

$x$6

This is a precise characterization of when a scalar reward suffices, not a blanket claim that every conceivable objective is reducible to reward maximization (Bowling et al., 2022).

The limits are as important as the theorem. The same source identifies constrained MDPs, risk-sensitive objectives such as variance penalty and CVaR, and multi-objective lexicographic preferences as cases that can violate the required axioms. It also notes that human preferences often violate vNM independence and continuity, and that hyperbolic discounting does not correspond to a single exponential $x$7 unless history is included in state. A plausible implication is that many practically important objectives require either enriched state, non-Markovian reward, or a departure from scalar-reward formulations (Bowling et al., 2022).

2. Reward design and shaping

A central distinction in the literature is between reward as a specification supplied by a designer and reward as an object deliberately shaped to improve learnability. “Internal Model from Observations for Reward Shaping” addresses the case where expert actions are unavailable and only state-only trajectories are observed. It fits an internal predictive model $x$8, $x$9, and defines shaping reward from prediction error: $y$0 with linear, hyperbolic tangent, and Gaussian choices for $y$1. The method produces dense shaping rewards directly from expert state distributions and was evaluated in Reacher, Mover with obstacle, Flappy Bird, and Super Mario Bros. (Kimura et al., 2018).

A more explicit design-oriented treatment appears in “Reward Design for Reinforcement Learning Agents,” which organizes reward construction into teacher-driven, adaptive teacher-driven, and agent-driven regimes. In the non-adaptive ExPRD setting, the expert seeks a new reward $y$2 that preserves invariance of the optimal policy set while maximizing informativeness and maintaining interpretability; in the adaptive EXPADARD setting, the reward is re-optimized against the learner’s current policy; in EXPLORS, the agent meta-learns intrinsic rewards online, combined with an exploration bonus. The thesis emphasizes sparsity, delayed feedback, ambiguity, unintended behaviors, and interpretability as recurring reward-design constraints (Devidze, 27 Mar 2025).

Tool-use settings make the design problem especially granular. “ToolRL” decomposes step-wise reward into a format component and a correctness component, with correctness itself split into tool-name matching, parameter-name matching, and parameter-value matching. The composite step reward is

$y$3

The paper compares static scales, a coarse two-stage schedule, and a fine-grained dynamic schedule that smoothly shifts emphasis from format to correctness over training progress. It reports that fine-grained decomposition and immediate step-wise delivery stabilize credit assignment in multi-step tool trajectories, whereas coarse reward signals and abrupt re-weighting harm generalization (Qian et al., 16 Apr 2025).

Taken together, these works treat reward design as control over signal density, interpretability, temporal placement, and granularity rather than as a single scalar chosen once and left unchanged.

3. Reward learning from preferences, rankings, and backward inference

In modern alignment pipelines, reward is often learned rather than manually specified. The survey “A Comprehensive Survey of Reward Models” defines a reward model as a parameterized function $y$4 and summarizes the dominant preference-learning setup through the Bradley–Terry model,

$y$5

with pairwise log-likelihood loss

$y$6

The same survey distinguishes pairwise comparisons, numeric ratings, multidimensional feedback, active elicitation, explicit reward modeling, implicit approaches such as DPO, and process- versus outcome-level reward granularity (Zhong et al., 12 Apr 2025).

A trajectory-ranking variant appears in “Fantastic Rewards and How to Tame Them,” which casts reward learning for task-oriented dialogue as a listwise ranking problem over batches of dialogue trajectories $y$7 with automatic combined scores $y$8. The trajectory return is

$y$9

and two generalized objectives are introduced. RewardNet minimizes cross-entropy between normalized ground-truth scores and transformed model returns, while RewardMLE fits the permutation likelihood of the ground-truth ordering under a Plackett–Luce-style factorization. The authors motivate RewardMLE as more robust when $O$0 is noisy because it uses rank order rather than absolute scores (Feng et al., 2023).

A markedly different proposal is FLIP, from “Small Reward Models via Backward Inference.” Instead of judging a response directly, FLIP infers the instruction $O$1 that would most plausibly have produced response $O$2, then scores the response by similarity between inferred and original instructions: $O$3 The method is explicitly reference-free and rubric-free, and reframes reward modeling as backward inference rather than forward judgment. This suggests a broader conceptual point: reward learning need not be limited to direct preference prediction if the task admits an alternative generative inversion that is easier for the model family in use (Wang et al., 14 Feb 2026).

The scalar-reward assumption itself has been challenged. “Beyond Reward: Offline Preference-guided Policy Optimization” argues that separately learning a scalar reward can be an information bottleneck in offline preference-based RL and replaces it with a high-dimensional context $O$4 learned through hindsight information matching and a preference modeling objective. The paper’s position is not that reward is irrelevant, but that scalarization may discard structure needed for downstream policy optimization (Kang et al., 2023).

4. Reward-guided optimization

Once a reward or reward model is available, policy optimization typically maximizes reward subject to regularization. In the survey formulation for RLHF, the objective is

$O$5

which captures the standard use of PPO-style KL-constrained updates in language-model alignment (Zhong et al., 12 Apr 2025).

The task-oriented dialogue pipeline of Feng et al. is a concrete two-stage instance. Stage 1 learns $O$6 with RewardNet or RewardMLE; Stage 2 fixes $O$7 and trains a MinTL policy whose action space is full response generation. Because classical REINFORCE is unstable in a vocabulary of approximately $O$8K actions, the paper replaces direct sampling with a Gumbel-Softmax reparameterization and adds weighted behavior cloning,

$O$9

On MultiWOZ 2.0, RewardNet+GS reaches Combined $A$0 and RewardMLE+GS reaches $A$1, exceeding MinTL $A$2, UBAR $A$3, GPT-Critic $A$4, and CASPI $A$5 (Feng et al., 2023).

Reward-guided optimization also appears in reference-free summarization. “Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization” defines a topic reward by extracting topic phrases from each source document and the generated summary, embedding them, computing a cosine-similarity matrix, and then forming document-level Coverage, Precision, and a Topic-$A$6 reward $A$7, averaged across documents into $A$8. This reward is combined with a length reward through inverse-variance weighting and optimized under GRPO (Li et al., 11 Sep 2025).

Multimodal settings use the same pattern with different reward parameterizations. “InternLM-XComposer2.5-Reward” defines

$A$9

over pooled output-token hidden states, trains it with pairwise logistic ranking loss, and then deploys it for PPO fine-tuning, Best-of-$\pi$0 sampling, and data filtering. With $\pi$1, WildVision accuracy rises from $\pi$2 to $\pi$3; the reward model also reports overall $\pi$4 and macro $\pi$5 on VLRewardBench (Zang et al., 21 Jan 2025).

These examples show that reward-guided optimization is now instantiated in token generation, tool calling, dialogue management, summarization, and multimodal generation, with KL penalties, reparameterization tricks, and group-relative normalization used to make the reward signal operationally usable.

5. Evaluation, misspecification, and reward hacking

Because learned rewards are proxies, evaluation has become a distinct research problem. RewardBench provides a benchmark of $\pi$6 manually verified prompt–chosen–rejected triplets across capabilities, instruction following, false refusals, safety, and code, and evaluates over forty reward models using accuracy, AUC-ROC, and calibration. The reported leaderboard shows classifier-style reward models leading overall, with berkeley-nest/Starling-RM-34B at $\pi$7 average accuracy, but the “Chat-Hard” subset remains difficult, and the benchmark analysis highlights reasoning limitations, instruction-following shortcomings, and persistent length bias (Lambert et al., 2024).

Even a well-evaluated reward model may be fragile under misspecified behavior models. “On the Sensitivity of Reward Inference to Misspecified Human Models” formalizes reward inference from demonstrations under a true policy $\pi$8 and a learner’s model policy $\pi$9, measuring misspecification by a weighted KL divergence $e$0 and reward error by $e$1. Theorem 1 shows worst-case instability: arbitrarily small model errors can induce arbitrarily large reward errors. Theorem 2 gives a stability result under strong log-concavity: $e$2 This dual result is important: reward inference is neither uniformly hopeless nor uniformly reliable (Hong et al., 2022).

Reward hacking is the complementary failure mode on the optimization side. “Helping or Herding?” shows that reward models are underspecified: models with similar in-distribution performance can diverge under alignment-induced distribution shift. The paper defines overoptimization as regimes where optimizing one reward model increases its own score while another reward model trained on the same data decreases or stays flat, and studies ensemble aggregation

$e$3

Pretrain-seed ensembles outperform fine-tune-seed ensembles and individual reward models, and they mitigate but do not eliminate reward hacking because shared blind spots persist across ensemble members (Eisenstein et al., 2023).

A common misconception is that better held-out preference accuracy automatically implies safer downstream optimization. The combined evidence from RewardBench, misspecification analysis, and reward-ensemble studies argues against that simplification.

6. Domain-specific reward constructions and recurring tensions

Reward functions vary sharply by domain because they operationalize different task semantics. In technology forecasting, Nazari and Weiss define

$e$4

balancing novelty/divergence, stability/similarity, entropy shifts, and absolute distribution shifts to refine topic models iteratively with tabular Q-learning. In the quantum-communication case study, the method is used to identify, rank, and track emerging topics (Nazari et al., 22 Jan 2025).

In adversarial information retrieval, reward is explicitly strategic. “Topic-oriented Adversarial Attacks against Black-box Neural Ranking Models” defines a reward that either applies a flat penalty $e$5 if the perturbed document fails to overtake anchors for any query in a topic group, or else rewards the maximum surrogate score gap plus a naturalness bonus. The naturalness term differs by edit type—language-model and next-sentence-prediction scores for trigger generation, sentence-similarity preservation for word substitution—and controls the attack-versus-fluency trade-off (Liu et al., 2023).

Creative and human-centered settings expose another tension: reward can steer search without fully determining intent. In a 3D parametric chair-design task modeled as an MDP, goal-aligned and goal-agnostic rewards were compared at every design step. With reward, participants made more actions, acted faster, and optimized goal-aligned more than goal-agnostic scores while preserving diversity across designs; for “unique,” a consensus-based likelihood reward was perceived as less useful than for “cheerful” or “dependable.” This suggests that reward quality depends not only on scalar shape but also on how well the reward captures the semantic structure of the goal (Nath et al., 28 Apr 2026).

Across these domains, several recurring tensions recur in explicit form: dense versus sparse feedback, novelty versus coherence, correctness versus naturalness, interpretability versus expressiveness, and optimization power versus exploitability. The literature therefore treats reward not as a universal primitive with fixed meaning, but as a task-dependent operationalization whose formal validity, learnability, and robustness must be established anew in each setting.