Bayesian Reward Models in Decision Making

• Bayesian reward models are statistical approaches that express uncertainty in reward functions through probability distributions, facilitating principled inference in decision-making systems.
• They integrate techniques such as Bayesian inverse reinforcement learning, nonparametric methods, and Laplace approximations to efficiently infer and optimize reward structures.
• These models enhance robust and safe policy synthesis by mitigating reward hacking and adapting to uncertainty in diverse applications like imitation learning, recommendation, and LLM alignment.

Bayesian reward models constitute a fundamental class of statistical models in sequential decision making, reinforcement learning, imitation learning, and preference-based optimization, where the purpose is to infer, represent, or act upon reward functions under uncertainty. Bayesian reward models formalize the uncertainty inherent in the reward structure—be it from lack of data, misspecification, or subjective human feedback—by expressing beliefs as probability distributions, thereby enabling a principled approach to uncertainty quantification, robust policy synthesis, and safe exploration.

1. Bayesian Modeling of Reward Functions

Bayesian reward models encode the uncertainty over reward functions via explicit probability distributions. The reward function parameters (such as weights in a linear reward feature map, neural network weights, or finite-state reward machines) are treated as latent random variables. Posterior distributions over these parameters are updated from prior beliefs by observing data, which could be demonstrations, preferences, or direct feedback.

Formally, let $\theta$ parameterize the reward function $R_\theta$ and $\mathcal{D}$ denote observed data (demonstrations, preferences, etc.). The posterior is given by Bayes’ rule:

$$P(\theta \mid \mathcal{D}) \propto P(\mathcal{D} \mid \theta) P(\theta)$$

Likelihood models typically instantiate agent rationality and may involve softmax/Boltzmann policies over the value or Q-function induced by $R_\theta$, or pairwise preference models over trajectories. For settings with non-Markovian reward dependencies, the reward is defined over histories or via reward machines, and the observed data must be augmented to maintain correct sufficient statistics (Topper et al., 20 Jun 2024).

In large-scale reward inference or alignment of LLMs, the Bayesian approach is adapted via scalable approximations such as Laplace approximations over low-rank adaptation parameters (Laplace-LoRA), providing Gaussian posteriors over the reward model output (Yang et al., 20 Feb 2024).

2. Methodologies and Computational Frameworks

Bayesian reward models have been instantiated in diverse algorithmic settings:

• Bayesian Inverse Reinforcement Learning (BIRL): Recovers a posterior over reward functions given agent behavior, using likelihoods based on optimality models and priors over reward parameters. The classical BIRL formulation relies on computing the probability of expert action under a softmax policy, requiring repeated forward RL computation for each reward hypothesis. Recent advances propose operating directly in Q-value space to avoid forward planning at every inference step, as in ValueWalk, where the BeLLMan equation is analytically inverted to yield the reward from a sampled Q-function. Hamiltonian Monte Carlo facilitates efficient posterior sampling in high-dimensional Q-space (Bajgar et al., 15 Jul 2024).
• Nonparametric Bayesian Methods: To handle reward model uncertainty (where the true reward likelihood is complex or unknown), nonparametric models such as Dirichlet process mixtures enable adaptive complexity, particularly in multi-armed bandit settings (Urteaga et al., 2018).
• Preference-Based Bayesian Reward Learning: Bayesian Reward Extrapolation (B-REX) employs a Bradley–Terry likelihood over pairwise trajectory preferences and leverages successor features for computationally efficient posterior sampling (Brown et al., 2019, Brown et al., 2020). This model bypasses repeated MDP solutions and supports scalable Bayesian inference for high-dimensional settings such as visual imitation learning.
• Bayesian Optimization for Reward Search: In ill-posed settings (e.g., IRL with many policy-invariant rewards), Bayesian optimization, augmented with projection and kernel methods to account for policy equivalence, enables efficient exploration and quantification of ambiguity in the reward space (Balakrishnan et al., 2020).
• Laplace Approximations in LLM Alignment: Bayesian reward models for LLM alignment employ Laplace approximation over LoRA parameters fine-tuned from human feedback data, yielding uncertainty-aware reward predictions for large neural models (Yang et al., 20 Feb 2024).
• Bayesian Reward Model Ensembles in RLHF: Ensembles of reward models, each head representing a sampled reward function with uncertainty quantified by output variance, are used in robust RLHF pipelines. The policy objective is constructed to trade off performance under the nominal reward with robustness under the worst-case sampled reward (Yan et al., 18 Sep 2024).

3. Handling Uncertainty and Robustness

Quantifying and leveraging uncertainty is central to Bayesian reward modeling. Posterior uncertainty is used to:

• Mitigate overoptimization ("reward hacking"): In LLM alignment, uncertainty-aware reward models penalize deviations from the training distribution, mitigating reward hacking by adjusting the final reward as

$$\tilde{r}(x, y) = r_{\mathrm{MAP}}(x, y) - k \cdot \sqrt{\Lambda(x, y)}$$

where $\Lambda(x, y)$ is the posterior variance estimated via Laplace approximations (Yang et al., 20 Feb 2024).

• Establish robust objectives: In RLHF, optimizing with respect to the ensemble minimum or a convex combination thereof provides robustness to reward model misspecification (Yan et al., 18 Sep 2024). This is expressed in objective functions of the form

$$J_\lambda(\theta) := \lambda J_{\mathrm{perform}}(\theta) + (1-\lambda) J_{\mathrm{robust}}(\theta)$$

where $J_{\mathrm{robust}}$ is a minimum over the uncertainty set defined by the Bayesian ensemble (Yan et al., 18 Sep 2024).

• Safety and risk-aversion: In risk-averse Bayesian reward learning, uncertainty over reward parameters is quantified (e.g., by marginal entropy) and incorporated into decision-making by selecting weights that minimize exposure to high-uncertainty (possibly hazardous) outcomes (Ellis et al., 2021). This approach is particularly important in robotic navigation under limited demonstration coverage or distributional shift.

4. Computational Considerations and Efficiency

While Bayesian reward models provide strong uncertainty quantification, computational challenges arise due to the need for repeated planning (solving the BeLLMan equation) and intractable posteriors in high-dimensional reward parameter spaces. Key algorithmic advancements include:

• Posterior sampling in Q-space: ValueWalk shifts inference to Q-space, massively accelerating sampling by reversing the standard "reward-to-value" computation (Bajgar et al., 15 Jul 2024).
• Successor features and neural embedding caching: B-REX and Bayesian REX decouple feature extraction from reward inference and fix all network layers except the final reward linear map, allowing efficient MCMC sampling (Brown et al., 2019, Brown et al., 2020).
• Laplace and Gaussian process approximations: Laplace approximations (Laplace-LoRA) enable tractable posteriors for very large neural reward models (Yang et al., 20 Feb 2024). Gaussian processes, with specialized kernels, provide sample-efficient exploration in Bayesian optimization for IRL (Balakrishnan et al., 2020).
• Bi-level and uncertainty-aware optimization: In automated reward engineering, bi-level structures decouple design logic (e.g., with LLMs) from hyperparameter optimization (e.g., via uncertainty-aware Bayesian optimization with modified acquisition functions and anisotropic kernels) for increased sample efficiency (Yang et al., 3 Jul 2025, Koo et al., 22 Apr 2025).

5. Applications and Integration Across Domains

Bayesian reward models have seen extensive practical application in areas that include:

• Active Learning and Bayesian Optimization: By capturing both the expected reward and model uncertainty, Bayesian reward models underlie nonmyopic exploration in environmental sensing, energy harvesting, and experimental design (Ling et al., 2015).
• Bandit Algorithms: Nonparametric and geometric Bayesian models provide flexibility and improved regret in bandit problems, supporting both exploration–exploitation and pure exploration regimes (Basu et al., 2018, Urteaga et al., 2018).
• Slate Recommendation: Bayesian models integrating both reward and rank signals in recommender systems optimize for non-personalized slates, producing more accurate item scoring at scale (Aouali et al., 2021).
• Safe and Robust Imitation Learning: Bayesian IRL and preference-based extensions underpin safe imitation policies and enable high-confidence performance bounds critical for risk-sensitive applications (e.g., robotics, autonomous navigation), as well as diagnostic tools for detecting reward hacking (Brown et al., 2019, Brown et al., 2020, Ellis et al., 2021).
• LLM Alignment and RLHF: Bayesian reward modeling is central to LLM alignment, mitigating reward overoptimization in BoN sampling and providing more robust RLHF by leveraging uncertainty-aware ensembles (Yang et al., 20 Feb 2024, Yan et al., 18 Sep 2024, Koo et al., 22 Apr 2025).
• Reward Engineering: The use of Bayesian optimization, often integrated with explainability and uncertainty quantification, enables automated, data-efficient discovery and tuning of reward functions in RL systems (Koo et al., 22 Apr 2025, Yang et al., 3 Jul 2025).

6. Theoretical Properties, Guarantees, and Future Directions

Bayesian reward models support statistically grounded uncertainty estimation and robust policy optimization backed by precise performance analysis:

• Regret and performance bounds: Analysis using information-theoretic tools (e.g., relative entropy, Wasserstein distance) allows bounding minimum Bayesian regret and quantifying the inherent difficulty of learning under reward/function uncertainty (Gouverneur et al., 2022).
• Asymptotic Bayes-optimality: Supervised reward inference, when framed under mild assumptions, is asymptotically Bayes-optimal: the inferred reward converges uniformly to the conditional expectation given observed behavior, equating to the Bayes-optimal estimator under quadratic loss (Schwarzer et al., 25 Feb 2025).
• Optimality and invariance: Dense reward shaping using explainability methods such as SHAP and LIME maintains the optimal policy under the original sparse reward, so long as the shaping function is additive and potential-based (Koo et al., 22 Apr 2025).
• Unidentifiability and prior structure: Joint estimation of reward and subjective dynamics in model-based Bayesian IRL exposes the fundamental unidentifiability of the problem in the absence of informative priors. Rigorous regularization, such as controlling belief in expert dynamics accuracy, is necessary for robust inference (Wei et al., 2023).
• Non-Markovian reward modeling: Bayesian IRL over reward machines, where rewards depend on observation history, extends Bayesian reward models to settings with temporally extended or logical reward structures, with posterior inference formulated over the space of RMs using history-augmented likelihoods and novel annealing-based MCMC (Topper et al., 20 Jun 2024).

Prospective research directions include improving the scalability of fully Bayesian IRL in high-dimensional or continuous domains, integrating richer uncertainty quantification with planning and exploration, advancing ensemble and robustification techniques in LLM alignment, and developing frameworks that directly leverage epistemic uncertainty for active data acquisition and autonomous system safety.