Gaussian Reward Model (GaRM)

• Gaussian Reward Model (GaRM) is a family of approaches that use Gaussian distributions to encode reward uncertainty, smoothness, and tractable likelihoods across various domains.
• GaRM frameworks are applied in tabular Q-learning, spatial reward shaping for UI and robotics, Bayesian bandits, and preference-based RL, demonstrating broad versatility.
• These techniques improve learning dynamics through dense gradient signals, uncertainty quantification, and convergence guarantees under finite second-moment conditions.

Searching arXiv for the cited GaRM-related papers and core references. Gaussian Reward Model (GaRM) denotes a class of reward-modeling formulations in which rewards, latent utilities, targets, or reward-generating processes are represented with Gaussian structure rather than as purely deterministic scalars. Across reinforcement learning, bandits, robotics, GUI grounding, and RLHF, this Gaussian structure appears in several distinct but related forms: unbounded Gaussian reward distributions with finite second moments in tabular Q-learning (Miyamoto et al., 2020); spatial reward kernels defined by Gaussian densities over continuous domains such as screen coordinates (Tang et al., 21 Jul 2025) or robot-centered feature spaces (Kim et al., 2024, Kumra et al., 2021); Bayesian Gaussian beliefs over arm rewards in bandits (Reverdy et al., 2013, Urteaga et al., 2018, Gornet et al., 2024); and distributional reward models that predict both a reward mean and a reward variance for preference learning and policy optimization (Yin et al., 15 Aug 2025, Fang et al., 12 May 2026). Taken together, these works establish GaRM not as a single algorithm but as a modeling family in which Gaussian assumptions supply smoothness, uncertainty quantification, tractable likelihoods, or convergence guarantees.

1. Conceptual scope and formal variants

The broadest common structure in GaRM is that reward is modeled through Gaussian parameters—most often a mean, a variance, or a Gaussian density over a task space. One line of work studies rewards as random variables with unbounded Gaussian support, assuming for each state–action pair that $\mathbb{E}[r(s,a)^2] < \infty$ and thereby allowing $r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$ within standard discounted tabular MDPs (Miyamoto et al., 2020). In that setting, Gaussianity is a motivating example of an admissible unbounded reward law rather than a restrictive parametric requirement.

A second line uses Gaussian functions directly as reward kernels over continuous spaces. GUI-G$^2$ models a target UI element as a 2D Gaussian on the screen plane, with reward computed from the predicted box center and Gaussian overlap terms (Tang et al., 21 Jul 2025). TGRF and TPG define Gaussian-shaped reward functions over distances, action neighborhoods, or pose grids in socially-aware navigation and robotic manipulation, respectively (Kim et al., 2024, Kumra et al., 2021). In these formulations, the Gaussian is not the distribution of observed scalar rewards; it is the reward function itself.

A third line places Gaussian structure on latent reward beliefs. In Gaussian bandits, rewards for arm $i$ are sampled from $\mathcal{N}(m_i,\sigma_s^2)$ and the learner maintains Gaussian posteriors over the unknown mean reward (Reverdy et al., 2013). In nonparametric bandits, Gaussian mixtures serve as flexible per-arm reward models under reward-model uncertainty (Urteaga et al., 2018). In restless bandits with a latent linear Gaussian dynamical system, rewards are linear Gaussian observations of a shared hidden state (Gornet et al., 2024). Here GaRM is fundamentally Bayesian and state-estimation-driven.

A fourth line uses Gaussian latent utilities in preference learning. In generative query suggestion, GaRM outputs $(\mu_s,\sigma_s)$ for each candidate suggestion and models latent utility as $r_s\sim\mathcal{N}(\mu_s,\sigma_s^2)$, with pairwise preference probability obtained from a Gaussian-normalized margin (Yin et al., 15 Aug 2025). In variance-aware reward modeling for RLHF, Gaussian latent utilities are paired with two anchor labels to identify both reward mean and variance from preference data (Fang et al., 12 May 2026). This suggests a modern GaRM includes not only smooth reward prediction but also calibrated uncertainty as a first-class object.

2. Gaussian rewards in reinforcement learning theory

In discounted tabular RL, classical convergence analyses typically assume bounded rewards. The paper "Convergence of Q-value in case of Gaussian rewards" shows that this can be relaxed to the finite second-moment condition

$$\forall (s,a)\in S\times A,\quad \mathbb{E}[r(s,a)^2] < \infty,$$

which explicitly includes Gaussian rewards with $$\mathbb{E}[r(s,a)^2]=\mu(s,a)^2+\sigma^2(s,a)<\infty$$ (Miyamoto et al., 2020). Under finite state and action spaces, discount $\gamma\in(0,1)$, Robbins–Monro step sizes, and sufficient exploration, the Q-learning iterates converge almost surely to $r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$0 in sup norm (Miyamoto et al., 2020).

The significance of this result is technical and foundational. In earlier proofs, bounded rewards were used to obtain bounded TD errors and bounded second moments of the stochastic approximation noise. The relaxed analysis instead proves that finite second moments suffice to control the martingale difference term, allowing the stochastic approximation argument to go through under unbounded but square-integrable rewards (Miyamoto et al., 2020). For GaRM understood as an MDP with Gaussian reward noise, the central theoretical implication is that Gaussianity does not obstruct Q-learning convergence in the tabular discounted setting.

The same paper situates this relaxation as relevant to distributional reinforcement learning and Bayesian reinforcement learning, where Gaussian reward assumptions or Gaussian noise models arise naturally (Miyamoto et al., 2020). It also gives a policy-gradient result for what the paper calls distributed reinforcement learning, showing that differentiation can be interchanged with expectation under Lipschitz conditions on the stochastic network defining return distributions (Miyamoto et al., 2020). This suggests Gaussian reward assumptions function both as a realistic modeling choice and as a bridge toward probabilistic return models, although the paper itself establishes only convergence of the mean Q-function.

The limits are equally clear. The analysis is tabular, with only a partial continuous extension via continuity and a "ripple function" construction, and it does not provide convergence rates or deep-RL guarantees (Miyamoto et al., 2020). Heavy-tailed reward laws without finite second moments remain outside scope. Accordingly, the most precise encyclopedic statement is that Gaussian rewards are theoretically admissible in tabular Q-learning under finite variance, not that all Gaussian-based reward models in function approximation inherit the same convergence guarantee.

3. Gaussian reward functions as smooth shaping mechanisms

In task-level reward design, Gaussian functions are often used to replace sparse, thresholded, or piecewise rewards with dense spatially structured signals. GUI-G$r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$1 is a canonical recent example. For an interface element with bounding box $r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$2, the target is represented as a 2D Gaussian with mean at the element centroid and diagonal covariance determined by element width and height (Tang et al., 21 Jul 2025). The framework combines a point reward,

$r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$3

with a coverage reward given by the Bhattacharyya coefficient between predicted and target Gaussians (Tang et al., 21 Jul 2025). Adaptive variance is defined by $r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$4 and $r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$5 with default $r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$6 (Tang et al., 21 Jul 2025).

The reported empirical effect is a shift from sparse binary supervision to dense continuous optimization. On ScreenSpot-v2, GUI-G$r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$7 achieves 93.3% average accuracy, compared with 87.4% for point reward, 85.8% for IoU reward, and 86.5% for point+IoU; it also exceeds SE-GUI-7B at 90.3% and LPO-8B at 90.5% (Tang et al., 21 Jul 2025). On ScreenSpot-Pro it reaches 47.5%, compared with 38.1% for UI-TARS-72B, a 24.7% relative improvement (Tang et al., 21 Jul 2025). The paper attributes these gains to smooth global gradients, coverage-aware matching, and adaptive variance across UI scales (Tang et al., 21 Jul 2025).

A related but mechanically different use of Gaussian reward shaping appears in socially-aware navigation and manipulation. TGRF defines a normalized Gaussian reward term

$r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$8

where $r(s,a)\sim\mathcal{N}(\mu(s,a),\sigma^2(s,a))$9 sets peak magnitude, $^2$0 the preferred feature value, and $^2$1 the width (Kim et al., 2024). This is used to encode discomfort near humans and progress toward the goal in navigation, reducing the burden of manually balancing heterogeneous reward terms (Kim et al., 2024). In multi-step manipulation, the TPG reward computes a task-progress score and then smooths it over the image grid with an anisotropic 2D Gaussian aligned with the gripper axis, yielding a dense reward map over neighboring action pixels (Kumra et al., 2021).

These uses share three structural properties. First, Gaussian rewards create dense gradients over the entire relevant domain rather than only inside target regions or at terminal states. Second, the variance parameter plays the role of tolerance, scale, or locality. Third, anisotropy and adaptive scaling are often indispensable: GUI-G$^2$2 ties covariance to element dimensions, while TPG uses anisotropic smoothing with ratio $^2$3 (Tang et al., 21 Jul 2025, Kumra et al., 2021). This suggests that in GaRM-as-reward-shaping, covariance design is as important as the choice of mean.

4. Gaussian reward beliefs in bandits and dynamical systems

In bandit theory, Gaussian reward models are longstanding and analytically central. In generalized Gaussian multi-armed bandits, arm $^2$4 yields i.i.d. rewards

$^2$5

with unknown mean and known common variance (Reverdy et al., 2013). The learner maintains Gaussian priors over arm means and updates them by normal-normal conjugacy, then selects arms using an upper credible limit

$^2$6

or a softmax over these UCL values in the stochastic variant (Reverdy et al., 2013). The resulting algorithms achieve logarithmic expected regret, and correlated Gaussian priors can reduce posterior variance and improve performance (Reverdy et al., 2013). In that literature, GaRM is both an environmental model and a cognitive model for human explore–exploit behavior.

Bayesian nonparametric extensions relax the single-Gaussian assumption while preserving Gaussian components. In "Nonparametric Gaussian Mixture Models for the Multi-Armed Bandit", each arm’s reward distribution is modeled as a Dirichlet-process Gaussian mixture, allowing multimodality, heavy tails, and heterogeneous arm-specific reward laws (Urteaga et al., 2018). Thompson sampling is then performed using posterior predictive rewards derived from the learned mixture. The method achieves asymptotic regret

$^2$7

and empirically outperforms several baselines in averaged cumulative regret and regret volatility across Gaussian, mixture, outlier-corrupted, exponential, and real-world settings (Urteaga et al., 2018). Here GaRM becomes a flexible density model rather than a single parametric Gaussian.

A more structured dynamical formulation appears in restless bandits with latent linear Gaussian state dynamics. There the hidden state evolves as

$^2$8

and reward is

$^2$9

so rewards for all arms are correlated through a shared Gaussian state (Gornet et al., 2024). A modified Kalman filter yields linear predictors expressing each arm’s future reward as a linear combination of previously observed rewards, regardless of which arm generated them (Gornet et al., 2024). This paper therefore exemplifies a GaRM in which cross-arm generalization arises from Gaussian latent-state inference rather than from direct arm-wise reward regression.

Across these bandit settings, a common pattern emerges: Gaussian reward models are valuable when uncertainty must be updated sequentially and exploited algorithmically. The precise Gaussian mechanism varies—conjugate priors, Gaussian mixtures, or Kalman filtering—but all three use Gaussian structure to couple estimation and decision-making.

5. Variance-aware Gaussian reward models in preference learning and RLHF

In contemporary alignment work, GaRM often means a reward model that predicts both a reward mean and a reward variance. In generative query suggestion, each suggestion $i$0 under history $i$1 is assigned parameters $i$2 and latent reward

$i$3

(Yin et al., 15 Aug 2025). For a winner–loser pair $i$4, the preference probability is approximated as

$i$5

leading to a pairwise objective with a variance regularizer

$i$6

(Yin et al., 15 Aug 2025). The model is then used in RL via both the mean score and a confidence term derived from $i$7, motivated by a lower bound on the Bhattacharyya distance between winner and loser Gaussians (Yin et al., 15 Aug 2025). On live A/B tests and offline human evaluation, the full RL-GaRM pipeline outperforms Bradley–Terry and paired reward models, reaching a 34.03% relative CTR increase and the best GSB score among the compared systems (Yin et al., 15 Aug 2025).

The more explicit identifiability analysis appears in "Variance-aware Reward Modeling with Anchor Guidance" (Fang et al., 12 May 2026). There latent utility is Gaussian,

$i$8

and pairwise probabilities depend only on the variance-normalized margin

$i$9

This implies non-identifiability under translation of $\mathcal{N}(m_i,\sigma_s^2)$0 and under joint positive scaling of $\mathcal{N}(m_i,\sigma_s^2)$1 from pairwise preferences alone (Fang et al., 12 May 2026). The paper resolves this by adding two anchor labels per response based on thresholds $\mathcal{N}(m_i,\sigma_s^2)$2, proving that the corresponding anchor probabilities are sufficient to identify both reward mean and variance functions (Fang et al., 12 May 2026). It further establishes a non-asymptotic convergence rate for the estimated mean and log-variance functions and shows improved reward modeling and downstream RLHF across several diverging-preference datasets (Fang et al., 12 May 2026).

A different but related probabilistic treatment appears in likelihood reward redistribution. LRR models each per-step reward as Gaussian,

$\mathcal{N}(m_i,\sigma_s^2)$3

and uses a leave-one-out likelihood constructed from the observed episodic return to infer dense per-step rewards (Xiao et al., 20 Mar 2025). The Gaussian negative log-likelihood introduces a natural uncertainty regularization term $\mathcal{N}(m_i,\sigma_s^2)$4, and the paper proves that if variance is fixed and no noise is introduced, the likelihood formulation reduces to conventional MSE-based return decomposition (Xiao et al., 20 Mar 2025). This suggests a broader interpretation of GaRM in RLHF and RL: learning variance is useful not only for calibration but also for credit assignment and robustness.

The common misconception addressed by these works is that merely adding a variance head to a reward model automatically yields meaningful uncertainty. The anchor-guidance results show this is false in pairwise-only settings because the scale is non-identifiable (Fang et al., 12 May 2026). The query-suggestion results show that when variance is regularized and operationalized in RL, it can materially improve downstream optimization (Yin et al., 15 Aug 2025). Together they define a variance-aware GaRM as a model class that must solve both estimation and identification.

6. Methodological synthesis, limitations, and open directions

A coherent synthesis of GaRM across domains has four recurrent design axes. The first is the role of the Gaussian object itself. In some works it is the reward distribution over scalar outcomes (Miyamoto et al., 2020, Reverdy et al., 2013, Yin et al., 15 Aug 2025, Fang et al., 12 May 2026); in others it is a kernelized reward surface over positions or actions (Tang et al., 21 Jul 2025, Kim et al., 2024, Kumra et al., 2021); in others it is a latent-state or mixture model that induces reward uncertainty indirectly (Urteaga et al., 2018, Gornet et al., 2024). The second is the meaning of variance: noise scale in tabular convergence theory, tolerance radius in shaping rewards, posterior uncertainty in bandits, or disagreement/uncertainty in preference models.

The third axis is how Gaussian structure interfaces with optimization. In tabular RL, finite second moments allow stochastic approximation convergence (Miyamoto et al., 2020). In GUI grounding and navigation, Gaussian rewards produce dense gradients and smoother learning dynamics (Tang et al., 21 Jul 2025, Kim et al., 2024). In bandits, Gaussianity enables conjugate updates, credible bounds, Kalman filtering, and nonparametric mixture inference (Reverdy et al., 2013, Urteaga et al., 2018, Gornet et al., 2024). In RLHF-like settings, Gaussian mean–variance models affect both reward prediction and policy optimization, because the learned reward is used to rank, select, or reinforce generations (Yin et al., 15 Aug 2025, Fang et al., 12 May 2026).

The fourth axis is identifiability and calibration. Several Gaussian formulations are immediately well-posed because they rely on observable rewards or fixed priors. Others are not. Pairwise Gaussian reward models without anchors suffer from scale non-identifiability (Fang et al., 12 May 2026). Gaussian reward redistribution with learned variance requires careful handling because uncertainty interacts with the leave-one-out construction and can otherwise be used to absorb errors (Xiao et al., 20 Mar 2025). Spatial Gaussian reward models require covariance calibration: too small a variance yields brittle, steep reward landscapes, while too large a variance flattens them and reduces precision (Tang et al., 21 Jul 2025, Kim et al., 2024).

Open directions are stated explicitly across the cited works. Deep-RL convergence with unbounded Gaussian rewards remains unresolved beyond limited continuity-based extensions (Miyamoto et al., 2020). Spatial GaRM may need mixtures of Gaussians or learned covariances to capture irregular or multimodal targets, especially in cluttered interfaces (Tang et al., 21 Jul 2025). Nonparametric and hierarchical Gaussian reward models in bandits point toward richer cross-task sharing under reward-model uncertainty (Urteaga et al., 2018). In alignment, anchor-guided variance-aware models suggest a path toward pluralistic, uncertainty-calibrated reward learning, but their theory currently assumes fixed feature representations and coarse response-level anchors (Fang et al., 12 May 2026).

In its most general encyclopedic sense, Gaussian Reward Model refers to any reward-modeling scheme that uses Gaussian structure to encode uncertainty, smoothness, scale, or latent dependence. The technical literature shows that this label covers several non-equivalent constructions, but also that the Gaussian choice is rarely arbitrary: it is typically motivated by tractable likelihoods, central-limit considerations, smooth spatial feedback, Bayesian conjugacy, or the need to represent reward uncertainty explicitly.