Reward Noise in RL and Generative Models

• Reward Noise is the stochastic corruption or variance in observed reward signals, arising from intrinsic and extrinsic factors in environments and models.
• It affects learning dynamics by altering gradient estimation, convergence guarantees, and overall policy performance across reinforcement learning and preference optimization.
• Robust algorithms such as unbiased gradient correction, reward dithering, and label denoising are used to mitigate noise effects and enhance model performance.

Reward noise refers to stochasticity, corruption, or variance in the observed reward signal in reinforcement learning (RL), preference optimization, and generative modeling. Reward noise can be intrinsic to the environment (e.g., stochastic transitions, sensor errors), extrinsically injected for algorithmic purposes (e.g., dithering, exploration), or arise from downstream modeling artifacts (e.g., imperfect human labels, policy-dependent feedback, or automated verifiers). The presence of reward noise fundamentally alters the learning dynamics, gradient estimation, convergence guarantees, and achievable policy performance. This article surveys contemporary definitions, formal models, algorithmic strategies for robustness, and practical implications across classical RL, LLM alignment, diffusion model preference optimization, and multi-agent systems.

1. Formal Models of Reward Noise

Reward noise is classically modeled as a random perturbation to the true reward function. If the environment's ground-truth reward is $r^*(s, a)$, the observed noise-perturbed reward can be written as

$\tilde{r}(s, a) = r^*(s, a) + \varepsilon(s, a)$

where $\varepsilon$ is a random variable. In some settings (e.g., financial RL), the reward is dominated by exogenous stochastic fluctuations in the environment (Goluža et al., 2024). In RL with verifiable rewards (RLVR), the observed reward is modeled as a stochastic channel with a confusion matrix, allowing for asymmetric false positive and false negative rates: $$P(\tilde{r}=1\mid r^*=0)=\rho_0,\qquad P(\tilde{r}=0\mid r^*=1)=\rho_1$$ with $\rho_0$ and $\rho_1$ representing the class-conditional error rates of the verifier (Cai et al., 1 Oct 2025, mansouri et al., 21 Oct 2025, Wang et al., 2018). In continuous settings, noise is often assumed to be zero-mean Gaussian: $\varepsilon \sim \mathcal{N}(0, \sigma^2)$ with $\sigma$ controlling the noise scale (Ma et al., 10 Jun 2025, Vivanti et al., 2019, Azizzadenesheli et al., 2023). In process reward modeling, "policy-induced" label noise arises when reward is estimated by Monte Carlo trajectories, which introduces false positives (incorrect steps leading to correct outcomes via self-correction) and false negatives (correct steps followed by failed completions) (Xie et al., 19 Jan 2026).

In generative modeling with diffusion models, stochasticity is inherent in the initial noise vector and the denoising process, with reward signals (e.g., human preference models) introducing alignment variance and potential label noise at both pixel and latent levels (Wei et al., 23 Jun 2025, Eyring et al., 13 Aug 2025, Eyring et al., 2024, Kasaei et al., 22 Sep 2025, Zhang et al., 3 Feb 2025, Liu et al., 11 Feb 2026).

2. Pathologies and Challenges Induced by Reward Noise

Reward noise induces several learning pathologies:

• Brittleness and Exploration Traps: In Q-learning, variance differences between states lead to the "Boring Areas Trap," where agents become stuck in low-variance regions even if expected reward is lower (Vivanti et al., 2019).
• Manipulative Value Estimation: Value estimators (consultants) favor low-variance rewards, biasing learning toward "boring" regions and potentially suboptimal outcomes (Vivanti et al., 2019).
• Spurious Positive Feedback: In process reward models and generative preference optimization, Monte Carlo estimation and binary outcomes permit "spurious successes," where the system guesses the correct final label while generating invalid explanations or critiques, thus producing reward-label inconsistencies (Wang et al., 12 Jan 2026, Xie et al., 19 Jan 2026).
• False Positives vs. False Negatives: False positives (unintended trajectories incorrectly rewarded) are more detrimental in gradient-based learning than false negatives, as they introduce systematic bias and sluggish convergence (Huang et al., 2024).
• Intractability of Naive Policies: Naive or affine policies in noisy reward selection problems can be arbitrarily suboptimal compared to thresholding or prophet policies, especially under heteroscedastic noise (Azizzadenesheli et al., 2023).

Empirical findings demonstrate large drops in policy performance under even moderate reward noise when no corrective measures are taken (Wang et al., 2018, Cai et al., 1 Oct 2025, mansouri et al., 21 Oct 2025, Wei et al., 23 Jun 2025).

3. Robust Algorithms and Correction Techniques

A range of algorithmic strategies have been developed to address, exploit, or correct for reward noise, including:

Purpose/Setting	Notable Methods	arXiv References
Unbiased gradient estimation	Natarajan correction, backward/forward correction, surrogate rewards	(mansouri et al., 21 Oct 2025, Wang et al., 2018, Cai et al., 1 Oct 2025)
Variance equalization	Adaptive Symmetric Reward Noising (ASRN)	(Vivanti et al., 2019)
Enhanced exploration	Random Reward Perturbation (RRP), reward dithering (ReDit)	(Ma et al., 10 Jun 2025, Wei et al., 23 Jun 2025)
Process label denoising	Reflection-aware correction, Noise-Aware Iterative Training (NAIT)	(Xie et al., 19 Jan 2026)
Baseline subtraction	Imitation-reinforcement feedback in financial RL	(Goluža et al., 2024)
Category- and composite-reward	Category-aware reward selection and compositional reward optimization	(Kasaei et al., 22 Sep 2025, Eyring et al., 2024)
Preference alignment in diffusion	Noise hypernetworks, reward-tilted distributions, latent reward modeling	(Eyring et al., 13 Aug 2025, Zhang et al., 3 Feb 2025, Liu et al., 11 Feb 2026)

Noise Correction and Surrogate Reward

When the reward is subject to known (or estimable) noise channels, an unbiased estimator $\hat{r}$ can be constructed by inverting the confusion matrix or using analytic correction: $$\hat{r} = \frac{\tilde{r} - \rho_0}{1 - \rho_0 - \rho_1}$$ This estimator is guaranteed to be unbiased in expectation and can be integrated into both Q-learning, PPO, and group-relative objectives (mansouri et al., 21 Oct 2025, Cai et al., 1 Oct 2025, Wang et al., 2018).

Exploration via Reward Noise

Deliberate reward noise is utilized to increase the variance of value function estimates, promoting sampling diversity and improved exploration. Random Reward Perturbation (RRP), for example, simply augments each reward by i.i.d. Gaussian noise, resulting in improved sample efficiency and the escape of local optima, especially in sparse-reward settings (Ma et al., 10 Jun 2025). Reward dithering (ReDit) injects random perturbations into discrete, flat reward landscapes in LLM alignment, smoothing gradients and accelerating convergence (Wei et al., 23 Jun 2025).

Reward-Denoising in Diffusion and Multi-Agent RL

In diffusion generative models, reward-guided initial noise optimization techniques (e.g., ReNO, CARINOX) and amortized approaches (Noise Hypernetworks) modulate the input noise distribution to better align model outputs with human preference rewards (Eyring et al., 2024, Eyring et al., 13 Aug 2025, Kasaei et al., 22 Sep 2025). Distributional reinforcement learning for multi-agent systems decomposes global noisy rewards into tractable local mixtures via GMMs and denoising diffusion probabilistic models, enabling per-agent robustness and risk-sensitive optimality (Geng et al., 2023).

4. Statistical and Theoretical Guarantees

Robust reward-noise correction methods provide several theoretical guarantees:

• Policy improvement and optimality: Under proper correction (e.g., Natarajan-style, backward and forward methods), unbiased estimated rewards yield standard convergence results and policy improvement bounds (mansouri et al., 21 Oct 2025, Cai et al., 1 Oct 2025, Wang et al., 2018).
• Variance equalization: ASRN installs symmetry in reward variance without shifting means, preventing state-value traps (Vivanti et al., 2019).
• Sample complexity: Corrected Q-learning preserves sample complexity modulo a multiplicative factor dependent on the noise channel determinant—arising from increased variance but preserved unbiasedness (Wang et al., 2018).
• Distributional RL monotonicity: Gaussian mixture decomposition in multi-agent settings preserves joint monotonicity, so local greedy actions compose to the global optimum (Geng et al., 2023).
• Preference calibration: In step-level preference optimization for diffusion, latent-space, noise-aware reward models produce faster convergence, improved empirical alignment, and reduced computational burden (Zhang et al., 3 Feb 2025, Liu et al., 11 Feb 2026).

5. Practical Implementations and Empirical Results

Reward-noise-aware algorithms are widely validated across RL, LLM alignment, generative modeling, and financial RL:

• RL with perturbed rewards: Surrogate reward correction restores performance losses of up to 80% under $\tilde{r}(s, a) = r^*(s, a) + \varepsilon(s, a)$0 to $\tilde{r}(s, a) = r^*(s, a) + \varepsilon(s, a)$1 error rates in Atari PPO (Wang et al., 2018).
• Policy optimization with noise dither: ReDit matches or exceeds baseline accuracy with 90% fewer training steps and provides stable policy gradients (Wei et al., 23 Jun 2025).
• RLVR with noisy verifiers: Backward and forward corrections yield robust policy improvement under high false negative rates for math reasoning benchmarks (Cai et al., 1 Oct 2025).
• Diffusion image alignment: Noise Hypernetworks and ReNO substantially improve compositional prompt fidelity and generalization at a fraction of the compute cost of explicit test-time optimization (Eyring et al., 13 Aug 2025, Eyring et al., 2024).
• VLM reward noise: The BiMI criterion reduces false positive reward rates from $\tilde{r}(s, a) = r^*(s, a) + \varepsilon(s, a)$2 to $\tilde{r}(s, a) = r^*(s, a) + \varepsilon(s, a)$3 in instruction-following navigation tasks, yielding accelerated RL convergence (Huang et al., 2024).
• Multi-agent distributional RL: Gaussian mixture decomposition with diffusion-augmented synthetic sampling rivals noise-free upper bounds on Multi-Particle Environments and SMAC challenges (Geng et al., 2023).

6. Limitations, Open Problems, and Future Directions

Performance and theoretical guarantees of reward-noise management techniques depend on accurate estimation of noise statistics (confusion matrices, variance structure), stationary reward processes, and well-behaved noise models (e.g., invertible channels, small-movement assumptions in diffusion). Overestimation of noise rates can lead to variance blow-up and decreased policy reliability (mansouri et al., 21 Oct 2025, Cai et al., 1 Oct 2025). Poorly calibrated or non-representative reward models can induce reward hacking and unintended optimization artifacts (Eyring et al., 13 Aug 2025, Eyring et al., 2024). Reflection-aware label correction and iterative relabeling mitigate some of these risks in process reward modeling, though high-variance, discrete, or non-stationary rewards remain structurally challenging (Xie et al., 19 Jan 2026, Wang et al., 12 Jan 2026). Research continues on scalable label-denoising, adaptive noise scheduling, latent reward shaping, and uncertainty quantification for both RL and generative model alignment pipelines.

• "Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models" (Eyring et al., 13 Aug 2025)
• "Towards Robust Process Reward Modeling via Noise-aware Learning" (Xie et al., 19 Jan 2026)
• "Reward Selection with Noisy Observations" (Azizzadenesheli et al., 2023)
• "Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modeling" (Liu et al., 11 Feb 2026)
• "CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration" (Kasaei et al., 22 Sep 2025)
• "The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards" (Huang et al., 2024)
• "Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients" (mansouri et al., 21 Oct 2025)
• "ReDit: Reward Dithering for Improved LLM Policy Optimization" (Wei et al., 23 Jun 2025)
• "Robot See, Robot Do: Imitation Reward for Noisy Financial Environments" (Goluža et al., 2024)
• "Noise-based reward-modulated learning" (Fernández et al., 31 Mar 2025)
• "Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learning" (Geng et al., 2023)
• "Adaptive Symmetric Reward Noising for Reinforcement Learning" (Vivanti et al., 2019)
• "ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization" (Eyring et al., 2024)
• "Reward Modeling from Natural Language Human Feedback" (Wang et al., 12 Jan 2026)
• "Exploration by Random Reward Perturbation" (Ma et al., 10 Jun 2025)
• "Reinforcement Learning with Perturbed Rewards" (Wang et al., 2018)
• "Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization" (Zhang et al., 3 Feb 2025)
• "Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers" (Cai et al., 1 Oct 2025)
• "Reinforcement Learning with Stochastic Reward Machines" (Corazza et al., 16 Oct 2025)