Noisy Reward Models in Open-Ended NLP

• Noisy reward models are key components for aligning open-ended NLP systems, capturing imperfect human feedback and inherent data artifacts.
• They incorporate strategies like data preprocessing, tree-structured sampling, and iterative neural reward functions to reduce sensitivity to textual noise.
• Robust evaluation protocols, including energy-based and weak supervision techniques, enhance performance across various NLP tasks by defending against reward hacking.

Noisy reward models are a central obstacle when aligning neural language systems for open-ended NLP tasks. These models—whether explicit, implicit, or proxy—provide the signals by which reinforcement learning or preference optimization drives language behaviors. In open-ended and generative contexts, reward signals are notoriously noisy, arising from imperfect human feedback, annotation variance, context ambiguity, or systematic brittleness to data and task artifacts. This article summarizes the empirical characteristics, theoretical analyses, robust modeling architectures, calibration schemes, benchmarking protocols, and mitigation strategies for managing noisy reward models in modern open-ended NLP workflows.

1. Empirical Sensitivity of Transformer Models to Textual Noise

Empirical studies consistently demonstrate that popular transformer-based NLP models (BERT, RoBERTa, ALBERT, XLNet, T5) exhibit systematic degradation across task types as input text is corrupted with even modest levels of noise—principally, spelling mistakes or typos simulating real-world user-generated content. The noise injection procedure is formalized for each data point $x_i \in D_0$ as

$x_{i,k}^{(\text{noise})} = \text{Noise}(x_i, k\%)$

where $k\%$ of characters are randomly replaced with neighboring QWERTY keys, creating noisy datasets $$D_k = \{ (x_{i,k}^{(\text{noise})}, y_i) : \forall i \}$$.

Experimental results show monotonic performance drops with increasing $k$ across standard benchmarks:

• Text classification (IMDB, SST-2, F1): BERT F1 falls from 0.938 (0% noise) to 0.758 (25% noise)
• Textual similarity (STS-B, Pearson/Spearman): Correlation drops from 0.896 to 0.355 for BERT at 25% noise
• Question answering (SQuAD2.0, F1): BERT drops from ≈71.8 to ≈33.0 as noise increases to 25%
• NER (CoNLL-2003, F1): BERT performance drops from ≈94.49 to ≈75.25
• Text summarization (BillSum, ROUGE1-F1): t5-small falls from 57.01 to 43.76 at 25% noise

This trend demonstrates that transformer RMs are highly sensitive to noise accumulation, threatening their reliability in open-ended, user-facing NLP deployments (Bagla et al., 2021).

2. Topological and Structural Generalization of Reward Information

The topology of the preference dataset—how supervision links are structured—directly impacts reward generalization. The “tree-structured preference” sampling, in which response chains share prefixes and dependencies, shapes the induced Bayesian network (IBN) underlying the reward learning process. Analytically, organizing the data as a tree rather than a set of independent chains reduces reward uncertainty by up to $\Theta(\log n / \log\log n)$ (where $n$ is dataset size). This “information compression” results in more robust and less noisy reward estimates, empirically boosting alignment win rates by 65% over chain baselines for conversation, summarization, and math tasks (Qiu et al., 15 Feb 2024). The RLHF pipeline can be viewed as an autoencoding process: reward model training “encodes” human preference distributions, while policy optimization “decodes” this information.

3. Mitigating Noisy Reward Signals: Modeling and Calibration Techniques

A diverse array of methods targets the mitigation of reward noise across the modeling stack:

A. Preprocessing and Model Design

• Sophisticated text cleaning and correction, while nontrivial, can lower input-level noise before model exposure (Bagla et al., 2021).
• Integrating character-level architectures or augmenting training with adversarially noised inputs may improve baseline robustness (Bagla et al., 2021).

B. Neural Reward Function Iteration

• Iteratively-adjusted neural reward functions, trained to discriminate between “solved” and “frontier” states, as in

$$L_v = \frac{1}{|O^-|}\sum_{o \in O^-} \left[R_y(o) + a\right]^2 + \frac{1}{|O^+|}\sum_{o \in O^+} \left[R_y(o) - a\right]^2$$

enable gradual, open-ended skill (and language behavior) discovery and exploration, adapting to the inherent noise present in high-dimensional or expressive domains (Meier et al., 2022).

C. Noise-Contrastive Alignment

• Both InfoNCA and NCA unify reward and preference data via noise-contrastive estimation, calibrating the LM’s likelihoods relative to reward-annotated data. NCA, in particular, optimizes absolute likelihoods, conferring greater stability under noisy or sparse feedback (Chen et al., 8 Feb 2024).

D. Post-hoc Energy-Based Models

• Energy-based post-processing of RM scores employs conflict-aware filtering (removing instances where model and annotation disagree), label-noise-aware contrastive estimation, and hybrid-initialization for refined reward inference. The model explicitly represents a probability density over reward values, improving resilience to annotation noise and slowing reward hacking in RLHF pipelines (Lochab et al., 17 Apr 2025).

E. Weak Supervision and Aggregated Annotation

• Programmatic labeling, using domain heuristics (text length, sentiment, lexical diversity) and confidence-weighted Snorkel-style models, allows the expansion of preference datasets. When baseline data is scarce, weak supervision improves F1, but its efficacy drops with larger, cleaner datasets (Hauptvogel et al., 28 Oct 2024).

F. Targeted Heuristics and Proxy Task Decomposition

• Decomposing open-ended evaluations into targeted proxy-QA pairs (ProxyReward framework) replaces holistic ratings with a checklist derived reward:

$$S(r_i) = \frac{1}{\ell}\sum_{j=1}^\ell F(a'_{ij}, \tilde{a}_{ij})$$

where $F$ is a strict match. This approach outperforms LLM-as-a-judge “global” ratings for long-context, open-ended generation, surpassing even GPT-4-Turbo in objective F1 metrics (Guo et al., 19 Jun 2025).

4. Robustness, Collapse, and Systematic Failure Modes under Noise

Reward model brittleness and collapse have been systematically characterized:

• Reward Collapse: Traditional ranking objectives produce identical reward distributions across prompts in the overparameterized regime, erasing prompt sensitivity. Closed-form analysis shows convergence to prompt-invariant Beta distributions, independent of semantic context. To address this, prompt-aware objective functions modulate utility on a per-prompt basis, restoring variability and avoiding collapse (Song et al., 2023).
• Transformation Robustness: reWordBench benchmarks show that SOTA RMs can drop from 95%+ to below 73% accuracy or even below-random performance under controlled, meaning-preserving perturbations. Explicit regularization (encouraging RM(x, y) ≈ RM(x, ỹ) for paraphrases ỹ) halves such degradation and improves alignment utility (Wu et al., 14 Mar 2025).
• Reward Hacking at Inference: Aggressively overoptimizing for imperfect proxy RMs (via Best-of-$n$, Soft-BoN, Best-of-Poisson sampling) drives the model into “reward hacking” regimes—true rewards plateau and fall, even as proxy/KL divergence rises. The HedgeTune algorithm strategically selects sampling parameters to hedge against this failure, maximizing true utility while automatically halting before hacking onset (Khalaf et al., 24 Jun 2025).

5. Specialized and Endogenous Approaches for Open-Ended Tasks

A. Endogenous, Self-Modifying Reward Functions (RULE)

• Agents can dynamically update their reward functions (coefficients θi) in response to deviation from internal expectations, copying updated “goal priorities” to future generations. This mechanism, governed by

$$E_i(\tau) \leftarrow E_i(\tau) + \delta_E, \quad \theta_i \leftarrow \theta_i + \delta_\theta$$

provides a self-calibrating, evolution-inspired solution to open-ended environments where external reward specification is infeasible or noisy (Bailey, 2 May 2024).

B. Reflection and Internal Self-Judgment

• Models enhanced with internal reflection (self-critique) and chain-of-thought expansion, evaluated by token-level confidence metrics, can resolve ambiguous or under-specified benchmarks by expanding and refining their own reasoning—even in the absence of perfect external rewards (Zhao et al., 21 Nov 2024, Xu et al., 16 Jun 2025).

C. Semantic and Targeted Reward Models

• Lightweight, semantically-focused RMs (e.g., PrefBERT) trained on human ratings of long-form, open-ended answers provide score signals more tightly aligned with content quality than traditional overlap or embedding-based metrics. Open-ended RL training using these models as reward functions produces LLMs that more closely match human preferences across nuanced style, coherence, and informativeness axes (Li et al., 18 Jun 2025).

6. Practical Evaluation, Benchmarking, and Empirical Results

State-of-the-art studies rely on comprehensive evaluation schemes:

• Explicit benchmarks (RewardBench, O$^2$-QA, ProxyQA) stress-test RM robustness, generalization, and reward signal quality across transformations and domains.
• Composite reward designs (as in O$^2$-Searcher) combine formatting, diversity, and factuality metrics into unified reward functions,

$$r_o = \gamma_0 r_{o,\text{fm}} + \gamma_1 r_{o,\text{div}} + \gamma_2 r_{o,\text{f1}}$$

enabling robust RL training under highly variable answer types and levels of inherent noise (Mei et al., 22 May 2025).

• Empirical results consistently indicate that properly regularized, semantically aware, and topology-informed reward models outperform naive or conventional baselines, with up to 20% higher open-ended generation F1 (Guo et al., 19 Jun 2025), markedly improved resistance to style or formatting hacks (Wu et al., 14 Mar 2025), and diminished reward hacking under aggressive sampling (Khalaf et al., 24 Jun 2025).

7. Limitations, Open Problems, and Prospects

While current approaches can substantially mitigate the impact of reward noise, significant challenges persist:

• No universal prescription: Neither model-based, data-centric, nor sampling-time interventions alone suffice for all settings; cohesive pipelines incorporating multiple strategies are required.
• Reward hacking is irreducible under poorly designed proxies or excessive overoptimization: hedging schemes (HedgeTune) and prompt-aware calibration provide partial safeguards only when the true reward can be reliably estimated at calibration time (Khalaf et al., 24 Jun 2025).
• Invariance to input transformations remains imperfect, particularly in settings where context, long-form structure, or surface variation is high (Wu et al., 14 Mar 2025).
• Scaling of weak supervision can introduce instability or label bias when extended to very large datasets without corresponding increases in gold-standard data (Hauptvogel et al., 28 Oct 2024).
• Automated proxy-QA construction (e.g., ProxyReward (Guo et al., 19 Jun 2025)) may propagate biases from generative sources, highlighting the need for future refinement with human feedback.

A plausible implication is that future developments in reward modeling for open-ended NLP will increasingly leverage richer structural/topological information, ongoing self-calibration, and context-sensitive, semantically nuanced scoring—all while embedding robust regularization and hedging techniques to control overfitting and misalignment in the presence of intrinsic noise.

---

Summary Table: Major Families of Noise Mitigation Techniques and Their Key Properties

Category	Representative Methods	Key Strengths
Preprocessing & Cleaning	Spelling correction, filtering	Reduces surface noise
Neural Reward Iteration	Iterative NNs, NCA/InfoNCA	Adapts to feedback
Topological/Structural	Tree-structured RLHF, IBN	Variance reduction
Post hoc Calibration	EBRM, robust RM training, HedgeTune	Defenses vs. hacking
Weak/Proxy Supervision	Heuristics, ProxyReward, PrefBERT	Scalable, targeted
Reflection/Endogenous	RULE, self-critique, R3, RPR	Adaptivity, robustness

These approaches, while distinct in mechanism, are complementary and increasingly integrated in state-of-the-art pipelines for robust, high-fidelity reward modeling in open-ended natural language systems.