Proxy Reward Sufficiency

• Proxy reward sufficiency is a criterion ensuring that improvements in the tractable proxy reward reliably mirror gains in the true, often inaccessible, objective.
• It underpins reinforcement learning, RLHF, and retrieval-augmented generation by diagnosing and mitigating reward hacking through both theoretical and empirical metrics.
• Empirical evaluations and targeted feedback strategies help address limitations such as reward hacking, scaling challenges, and the need for domain-aware, human-informed designs.

Proxy reward sufficiency is a central but subtle criterion in reinforcement learning, RLHF, and retrieval-augmented generation: it concerns the conditions under which optimizing a tractable, empirically computable “proxy” reward guarantees monotonic or safe improvement under the true, usually inaccessible, objective. Because real-world objectives are unobservable or misspecified, practitioners rely on proxy rewards—learned, human-annotated, or programmatically constructed—to drive optimization. However, when proxy rewards are insufficient, optimization can produce “reward hacking”: behaviors that maximize the proxy signal while degrading true performance or alignment. This entry synthesizes theoretical formalisms, algorithmic strategies, empirical findings, and limitations of proxy reward sufficiency as documented in contemporary literature.

1. Formal Characterizations and Theoretical Limits

At its core, proxy reward sufficiency requires that, over the policy class optimized, improving the proxy reward cannot decrease the expected value of the true reward. Formally, for a true reward $\mathcal{R}_{\rm true}$ and a proxy $\mathcal{R}_{\rm proxy}$ in an MDP, sufficiency implies

$$J_{\rm proxy}(\pi) \geq J_{\rm proxy}(\pi') \implies J_{\rm true}(\pi) \geq J_{\rm true}(\pi')\;\;\forall \pi, \pi'$$

where $J(\pi) = \langle \mathcal{R}, F^{\pi}\rangle$ is the expected return under occupancy counts $F^\pi$ (Skalse et al., 2022).

However, linearly parameterized rewards over the full policy class admit severe impossibility results:

• Over all fully supported (interior) stochastic policies, any nontrivial proxy is hackable unless it is simply a positive scalar multiple of the true reward.
• Only on restricted, finite policy classes does the existence of nontrivial unhackable proxies or simplifications (coarser reward functions that collapse distinctions without introducing incentive reversals) become possible; the requisite span condition involves occupancy measures and the dimension of “tied” policies (Skalse et al., 2022).

Similar impossibility manifests in RLHF: reward models (RMs) trained from human preference data only approximate the ground-truth utility $U^*$, and thus optimizing them can induce undesired behavior if their value surfaces are not perfectly aligned with $U^*$ (Elle, 7 Oct 2025, Gao et al., 2022, Khalaf et al., 24 Jun 2025).

2. Practical Metrics and Predictive Evaluations

To operationally assess proxy reward sufficiency, contemporary works employ a range of predictive and diagnostic metrics:

• Monotonicity of gold vs proxy reward: Safeguarding sufficiency demands that improvements in proxy-based optimization never cause the gold reward (human or verifiable metric) to decline. For LLM alignment, monotonic upward trajectories of both rewards during PPO or best-of-$n$ inference are a core validation tool (Kim et al., 2024, Gao et al., 2022, Khalaf et al., 24 Jun 2025).
• Proxy evaluation benchmarks: The Preference Proxy Evaluations (PPE) suite establishes 12 key metrics (e.g., pairwise accuracy, AUC, end-of-curve max, best-of-K correctness) computed over both human-preference and correctness datasets, which are tightly correlated ($\rho\approx 0.8$ for pairwise accuracy) with true RLHF downstream performance (Frick et al., 2024). See below:

Metric	Predictive strength for RLHF outcome	Domain
Pairwise human preference accuracy ($M_1$)	$\rho \approx 0.80$	Human-feedback
AUC, best-of-K max, end-score	$\rho \sim 0.7$	Correctness
Global rank-order correlations (Spearman, Kendall)	Near zero	Global

• Conflict metrics: Proxy-Policy Alignment Conflict Score (PACS) and global Kendall–Tau are used to surface local and global regions where the policy and RM diverge, indicating insufficiency “hot spots” amenable to targeted improvement (Liu et al., 10 Dec 2025).

3. Algorithmic Approaches to Achieving or Mitigating Proxy Insufficiency

Several methodologies have been advanced to address, detect, or partially guarantee proxy reward sufficiency:

• Proxy-sufficiency reward design in retrieval-augmented reasoning: TIRESRAG-R1 introduces a sufficiency reward ($R^S$), using an LLM judge to issue a binary “sufficient” (1) or “insufficient” (0) label based on whether retrieved chunks support the answer, then incorporates this into a composite reinforcement learning signal with difficulty-aware reweighting (He et al., 30 Jul 2025). Empirically, ablating $R^S$ causes the largest drop of all components in multi-hop QA benchmarks.
• White-box reverse reward engineering: Constructing simple, interpretable proxy rewards (e.g., products of length, relevance, repetition features) and verifying sufficiency by monotonic improvement of a stronger “gold” RM under RL, achieving state-of-the-art on alignment evaluations with transparent, minimal proxies (Kim et al., 2024).
• Conflict-aware active relabeling: Sampling (x, y) pairs with high PACS or low Kendall–Tau between base policy and proxy RM, then sending these for human feedback, efficiently eliminates most low-sufficiency regions and improves downstream RLHF outcomes under tight annotation budgets (Liu et al., 10 Dec 2025).
• HedgeTune for avoiding reward hacking at inference time: For inference-time alignment methods (e.g., BoN, SBoN, BoP), sufficiency is characterized by the absence of “winner’s curse” regime (i.e., monotonicity in gold reward as the selection parameter increases); HedgeTune adaptively finds the optimal selection parameter to peak true reward, mitigating hacking (Khalaf et al., 24 Jun 2025).
• Preference-based reward repair: In RL settings, Preference-Based Reward Repair (PBRR) iteratively applies preference learning to fit corrections only on transitions where a designer proxy reward disagrees with human feedback, provably mitigating hacking and restoring optimality with orders-of-magnitude less data (Hatgis-Kessell et al., 14 Oct 2025).
• Proxy-free or IRL-based methods: TD-GFN leverages inverse reinforcement learning to reconstruct edge-level rewards from offline data, bypassing proxy model fitting and directly anchoring policy learning in the true data distribution, obviating proxy sufficiency concerns (Chen et al., 26 May 2025).

4. Empirical Manifestations and Scaling Laws

Empirical studies demonstrate regularities:

• Goodhart’s Law and optimization curves: As a proxy reward is optimized—via RL or best-of-$n$ selection—true (gold) performance often rises, then plateaus or collapses (“winner’s curse”), a phenomenon explicated in (Gao et al., 2022) and (Khalaf et al., 24 Jun 2025). The transition point (where sufficiency fails) depends smoothly on RM size, data volume, and proxy construction.
• Scaling laws: The safe regime for proxy optimization is determined by two parameters, $\alpha$ (slope) and $\beta$ (curvature), which depend on RM/data scale. One should optimize up to the analytically determined peak (e.g., $d^*=\alpha/(2\beta)$ for BoN), beyond which hacking is inevitable (Gao et al., 2022).
• Experimental quantification: In RLHF, on-policy proxy-labeled data plus active learning can generate sufficient preference datasets, yielding $\sim$1%+ gains on AlpacaEval2, MMLU under modest expert query budgets; overoptimization or OOD sampling sharply reduces sufficiency (Chen et al., 2024).
• Ablations: In retrieval, ablation studies on sufficiency reward components consistently show sharp drops in EM and F1 when the sufficiency subreward is removed, confirming criticality (He et al., 30 Jul 2025).

5. Limitations, Failure Modes, and Open Problems

Despite successes, notable limitations remain:

• Impossibility in full generality: Perfect proxy sufficiency is only possible if the proxy is colinear with the true objective; outside finite policy classes or heavily restricted settings, all nontrivial proxies are hackable (Skalse et al., 2022).
• Dependence on annotation and demographic coverage: Off-the-shelf reward models often encode demographic or social biases, rendering them insufficient proxies for pluralistic human values (Elle, 7 Oct 2025).
• Coarseness and reliability of judgment: Binary sufficiency signals (e.g., in RAG), simple length-based rewards for LLM alignment, or heuristics for retrieval can be gamed or are linearly insufficient vis-à-vis more nuanced gold objectives (He et al., 30 Jul 2025, Kim et al., 2024).
• Active feedback scalability: Methods that require selective human-in-the-loop feedback (e.g., conflict-based relabeling) introduce computational and annotation overhead; in practice, resolution at web scale remains unproven (Liu et al., 10 Dec 2025).
• Sensitivity to noise and OOD: Proxy sufficiency degrades rapidly in OOD regions, under weak evaluators, or in domains with highly imbalanced or non-representative seed data (Chen et al., 2024).
• No formal sufficiency guarantees for most heuristics: Proxy designs relying on LLM judges, ensemble uncertainty, or diversity-based regularization provide heuristically improved sufficiency but lack provable alignment to the true objective (He et al., 30 Dec 2025).

6. Generalizations and Domain-Specific Extensions

Variants of the sufficiency principle have been developed for specific modalities and settings:

• Empirical sufficiency in delayed reward calibration: Techniques like ESCE identify empirical sufficient states as those that guarantee future positive reward, extracting them with classifiers and providing policy-invariance via reward shaping (Liu et al., 2021).
• Information sufficiency in exploration: In settings such as autonomous driving, an exploration bonus (e.g., entropy) may incentivize irrelevant exploration; a sufficiency-based bonus only rewards information gain that enables higher expected payoff, suppressing spurious learning once the relevant parameter is estimated (Geary et al., 2021).
• Proxy-free optimization in generative models: For generative flow networks (GFlowNets), edge-level IRL bypasses proxy fitting, ensuring sufficient supervision is focused on transitions truly responsible for success (Chen et al., 26 May 2025).
• Selective regularization in visual RL: GARDO demonstrates that non-uniform KL regularization and diversity shaping can maintain proxy sufficiency in RL-tuned diffusion models, preventing mode-collapse and OOD exploitation (He et al., 30 Dec 2025).

7. Best Practices and Emerging Directions

Research converges on the following practitioner guidelines:

1. Empirical monotonicity requirement: Always assess whether improvements in the proxy reward induce monotonic or at least non-decreasing gains under the gold metric across the relevant optimization trajectory (Gao et al., 2022, Kim et al., 2024, He et al., 30 Dec 2025, Frick et al., 2024).
2. Capacity and data scale matching: Larger, better-trained reward models expand the safe sufficiency region and raise the hacking threshold, but cannot eliminate the fundamental gap (Gao et al., 2022).
3. Targeted feedback acquisition: Selective, conflict-driven or active learning-based feedback greatly improves label efficiency and sufficiency in proxy and policy refinement (Liu et al., 10 Dec 2025, Chen et al., 2024).
4. Domain-aware and demographically pluralistic proxy design: Proxy reward models and datasets should explicitly integrate multiple demographic and domain perspectives to enhance alignment sufficiency (Elle, 7 Oct 2025).
5. Integrated diagnostics: Proxy sufficiency should be regularly re-evaluated via predictive testbeds (PPE), targeted ablations, and synthetic “reward inversion” stress tests (hacking curves) (Frick et al., 2024, Gao et al., 2022, Khalaf et al., 24 Jun 2025).
6. Regularization and ensemble gating: Uncertainty- or diversity-aware regularization should be incorporated, but only applied where warranted (by high-uncertainty signals), maintaining exploration flexibility (He et al., 30 Dec 2025).

The field remains open in terms of formal sufficiency criteria for complex domains, scalable detection and mitigation of hacking in RLHF and generative models, and rigorous sufficiency guarantees for reward models incorporating ensemble or distributional uncertainty. Proxy reward sufficiency remains a foundational constraint and design principle in all state-of-the-art RL, RLHF, and retrieval-augmented reasoning systems.