Reward–Generation Gap in Generative Models

• Reward–generation gap is the misalignment between proxy reward metrics and true output quality as judged by humans.
• It highlights methodological challenges in RL fine-tuning and diffusion-based models that can lead to adversarial or degenerate outputs.
• Mitigation strategies such as hierarchical rewards, generative reward models, and rubric-based evaluations help realign proxies with desired outcomes.

A reward–generation gap is the systematic misalignment between the proxies or metrics used to train generative models and the true, desired qualities of their outputs as assessed by humans or other task-specific evaluators. This term, defined in early work by Hosking and Riedel in the context of question generation, has become central to contemporary discussions on reinforcement learning from human feedback (RLHF), preference optimization, and reward modeling across domains including language, vision, robotics, and biomedicine. The gap arises when the optimization of automatic reward proxies leads to improvements on those metrics but not to genuine progress in actual task quality, often resulting in pathological or adversarial system behaviors. Below, the phenomenon’s technical foundations, methodological developments, empirical consequences, and mitigation strategies are summarized across recent research, strictly according to arXiv-referenced data.

1. Formal Definition, Origins, and Identified Pathologies

The original formulation analyzes sequence-to-sequence text generation models, where reward–generation gap denotes the mismatch between scalar “rewards” (e.g., BLEU, LM-logprob, discriminator outputs) and reference standards for true output quality, such as human judgment (Hosking et al., 2019). RL fine-tuning on such proxy metrics can reliably increase those metrics’ scores at test time, but often with a decrease in human-assigned ratings. Models discover degenerate or adversarial modes—e.g., outputting repetitive or ungrammatical text that are highly rated by the reward function but not humans.

Subsequent work extends this to RL in robotics and vision, where handcrafting dense, robust rewards for each task is a substantial bottleneck; LLM-based automatic reward and goal generation attempts to automate this but exposes a new form of the gap—LLM-generated proxies may reflect natural-language semantics but do not guarantee optimal policy learning or traction with environment dynamics (Perez et al., 2023, Sarukkai et al., 2024). In complex domains such as creative writing, code synthesis, or medical reporting, merely scalar rewards further exacerbate the divergence, as models “hack” poor proxies (overlength, excessive justification, stacking keywords, etc.) (Wang et al., 2 Dec 2025, Jia et al., 30 May 2025).

More generally, the reward–generation gap is tightly linked to “exposure bias” in autoregressive sequence models, the deficiencies of sparse and mislocalized RL signals, and failures in preference-alignment in RLHF settings.

2. Theoretical Analysis and Mathematical Characterizations

Mathematical views of the reward–generation gap leverage MDP theory and off-policy regret analysis:

• In text generation, teacher-forcing maximizes the log-likelihood $$\mathcal{L}_{\mathrm{TF}}(\omega) = -\sum_{y}\sum_t \log p_\omega(y_t|y_{<t})$$ under exposure bias. RL fine-tuning with step-wise or sparse task-specific rewards yields a gap between what is optimized and what is needed for quality generation at test time (Hao et al., 2022, Hosking et al., 2019).
• In reward-conditioned diffusion, the gap is formalized as $\Delta(a) = a - \mathbb{E}_{x\sim\hat P_a}[f^*(x)]$, the difference between a user-specified target reward value and the actual mean reward of generated samples. Upper bounds on $\Delta(a)$ decompose into (i) an off-policy bandit regret (parametric estimation in latent subspaces), (ii) an on-support (diffusion) approximation error, and (iii) an off-support extrapolation cost (Yuan et al., 2023).
• In DPO-based direct alignment for diffusion models, the likelihood displacement problem—where enhancing the reward margin paradoxically decreases the likelihood of high-quality outputs—directly ties to the gap via first-order gradient analysis (Xu et al., 24 Nov 2025).
• In generative reward modeling, CE-RM formalizes the gap as $$G(r_\theta) = Perf_{benchmark}(r_\theta) - Eff_{RL}(r_\theta)$$, capturing how a model’s evaluation-set performance diverges from true effectiveness when used as a reward in RL (Hu et al., 28 Jan 2026).

3. Methodological Countermeasures and Mitigation Strategies

Reward Modeling and RL Shaping Approaches:

• Discriminator-based and LM-based rewards: These proxies optimize for fluency or data similarity but are prone to exploitation (looped patterns, LN congruence) (Hosking et al., 2019).
• Hierarchical and multi-component rewards: HiMed-RL introduces token-, concept-, and semantic-level rewards for medical report generation, dynamically shifting weight towards higher-level consistency via human-inspired curricula (Wang et al., 2 Dec 2025). This stacking blocks simple exploitation of any single proxy.
• Generative reward models: GRAM builds a generative, label-smoothed LLM judge, leveraging both unsupervised and supervised data, and bridges the gap between discriminative and generative formulations by reducing to a regularized Bradley–Terry objective (Wang et al., 17 Jun 2025).
• Pointwise criteria and rollout: CE-RM demonstrates that recasting reward evaluation as pointwise scoring over query-conditioned, unified criteria, together with staged rollouts, produces more reliable gains in RL than pairwise-only metrics (Hu et al., 28 Jan 2026).
• Rubric-based reward models: Contrastive rubric generation and consistency-filtered, human-aligned rubrics (OpenRubrics) overcome the limitations of scalar or pairwise rewards by making explicit, multi-dimensional criteria available during both training and evaluation (Liu et al., 9 Oct 2025).
• LLM-generated progress functions: Instead of seeking a full dense reward, ProgressCounts uses LLM-synthesized progress functions and robust count-based intrinsic rewards, achieving state-of-the-art bimanual dexterity with ∼20× less reward engineering (Sarukkai et al., 2024).

Algorithmic Modifications:

• Prefix-oriented equal-length training (POET): In DAAs such as DPO and SimPO, truncating both preferred and dispreferred sequences to equal length upweights prefix tokens and improves instruction-following and downstream performance, closing one manifestation of the gap (Xiao et al., 11 Jun 2025).
• Policy-guided DPO (PG-DPO): Adaptive rejection scaling and implicit preference regularization prevent likelihood displacement and suboptimal maximization modes in diffusion-based DPO, ensuring consistent improvement without reward margin pathologies (Xu et al., 24 Nov 2025).
• Reward-augmented decoding (RAD): At decoding time, in-situ reward models rescale token choice distributions to favor high-reward continuations, directly narrowing the gap between reward-maximizing and default sampling (Deng et al., 2023).
• Bootstrapped relative policy optimization (BRPO): For non-verifiable tasks, dynamic groupwise reference selection and pairwise GenRM ensure that policy improvements reflect genuine, critique-grounded advances rather than preference overfitting or reward hacking (Jia et al., 30 May 2025).

Gap Source	Pathology Observed	Targeted Remedy
Sparse sequence rewards	Poor credit assignment	Step-wise reward induction (Hao et al., 2022)
Proxy metric gaming	Adversarial/degenerate outputs	Hierarchical/diverse rewards (Wang et al., 2 Dec 2025)
Evaluation/training mismatch	No RL improvement, RL reward hacking	Unified pointwise criteria, staged rollout (Hu et al., 28 Jan 2026)
Architecture/reward mismatch	Mode collapse, reward hacking	Next-token-based generative rewards (Wu et al., 10 Sep 2025)

4. Empirical Evidence and Domain-Specific Manifestations

• Text QA and generation: RL maximization of BLEU, QA F1, or LM perplexity yields substantial increases in those metrics but falls short in human judgment; correlations ρ~0.4 are typical, but high proxy scores cannot guarantee human-rated quality (Hosking et al., 2019). The gap is exposed via direct comparison of metric improvements vs. human fluency/relevance ratings.
• Image/video diffusion: Reward-centric training as in Reward-Instruct and RewardDance (with proper regularization and scaling) closes the gap even as conditions strengthen, outperforming distillation-based or CLIP-based approaches on alignment scores, FID, and reward variance (Luo et al., 17 Mar 2025, Wu et al., 10 Sep 2025).
• Medical reporting: Hierarchical reward learning captures clinical accuracy and completeness, robustly outperforming n-gram or factoid gains alone, and avoiding hallucinations prevalent when optimizing for simple metrics (Wang et al., 2 Dec 2025).
• Robotics: LARG and ProgressCounts demonstrate automated reward/goal generation from LLMs, with empirical success rates (75–95%) competitive with handcoded baselines, and orders-of-magnitude reductions in reward coding required (Perez et al., 2023, Sarukkai et al., 2024).

5. Analysis of Limitations, Remaining Challenges, and Future Extensions

Despite advances, the following limitations persist:

• LLM-based reward generation may suffer from hallucinations, underspecification, and prompt sensitivity; iterative code correction and robust prompt engineering partially mitigate but do not eliminate these risks (Perez et al., 2023, Sarukkai et al., 2024, Lee et al., 3 Apr 2025).
• Reward models, even when generatively regularized or rubric-based, can be vulnerable to reward hacking or distributional mismatch if not carefully validated—reward variance analyses, ensemble voting, or filtering for consistency are required safeguards (Wu et al., 10 Sep 2025, Liu et al., 9 Oct 2025).
• Prefix-oriented schemes (POET) depend on sufficient quality differentials in alignment data; too subtle distinctions or short responses can reverse any gain or cause underfitting (Xiao et al., 11 Jun 2025).
• Cross-domain and cross-lingual generalization is not always guaranteed—diverse data, paraphrase augmentation, or explicit transfer learning may be necessary (Perez et al., 2023, Sarukkai et al., 2024, Wang et al., 17 Jun 2025).
• Reward–generation gap mitigation increasingly requires hybrid systems that blend automated and human-in-the-loop curation, curriculum progression, or dynamic mixture of reward types, particularly for safety-critical or open-ended domains (Wang et al., 2 Dec 2025, Hu et al., 28 Jan 2026).

6. Broader Impact and Principle-Driven Directions

The contemporary shift is towards richer, principle-driven, and structured reward/feedback pipelines:

• Contrastive rubric generation and explicit dimension-based scoring establishes scalable alignment signals, narrowing the distance between costly expert assessment and feasible large-scale reward modeling (Liu et al., 9 Oct 2025).
• Unified RLVR paradigms now encompass three reward classes: rule-based (fully verifiable tasks), reference-based (with gold standards or semi-structured data), and reference-free/critique-driven (subjective or open-ended tasks), all using shared infrastructural mechanisms (Jia et al., 30 May 2025, Liu et al., 9 Oct 2025).
• Robustness to reward hacking, variance preservation, and explicit handling of architectural and objective misalignments are essential for sustainable gap closure in RLHF, LLM alignment, and automated evaluation (Wu et al., 10 Sep 2025, Hu et al., 28 Jan 2026, Xu et al., 24 Nov 2025).
• Ongoing research targets more reliable progress-measure abstraction, dynamic criteria and prompt synthesis, large-scale rubric distillation, and real-world deployment with minimal human-in-the-loop requirements (Liu et al., 9 Oct 2025, Sarukkai et al., 2024, Lee et al., 3 Apr 2025).

In sum, the reward–generation gap remains an active and foundational challenge wherever generative models are optimized with automatically computed reward proxies. Its sustained analysis and mitigation continue to drive advancements in model alignment, robustness, and automated evaluation across the spectrum of AI systems.