LLM-based Reward Generation in RL

Updated 9 November 2025

LLM-based reward generation is a technique that integrates large language models with reinforcement learning to automate reward design and improve sample efficiency.
It employs methods such as direct reward synthesis, reward model learning, and rubric-driven preference evaluation to generate dynamic, task-specific rewards.
This approach reduces human labor, enhances alignment with high-level objectives, and scales to complex and multi-agent decision-making tasks.

LLM-based reward generation is a field at the intersection of reinforcement learning (RL) and LLMs, in which LLMs—sometimes in conjunction with vision-LLMs (VLMs) or other AI systems—design, synthesize, or evaluate reward functions for autonomous learning agents. The goal is to automate the reward engineering process, increasing alignment with high-level objectives, improving sample efficiency, and reducing human labor or domain expertise requirements across a range of RL, decision making, or generative model tasks.

1. Foundational Principles and Core Motivations

Traditional RL depends critically on reward shaping, yet designing a reward function that robustly encodes task structure, induces optimal behavior, and avoids unintended incentives is often nontrivial and labor-intensive. LLM-based reward generation leverages LLMs’ capabilities in language understanding, code generation, commonsense reasoning, and few-shot learning to solve or ameliorate the following challenges:

Specification cost: Automating the reward design loop and reducing human-in-the-loop requirements (Sun et al., 18 Oct 2024, Sarukkai et al., 11 Oct 2024, Kwon et al., 2023).
Alignment: Ensuring reward functions reflect user-desired outcomes or multi-criterion objectives, beyond simple hand-coded metrics (Liu et al., 9 Oct 2025, Liu et al., 26 Nov 2024, Liu et al., 27 Apr 2024).
Data efficiency: Generating dense or informative per-step/process rewards for sample-efficient policy learning and improved credit assignment (Lin et al., 6 Feb 2025, 2505.20737, Zhang et al., 6 Jun 2024).
Generalization: Enabling reward modeling in out-of-distribution settings and across new tasks or unseen goals (Chen et al., 17 Feb 2025, Xia et al., 25 Feb 2025, Guo et al., 18 May 2025).
Scalability: Scaling automated reward synthesis to multi-agent, multi-step, or complex domains, including those with visual and language modalities (Yao et al., 19 Sep 2025, Fang et al., 30 May 2025).

These motivations underlie the architecture, evaluation, and rapid development of LLM-empowered reward design frameworks in modern RL research.

2. Algorithmic Frameworks and Methodological Taxonomy

LLM-based reward generation methodologies can be organized along several axes: direct reward code synthesis, reward model learning, reward-guided search, and preference/rubric-based supervision.

The CARD framework (Sun et al., 18 Oct 2024) typifies the modern LLM-in-the-loop reward code generation paradigm:

Coder: An LLM prompts with task/environment descriptions to produce dense Python reward functions (with scalar and sub-rewards).
Evaluator: An automated process that runs brief RL rollouts and produces process, trajectory, and trajectory preference feedback—using Trajectory Preference Evaluation (TPE) to assess whether the reward separates successes from failures:

$\bar{G}(\tau) = \frac{1}{|\tau|}\sum_{t=0}^{|\tau|} \gamma^t r(s_t, a_t)$

Reward is order-preserving iff

$\min_{\tau^+} \bar{G}(\tau^+) > \max_{\tau^-} \bar{G}(\tau^-)$

The loop typically requires just a few LLM iterations and demonstrates high sample efficiency (e.g., success rates exceeding expert-designed rewards on several tasks), with error rates reduced via code execution checks and feedback.

Other frameworks—such as PCGRLLM (Baek et al., 15 Feb 2025) and Boosted ROS Evolution (Heng et al., 10 Apr 2025)—augment reward proposal with self-alignment, environment-driven feedback, or two-stage evolution of reward observation spaces, sometimes incorporating content- or state-level execution tables to guide LLM proposals toward unexplored or high-value state features.

2.2 Learning Reward Models from Data

In cases where dense reward code is infeasible or uninterpretable, LLMs (and sometimes VLMs) may be trained as reward models mapping observations or trajectories to scalar feedback.

Automatic Reward Modeling And Planning (ARMAP) (Chen et al., 17 Feb 2025) learns a trajectory-level reward model by synthesizing positive/negative trajectory pairs via LLMs, then trains a VLM to optimize the cross-entropy between positive and negative scores:

$\mathcal{L}(\theta) = -\mathbb{E} \; \log \sigma \left( r_\theta(t, h^+) - r_\theta(t, h^-) \right)$

The learned reward model guides best-of-N sampling, Reflexion, or MCTS planning strategies, thereby boosting agent performance on diverse interactive benchmarks.

AgentRM (Xia et al., 25 Feb 2025) considers three approaches: explicit reward modeling via process value heads, implicit modeling through policy likelihood ratios, and LLM-as-judge preference modeling. These methods enable generalization to held-out tasks and scaling to strong policy models, highlighting the role of reward models as robust, search-time surrogates.
Efficient Linear Hidden-State Reward (ELHSR) (Guo et al., 18 May 2025) demonstrates that a frozen base LLM can be augmented with a lightweight linear reward head consuming transformer hidden states (or logits) to rapidly predict solution correctness, yielding rapid best-of-N gains with <0.005% the parameters of baseline models.

2.3 Preference-based and Rubric-driven Reward Synthesis

Several works have established preference modeling as a scalable alternative to direct numeric reward specification.

OpenRubrics (Liu et al., 9 Oct 2025) formalizes Contrastive Rubric Generation (CRG), in which an LLM, given preferred ( $\hat{y}^+$ ) and rejected ( $\hat{y}^-$ ) responses for a prompt $x$ , outputs a rubric $\mathcal{R}(x)$ by contrasting the two. The learning objective is

$\mathcal{L}_{\mathrm{CRG}} = -\sum_{i=1}^N \log p_\psi \big(\mathcal{R}(x_i) \mid x_i, \hat y_i^+, \hat y_i^- \big) + \lambda\,\Omega(\psi)$

with regularization $\Omega(\psi)$ . Rejection sampling ensures preference-label consistency:

$\hat \ell_i = h_\psi(x_i, \mathcal{R}(x_i), \hat y_i^+, \hat y_i^-), \quad \mathcal{R}^*(x_i) = \begin{cases} \mathcal{R}(x_i), & \hat\ell_i = \ell_i \ \emptyset, & \text{otherwise} \end{cases}$

The learned Rubric-RM model is supervised with

$\mathcal{L}_{\mathrm{rm}} = -\mathbb{E} \sum_{t=1}^{|\hat\ell|} \log p_\phi \big(\hat\ell_t \mid x, \hat y^+, \hat y^-, \mathcal R^*(x), \hat\ell_{<t}\big)$

This yields scalable, multi-criteria reward modeling transferable across alignment and biomedical benchmarks.

ReFINE/ER²Score (Liu et al., 26 Nov 2024) and MRScore (Liu et al., 27 Apr 2024) develop multi-criterion LLM-based reward heads (trained on GPT-4–scored report pairs) for radiology report evaluation, enforcing both overall and per-dimension margin constraints on reward outputs. These models achieve superior correlation to human ratings than n-gram or embedding-based metrics.

2.4 Intrinsic Motivation, Novelty, and Progressive Feedback

Several approaches leverage LLMs as sources of intrinsic motivation or process supervision:

ProgressCounts (Sarukkai et al., 11 Oct 2024) prompts LLMs to write progress functions (low-D state feature mappings), discretizes these into bins, and applies classic count-based exploration bonuses

$R_{total}(s_t, a_t) = R(s_t, a_t) + \lambda_c \cdot 1/\sqrt{c(B(s_{t+1}))}$

yielding SOTA sample efficiency on dexterous manipulation.

VSIMR+LLM Intrinsic (Quadros et al., 25 Aug 2025) combines VAE-based state novelty with LLM-driven state "helpfulness" scores, linearly mixing these with extrinsic rewards to drive both novelty search and semantically meaningful exploration.
Reward Rising Optimization (RRO) (2505.20737) and Process Reward Models prioritize per-step rewards with rising trajectories, efficiently expanding successful candidate sets until reward increases, drastically lowering sample complexity versus exhaustive PRM search.

3. LLM-based Reward Generation in Multi-Agent and Hierarchical Contexts

LLMs enable sophisticated credit assignment, emergent organization, and cooperation in multi-agent and hierarchical RL:

Potential-based credit assignment (Lin et al., 6 Feb 2025): LLMs transform agent-specific language descriptions into preference labels over agent transitions, which are then fitted as shape-invariant potential functions $\Phi_i$ :

$r_i^t = r_{\mathrm{team}}^t + \eta \left[ \Phi_i(s_{t+1}) - \Phi_i(s_t) \right]$

Pairwise preference labels are aggregated and modeled using Bradley-Terry logistic regression, yielding dense, agent-specific shaped rewards enabling efficient MARL learning.

ReSo: Collaborative Reward Model (CRM) (Zhou et al., 4 Mar 2025): MAS decomposition leverages a process-reward LLM to estimate the probability that a given agent's solution is correct for subtask $v_j$ , updating agent reputations and enabling self-organizing task assignment.
RE-GoT bi-level reward evolution (Yao et al., 19 Sep 2025) merges graph-of-thoughts (GoT) decompositions for structured task analysis with iterative LLM/VLM reward loops, in which VLM feedback on RL-generated rollout videos drives automated reward code refinement for subgoals and transitions.

4. Evaluation, Benchmarking, and Empirical Outcomes

LLM-based reward generation has been evaluated on diverse RL and generative modeling tasks:

Framework	Domain(s)	Key Metrics/Outcomes
CARD (Sun et al., 18 Oct 2024)	Meta-World, ManiSkill2	Outperforms baselines on 10/12 tasks; sometimes surpasses oracles
ProgressCounts (Sarukkai et al., 11 Oct 2024)	Bi-DexHands	SOTA sample efficiency (20x fewer reward-code samples than prior)
ARMAP (Chen et al., 17 Feb 2025)	WebShop, ScienceWorld, ALFWorld, Game24	Boosts success by 10–20% over sampling/greedy on public and API models
AgentRM (Xia et al., 25 Feb 2025)	WebNav, Embodied, Tool use	+8.8 points over baseline, scaling with policy size and task count
OpenRubrics (Liu et al., 9 Oct 2025)	Reward modeling/alignment	+6.8% over base RM, strong correlation with human preference
ReFINE (Liu et al., 26 Nov 2024) MRScore (Liu et al., 27 Apr 2024)	Medical report eval	τ = 0.751 (ReFINE), 0.25 (MRScore): modern LLM-RMs surpass BLEU, BERTScore

Methods deploying graph-of-thoughts, in-context RL with scalar rewards (Song et al., 21 May 2025), or MCTS-based search with inferred process rewards (Zhang et al., 6 Jun 2024) all demonstrate substantial improvements in both model selection, policy sample efficiency, and alignment with user/human criteria.

5. Limitations, Open Problems, and Future Directions

Despite empirical successes, several challenges persist:

Token and compute cost: Iterative reward design with LLMs can be expensive when many refinements or environment samples are needed, motivating work on parameter-efficient heads (ELHSR (Guo et al., 18 May 2025)), pruning strategies, and memory-augmented prompting.
Dependence on Environment Abstraction: Most frameworks (e.g., CARD, RoboMoRe) require full, accurate environment and API descriptions; integrating pixel-level or multimodal feedback is an open direction (Sun et al., 18 Oct 2024, Yao et al., 19 Sep 2025).
Alignment and bias: Reward models may encode LLM or data-induced biases; rubric-based filtering, margin enforcement, and synthetic label de-noising (OpenRubrics, MRScore) partially mitigate but do not eliminate this risk.
Adaptivity and generalization: The ability to generalize reward generation to new, out-of-distribution, or multi-objective settings remains underexplored; research in lifelong reward learning and compositional reward modeling is ongoing.
Human-in-the-loop vs. Self-supervision: Hybrid frameworks may ultimately outperform either extreme, combining scalable self-generated feedback with periodic human validation or guidance.

Key emerging directions include: integration with VLMs for visual+language reward grounding; using LLMs for credit assignment in increasingly complex agent pools; expanding to non-RL settings (e.g., text and code evaluation); and further theoretical understanding of emergent reward induction in LLM inference-time reasoning (ICRL), where test-time contextually-parameterized reward information dynamically shapes generation (Song et al., 21 May 2025).

LLM-based reward generation constitutes a rapidly maturing field that is fundamentally transforming how RL agents—and more generally, autonomous AI systems—are supervised, aligned, and optimized. Through methods ranging from direct code synthesis, rubric-driven supervision, and process-level reward modeling to preference inference and bi-level graph-structured reasoning, LLMs are increasingly central to the automation, interpretability, and scalability of reward specification in diverse sequential decision-making domains.