LLM-Driven Modular Reward Generation

• LLM-driven modular reward generation is a design doctrine that decomposes reward functions into interpretable, semantically meaningful modules for precise task optimization.
• It structures rewards via subtasks, rubric criteria, and operational tools to enable nuanced multi-agent coordination and stage-specific learning.
• Empirical studies show these techniques improve performance metrics and mitigate reward hacking while supporting diagnostics in domains like reinforcement learning and cyber defense.

Searching arXiv for papers on LLM-driven reward generation and rubric-based reward modeling. LLM-driven modular reward generation is a family of methods in which LLMs do not merely emit a single scalar reward, but instead synthesize, decompose, route, or evaluate reward signals through explicit modules such as subtasks, rubric criteria, reward code components, agent-specific credits, tool-selected verifiers, or pipeline stages. Across recent work, the modular unit varies—from DAG nodes in multi-agent reasoning, to rubric items in reward modeling, to programmatic sub-rewards in reinforcement learning, to stage-specific scorers in multi-step pipelines—but the common objective is to replace monolithic reward specification with structured, inspectable, and adaptable reward composition (Zhou et al., 4 Mar 2025, Liu et al., 9 Oct 2025, Sarukkai et al., 2024, Sun et al., 2024, Chen et al., 17 Feb 2025, Fan et al., 20 Apr 2026, Fu et al., 2 Jul 2026).

1. Conceptual foundations

The basic premise is that reward design is often easier to operationalize when the target objective is factored into semantically meaningful parts. In executable reward-code systems, the LLM generates separate terms for properties such as collision avoidance, lane keeping, comfort, progress, action costs, or game-specific constraints, then combines them into an aggregate reward used by RL (Han et al., 2024, Baek et al., 2024, Baek et al., 15 Feb 2025, Mukherjee et al., 20 Nov 2025). In rubric-based systems, the LLM generates explicit criteria—often divided into hard rules and principles—and the final reward is an aggregation of criterion-level judgments rather than an opaque scalar (Liu et al., 9 Oct 2025, Liu et al., 9 Mar 2026, Fu et al., 2 Jul 2026). In multi-agent and pipeline settings, modularity appears as per-agent, per-subtask, or per-stage scoring, so that downstream optimization is aligned with intermediate structure rather than only terminal success (Zhou et al., 4 Mar 2025, Lin et al., 6 Feb 2025, Fan et al., 20 Apr 2026).

A central distinction is between generation of reward representations and learning from those representations. Some systems use the LLM to author executable reward code directly, as in ChatPCG, PCGRLLM, CARD, highway driving reward evolution, and autonomous cyber defense (Baek et al., 2024, Baek et al., 15 Feb 2025, Sun et al., 2024, Han et al., 2024, Mukherjee et al., 20 Nov 2025). Others use the LLM to generate structured evaluative artifacts—rubrics, task intents, synthetic negatives, progress functions, or verifiers—which are then converted into reward signals by a separate aggregation or learning stage (Sarukkai et al., 2024, Chen et al., 17 Feb 2025, Liu et al., 9 Oct 2025, Liu et al., 9 Mar 2026, Fu et al., 2 Jul 2026). A further line argues that a reward model may already be latent inside the base LLM and can be elicited directly from logits, making reward generation effectively a structured extraction problem rather than a separate training problem (Li et al., 29 Jun 2025).

This suggests that “modular reward generation” is not a single algorithmic template but a design doctrine: align reward units with the causal, semantic, or operational decomposition of the task, then use those units for attribution, calibration, and optimization.

2. Main axes of modularity

Three recurrent decomposition axes appear across the literature: task structure, evaluative criteria, and operational tooling.

Modularity unit	Representative formulation	Example papers
Subtasks or graph nodes	$R_{\text{total}}(G)=\sum_{v_j\in V} r(a_j^*, v_j)$	ReSo (Zhou et al., 4 Mar 2025)
Rubric criteria	$r(x,y)=\sum_{k=1}^{K} w_k \cdot s_k(x,y;c_k)$	OpenRubrics (Liu et al., 9 Oct 2025), CDRRM (Liu et al., 9 Mar 2026), MRRG (Fu et al., 2 Jul 2026)
Reward code components	$R(s,a)\approx \sum_i w_i r_i(s,a)$	PCGRLLM (Baek et al., 15 Feb 2025), ChatPCG (Baek et al., 2024), driving (Han et al., 2024)
Hierarchical gated rewards	$$R=\omega_1R_{\mathrm{format}}+\omega_2R_{\mathrm{comp}}+\omega_3R_{\mathrm{func}}+\omega_4R_{\mathrm{syn}}+\omega_5R_{\mathrm{ppa}}$$	ChipSeek-R1 (Chen et al., 7 Jul 2025)
Per-agent potentials	$r'_i(s,a_i,s')=\gamma\Phi_i(s')-\Phi_i(s)$	LCA (Lin et al., 6 Feb 2025)
Tool-selected reward modules	$R(x)=\sum_{m\in\mathcal{M}(x)} w_m(x)\cdot r_m(x)$	RLAR (Feng et al., 28 Feb 2026)
Stage-specific scorers	$r^{(F)}_{\theta_F}(P,F)$ and $r^{(S)}_{\theta_S}(x_S(F),S)$	PARM (Fan et al., 20 Apr 2026)

Task-structured modularity is most explicit in ReSo, where the question is mapped to a DAG, each node is assigned an agent through a two-stage selection process, and the Collaborative Reward Model produces step-wise rewards $r(a_i,v_j)$ that update the Dynamic Agent Database (Zhou et al., 4 Mar 2025). Closely related ideas appear in pipeline-adapted reward models, where the stages are not graph nodes but sequential pipeline components; PARM trains separate reward models for formulation and code generation by using downstream execution to assign stage-specific preferences (Fan et al., 20 Apr 2026).

Criterion-structured modularity is the defining feature of rubric-based reward modeling. OpenRubrics formalizes the reward as a weighted sum over rubric criteria and explicitly distinguishes hard rules from principles, while MRRG expands the criterion set by eliciting rubrics from multiple evaluator roles and consolidating them into a single auditable scorer (Liu et al., 9 Oct 2025, Fu et al., 2 Jul 2026). CDRRM further emphasizes atomic, evidence-anchored rules and preference-consistency filtering, framing rubric construction as contrastive factor extraction followed by synthesis (Liu et al., 9 Mar 2026).

Operational modularity appears in systems that treat reward acquisition as tool orchestration. RLAR builds a reward tool library whose modules may be retrieved reward models, synthesized programmatic verifiers, or standard metrics, and selects or synthesizes modules per query (Feng et al., 28 Feb 2026). ARMAP similarly separates explorer LLMs, intent synthesizers, learned reward models, and planners, so that trajectory evaluation is a pluggable component rather than a fixed scalar oracle (Chen et al., 17 Feb 2025).

3. Formal mechanisms of reward construction

Despite different surface forms, most systems reduce to additive or pairwise-preference formulations. Additive aggregation is the dominant pattern. ReSo aggregates node rewards over a task graph (Zhou et al., 4 Mar 2025). OpenRubrics and related rubric systems aggregate criterion scores with weights (Liu et al., 9 Oct 2025, Liu et al., 9 Mar 2026, Fu et al., 2 Jul 2026). ChatPCG, PCGRLLM, highway driving reward generation, and cyber defense all use weighted sums of executable reward terms, though the weights are often embedded directly in LLM-produced code or YAML specifications rather than learned end-to-end (Baek et al., 2024, Baek et al., 15 Feb 2025, Han et al., 2024, Mukherjee et al., 20 Nov 2025).

A second recurring mechanism is pairwise preference learning. ARMAP trains a reward model on positive and negative trajectories with

$$\mathcal{L}(\theta) = -\mathbb{E}_{(x_m,h_m^+,h_m^-)}\left[\log \sigma\big(\mathcal{R}_\theta(x_m,h_m^+) - \mathcal{R}_\theta(x_m,h_m^-)\big)\right],$$

using automatically synthesized task intents and negative trajectories (Chen et al., 17 Feb 2025). LLM4PG adopts a Bradley–Terry-style preference model over trajectory returns learned from LLM judgments on natural-language trajectory abstractions (Shen et al., 2024). PARM uses a DPO-style logistic objective over stage-specific positive and negative pairs derived from execution outcomes in a two-stage pipeline (Fan et al., 20 Apr 2026). CDRRM defines pairwise rubric loss over aggregated rubric scores $r(x,y)=\sum_{k=1}^{K} w_k \cdot s_k(x,y;c_k)$0, again using a logistic comparison (Liu et al., 9 Mar 2026).

A third mechanism is hierarchical gating. ChipSeek-R1 is the clearest example: syntax, functional correctness, synthesizability, and PPA are separate reward modules, but higher-level rewards are only considered when prerequisite lower-level checks pass. The paper formalizes dependencies such as $r(x,y)=\sum_{k=1}^{K} w_k \cdot s_k(x,y;c_k)$1 and $r(x,y)=\sum_{k=1}^{K} w_k \cdot s_k(x,y;c_k)$2, which is explicitly intended to prevent reward hacking and avoid wasting expensive EDA evaluations (Chen et al., 7 Jul 2025).

Potential-based shaping remains important in environments where dense local signals are needed but policy invariance is desirable. LCA learns per-agent potentials from LLM-generated pairwise preferences and uses

$r(x,y)=\sum_{k=1}^{K} w_k \cdot s_k(x,y;c_k)$3

for agent-specific shaping in MARL (Lin et al., 6 Feb 2025). The highway driving framework explicitly invokes potential-based shaping through $r(x,y)=\sum_{k=1}^{K} w_k \cdot s_k(x,y;c_k)$4 to encode longitudinal progress while keeping the reward modular across safety, comfort, speed, and rules (Han et al., 2024).

A more radical formalization is endogenous reward extraction. The “Generalist Reward Models” work defines token-level endogenous reward as an inverse soft Bellman residual derived from logits,

$r(x,y)=\sum_{k=1}^{K} w_k \cdot s_k(x,y;c_k)$5

and shows theoretical equivalence to an offline inverse RL reward on the same data (Li et al., 29 Jun 2025). In that formulation, modularity can be induced at prompting time by eliciting domain- or criterion-specific rewards from the same base model.

4. Representative system families

One major family uses LLMs to write executable reward programs for RL. ChatPCG organizes reward terms around multiplayer game mechanics such as role differentiation, survivability, and damage balance, then iteratively aligns module outputs against gameplay logs (Baek et al., 2024). PCGRLLM extends this with feedback, self-alignment, and reasoning-based prompt engineering, exploring CoT, ToT, and GoT over modular reward code for story-conditioned level generation (Baek et al., 15 Feb 2025). CARD similarly constrains the LLM to emit a reward function that returns both a scalar and a dictionary of sub-rewards, then refines it using process feedback, trajectory feedback, and Trajectory Preference Evaluation (Sun et al., 2024). In autonomous driving and cyber defense, the LLM is given environment code or YAML schemas and asked to instantiate interpretable modules such as collision penalties, speed regulation, decoy placement incentives, and resource costs (Han et al., 2024, Mukherjee et al., 20 Nov 2025).

A second family uses LLMs to generate compact latent structures that are subsequently converted into rewards. ProgressCounts asks the LLM to author a progress function $r(x,y)=\sum_{k=1}^{K} w_k \cdot s_k(x,y;c_k)$6, then discretizes progress into bins and applies count-based intrinsic rewards over the induced low-dimensional state space (Sarukkai et al., 2024). The ROS-evolution framework similarly separates reward observation space state selection from operations over those states, using a state execution table as non-Markovian memory over prior exploration (Heng et al., 10 Apr 2025). These methods do not ask the LLM to write the final scalar reward directly; they ask it to produce the right latent coordinates for reward construction.

A third family centers on rubrics and explicit evaluative decomposition. OpenRubrics uses Contrastive Rubric Generation to derive hard rules and principles from chosen–rejected pairs, then trains a rubric generator and rubric-aware judge (Liu et al., 9 Oct 2025). CDRRM extends this with multi-dimensional contrastive profiling, evidence anchoring, pruning, and consistency filtering, emphasizing compact and context-aware rubrics (Liu et al., 9 Mar 2026). MRRG replaces the single evaluator with multiple complementary roles and shows that the resulting scorer can be used both for pairwise judging and for GRPO-style RLVR (Fu et al., 2 Jul 2026).

A fourth family uses modular rewards for coordination or planning. ReSo decomposes reasoning tasks into DAGs and assigns heterogeneous agents with CRM-guided fine-grained credit (Zhou et al., 4 Mar 2025). LCA generates agent-specific dense rewards from ego-centric language descriptions and learns per-agent potentials from multi-query LLM preferences (Lin et al., 6 Feb 2025). ARMAP, RLAR, and PARM all attach modular rewards to candidate trajectories or pipeline stages, enabling reranking, Reflexion, MCTS, or stage-wise selection without fine-tuning closed policy models (Chen et al., 17 Feb 2025, Feng et al., 28 Feb 2026, Fan et al., 20 Apr 2026).

5. Empirical performance and evaluation regimes

Empirical results indicate that modularization is not merely interpretability scaffolding; it often changes optimization behavior materially. ReSo reports 79.1%, 56.2%, and 33.7% on easy, medium, and hard Math-MAS, and 67.3%, 51.3%, and 32.3% on SciBench-MAS, with other MAS frameworks largely failing at hard difficulty (Zhou et al., 4 Mar 2025). OpenRubrics reports that Rubric-RM-8B surpasses strong size-matched baselines by 6.8% on average across reward-modeling benchmarks, with voting@5 reaching a 71.2 average, and also transfers to DPO-trained policy improvements on instruction-following and biomedical evaluations (Liu et al., 9 Oct 2025). CDRRM reports 88.3 average across RewardBench, RMBench, and RMB for the 14B SFT variant, while using only 3k samples for the rubric generator and 3k for the judge (Liu et al., 9 Mar 2026). MRRG improves preference validation accuracy across backbones and raises a Qwen2.5-3B-Instruct base policy from 57.8 to 63.7 on BiGGen and from 25.6 to 32.1 on HealthBench-Hard under RLVR (Fu et al., 2 Jul 2026).

In RL and control, ProgressCounts reaches an average success rate of 0.59 over 20 Bi-DexHands tasks, 13% above human-designed dense rewards and 4% above Eureka, while using only 4 policy samples per task versus Eureka’s 80, a 20× reduction in reward-function sampling (Sarukkai et al., 2024). CARD matches or outperforms baselines across Meta-World and ManiSkill2, showing better or comparable performance to expert-designed rewards on 10 out of 12 tasks and surpassing the oracle on 3 tasks (Sun et al., 2024). The highway driving framework reports a 22% higher average success rate than expert handcrafted rewards across three traffic configurations (Han et al., 2024). In cyber defense, LLM-designed blue-persona rewards change decoy deployment behavior and improve the 95th-percentile time to first impact against stealthy and aggressive red personas, with proactive-v2 reaching 18 against stealthy red and 15 against aggressive red, compared with 13 and 10 for the baseline blue policy (Mukherjee et al., 20 Nov 2025).

In domains with hard external verification, hierarchical or stage-specific modular rewards are especially strong. ChipSeek-R1 achieves state-of-the-art functional correctness on VerilogEval and RTLLM v1.1, generates 27 RTLLM designs surpassing the original human-written PPA, and reports a mean 40.01% EDAP drop among testbench-pass designs (Chen et al., 7 Jul 2025). PARM raises NL4Opt from ER 0.79 and SA 0.39 to ER 0.88 and SA 0.50, and improves GSM8K-stage pipeline accuracy from 0.927 to 0.963 while reaching ER 1.000 (Fan et al., 20 Apr 2026). RLAR reports consistent gains ranging from 10 to 60 across mathematics, coding, translation, and dialogue tasks, and its RewardBench-V2 selector reaches 90.44% accuracy on a 400-instance subset (Feng et al., 28 Feb 2026).

These results collectively suggest that modular rewards are particularly effective when the target objective is structurally heterogeneous: multi-step reasoning, multi-agent coordination, tool-mediated verification, or open-ended evaluation.

6. Reliability, failure modes, and future directions

The literature also identifies recurring pathologies. Reward hacking is explicit in ReSo, where an unfine-tuned PRM yields inflated scores for certain agents, motivating task-specific CRM specialization (Zhou et al., 4 Mar 2025). OpenRubrics notes brittleness from noisy or inconsistent rubrics and introduces preference-label consistency filtering to remove rubrics that do not reproduce the original preference (Liu et al., 9 Oct 2025). CDRRM emphasizes that even rubric-based systems can inherit verbosity and position bias unless criteria are evidence-anchored, atomic, and filtered for pair consistency (Liu et al., 9 Mar 2026). MRRG frames dimensional blind spots as a distinct failure mode of single-role rubric generation and shows that broader coverage reduces rubric hacking by making it harder to optimize only the visible criteria (Fu et al., 2 Jul 2026).

In executable reward-code systems, the dominant risks are semantic misunderstanding, poor scaling, and overfitting to local behavior statistics. The diagnostic-driven refinement study on sparse structured RL identifies reward flooding and semantic/API misunderstanding as the two dominant one-shot failure modes, with large gains from taxonomy-guided debugging in DoorKey-8x8 and KeyCorridor, but also shows that success-based diagnostics can misfire in dense-reward locomotion (Wang et al., 27 May 2026). ProgressCounts shows that asking the LLM to discretize numerically can fail catastrophically, whereas heuristic min/max normalization is robust (Sarukkai et al., 2024). ROS evolution identifies threshold sensitivity in switching between state-selection and operation-optimization phases (Heng et al., 10 Apr 2025). RLAR exposes distinct operational risks, including incorrect tool selection, unreliable code execution, and misleading repository documentation, and therefore inserts verification gates before adding tools to its library (Feng et al., 28 Feb 2026).

Future directions in the corpus are consistent. ReSo points to hierarchical rewards, curriculum learning, dynamic graph adaptation, and joint optimization of CRM-guided coordination with agent-level RL (Zhou et al., 4 Mar 2025). OpenRubrics and CDRRM both imply that better calibration, adversarial filtering, and domain-specific QA of rubric criteria remain open problems (Liu et al., 9 Oct 2025, Liu et al., 9 Mar 2026). MRRG identifies adaptive role selection as a natural extension beyond the fixed five-role pool (Fu et al., 2 Jul 2026). PARM suggests deeper multi-stage pipelines and richer credit propagation across stages (Fan et al., 20 Apr 2026). The endogenous reward perspective suggests a different path altogether: replacing separately trained reward models with prompt-conditioned extraction of latent rewards already present inside foundation models (Li et al., 29 Jun 2025).

A plausible implication is that the field is converging on a layered view of reward generation. At the lowest layer are verifiable local checks, executors, and programmatic modules; above them are semantically explicit criteria, subtasks, and agent- or stage-specific credits; above them are routing, calibration, and selection policies that decide which modules matter for the current input. The central research question is no longer simply how to obtain a reward, but how to organize reward structure so that optimization, attribution, and auditing remain aligned under distribution shift.