---
title: LLM-Driven Modular Reward Generation
url: https://www.emergentmind.com/topics/llm-driven-modular-reward-generation-6c2ecf9a-d641-462c-9e5f-723f4fcc098b
type: topic
---

# LLM-Driven Modular Reward Generation

Searching arXiv for recent papers on LLM-driven reward generation and rubric-based reward modeling.
LLM-driven modular reward generation is a family of methods in which large language models do not merely emit a single scalar reward, but instead synthesize, decompose, route, or evaluate reward signals through explicit modules such as subtasks, rubric criteria, reward code components, agent-specific credits, tool-selected verifiers, or pipeline stages. Across recent work, the modular unit varies—from DAG nodes in multi-agent reasoning, to rubric items in reward modeling, to programmatic sub-rewards in reinforcement learning, to stage-specific scorers in multi-step pipelines—but the common objective is to replace monolithic reward specification with structured, inspectable, and adaptable reward composition [2503.02390][2510.07743][2410.09187][2410.14660][2502.12130][2604.18327][2607.01830].

## 1. Conceptual foundations

The basic premise is that reward design is often easier to operationalize when the target objective is factored into semantically meaningful parts. In executable reward-code systems, the LLM generates separate terms for properties such as collision avoidance, lane keeping, comfort, progress, action costs, or game-specific constraints, then combines them into an aggregate reward used by RL [2406.10540][2406.11875][2502.10906][2511.16483]. In rubric-based systems, the LLM generates explicit criteria—often divided into hard rules and principles—and the final reward is an aggregation of criterion-level judgments rather than an opaque scalar [2510.07743][2603.08035][2607.01830]. In multi-agent and pipeline settings, modularity appears as per-agent, per-subtask, or per-stage scoring, so that downstream optimization is aligned with intermediate structure rather than only terminal success [2503.02390][2502.03723][2604.18327].

A central distinction is between *generation of reward representations* and *learning from those representations*. Some systems use the LLM to author executable reward code directly, as in ChatPCG, PCGRLLM, CARD, highway driving reward evolution, and autonomous cyber defense [2406.11875][2502.10906][2410.14660][2406.10540][2511.16483]. Others use the LLM to generate structured evaluative artifacts—rubrics, task intents, synthetic negatives, progress functions, or verifiers—which are then converted into reward signals by a separate aggregation or learning stage [2410.09187][2502.12130][2510.07743][2603.08035][2607.01830]. A further line argues that a reward model may already be latent inside the base LLM and can be elicited directly from logits, making reward generation effectively a structured extraction problem rather than a separate training problem [2506.23235].

This suggests that “modular reward generation” is not a single algorithmic template but a design doctrine: align reward units with the causal, semantic, or operational decomposition of the task, then use those units for attribution, calibration, and optimization.

## 2. Main axes of modularity

Three recurrent decomposition axes appear across the literature: task structure, evaluative criteria, and operational tooling.

| Modularity unit | Representative formulation | Example papers |
|---|---|---|
| Subtasks or graph nodes | $R_{\text{total}}(G)=\sum_{v_j\in V} r(a_j^*, v_j)$ | ReSo [2503.02390] |
| Rubric criteria | $r(x,y)=\sum_{k=1}^{K} w_k \cdot s_k(x,y;c_k)$ | OpenRubrics [2510.07743], CDRRM [2603.08035], MRRG [2607.01830] |
| Reward code components | $R(s,a)\approx \sum_i w_i r_i(s,a)$ | PCGRLLM [2502.10906], ChatPCG [2406.11875], driving [2406.10540] |
| Hierarchical gated rewards | $R=\omega_1R_{\mathrm{format}}+\omega_2R_{\mathrm{comp}}+\omega_3R_{\mathrm{func}}+\omega_4R_{\mathrm{syn}}+\omega_5R_{\mathrm{ppa}}$ | ChipSeek-R1 [2507.04736] |
| Per-agent potentials | $r'_i(s,a_i,s')=\gamma\Phi_i(s')-\Phi_i(s)$ | LCA [2502.03723] |
| Tool-selected reward modules | $R(x)=\sum_{m\in\mathcal{M}(x)} w_m(x)\cdot r_m(x)$ | RLAR [2603.00724] |
| Stage-specific scorers | $r^{(F)}_{\theta_F}(P,F)$ and $r^{(S)}_{\theta_S}(x_S(F),S)$ | PARM [2604.18327] |

Task-structured modularity is most explicit in ReSo, where the question is mapped to a DAG, each node is assigned an agent through a two-stage selection process, and the Collaborative Reward Model produces step-wise rewards $r(a_i,v_j)$ that update the Dynamic Agent Database [2503.02390]. Closely related ideas appear in pipeline-adapted reward models, where the stages are not graph nodes but sequential pipeline components; PARM trains separate reward models for formulation and code generation by using downstream execution to assign stage-specific preferences [2604.18327].

Criterion-structured modularity is the defining feature of rubric-based reward modeling. OpenRubrics formalizes the reward as a weighted sum over rubric criteria and explicitly distinguishes hard rules from principles, while MRRG expands the criterion set by eliciting rubrics from multiple evaluator roles and consolidating them into a single auditable scorer [2510.07743][2607.01830]. CDRRM further emphasizes atomic, evidence-anchored rules and preference-consistency filtering, framing rubric construction as contrastive factor extraction followed by synthesis [2603.08035].

Operational modularity appears in systems that treat reward acquisition as tool orchestration. RLAR builds a reward tool library whose modules may be retrieved reward models, synthesized programmatic verifiers, or standard metrics, and selects or synthesizes modules per query [2603.00724]. ARMAP similarly separates explorer LLMs, intent synthesizers, learned reward models, and planners, so that trajectory evaluation is a pluggable component rather than a fixed scalar oracle [2502.12130].

## 3. Formal mechanisms of reward construction

Despite different surface forms, most systems reduce to additive or pairwise-preference formulations. Additive aggregation is the dominant pattern. ReSo aggregates node rewards over a task graph [2503.02390]. OpenRubrics and related rubric systems aggregate criterion scores with weights [2510.07743][2603.08035][2607.01830]. ChatPCG, PCGRLLM, highway driving reward generation, and cyber defense all use weighted sums of executable reward terms, though the weights are often embedded directly in LLM-produced code or YAML specifications rather than learned end-to-end [2406.11875][2502.10906][2406.10540][2511.16483].

A second recurring mechanism is pairwise preference learning. ARMAP trains a reward model on positive and negative trajectories with
$$
\mathcal{L}(\theta) = -\mathbb{E}_{(x_m,h_m^+,h_m^-)}\left[\log \sigma\big(\mathcal{R}_\theta(x_m,h_m^+) - \mathcal{R}_\theta(x_m,h_m^-)\big)\right],
$$
using automatically synthesized task intents and negative trajectories [2502.12130]. LLM4PG adopts a Bradley–Terry-style preference model over trajectory returns learned from LLM judgments on natural-language trajectory abstractions [2406.19644]. PARM uses a DPO-style logistic objective over stage-specific positive and negative pairs derived from execution outcomes in a two-stage pipeline [2604.18327]. CDRRM defines pairwise rubric loss over aggregated rubric scores $s(x,y)$, again using a logistic comparison [2603.08035].

A third mechanism is hierarchical gating. ChipSeek-R1 is the clearest example: syntax, functional correctness, synthesizability, and PPA are separate reward modules, but higher-level rewards are only considered when prerequisite lower-level checks pass. The paper formalizes dependencies such as $R_{\mathrm{func}}=0 \Rightarrow R_{\mathrm{syn}}=0$ and $R_{\mathrm{syn}}=0 \Rightarrow R_{\mathrm{ppa}}=0$, which is explicitly intended to prevent reward hacking and avoid wasting expensive EDA evaluations [2507.04736].

Potential-based shaping remains important in environments where dense local signals are needed but policy invariance is desirable. LCA learns per-agent potentials from LLM-generated pairwise preferences and uses
$$
r'_i(s,a_i,s')=\gamma \Phi_i(s')-\Phi_i(s)
$$
for agent-specific shaping in MARL [2502.03723]. The highway driving framework explicitly invokes potential-based shaping through $F(s,a,s')=\gamma \Phi(s')-\Phi(s)$ to encode longitudinal progress while keeping the reward modular across safety, comfort, speed, and rules [2406.10540].

A more radical formalization is endogenous reward extraction. The “Generalist Reward Models” work defines token-level endogenous reward as an inverse soft Bellman residual derived from logits,
$$
\hat r(s,a)=\hat Q(s,a)-\alpha \log \sum_{a'} \exp(\hat Q(s\oplus a,a')/\alpha),
$$
and shows theoretical equivalence to an offline inverse RL reward on the same data [2506.23235]. In that formulation, modularity can be induced at prompting time by eliciting domain- or criterion-specific rewards from the same base model.

## 4. Representative system families

One major family uses LLMs to write executable reward programs for RL. ChatPCG organizes reward terms around multiplayer game mechanics such as role differentiation, survivability, and damage balance, then iteratively aligns module outputs against gameplay logs [2406.11875]. PCGRLLM extends this with feedback, self-alignment, and reasoning-based prompt engineering, exploring CoT, ToT, and GoT over modular reward code for story-conditioned level generation [2502.10906]. CARD similarly constrains the LLM to emit a reward function that returns both a scalar and a dictionary of sub-rewards, then refines it using process feedback, trajectory feedback, and Trajectory Preference Evaluation [2410.14660]. In autonomous driving and cyber defense, the LLM is given environment code or YAML schemas and asked to instantiate interpretable modules such as collision penalties, speed regulation, decoy placement incentives, and resource costs [2406.10540][2511.16483].

A second family uses LLMs to generate compact latent structures that are subsequently converted into rewards. ProgressCounts asks the LLM to author a progress function $P(s)$, then discretizes progress into bins and applies count-based intrinsic rewards over the induced low-dimensional state space [2410.09187]. The ROS-evolution framework similarly separates reward observation space state selection from operations over those states, using a state execution table as non-Markovian memory over prior exploration [2504.07596]. These methods do not ask the LLM to write the final scalar reward directly; they ask it to produce the right latent coordinates for reward construction.

A third family centers on rubrics and explicit evaluative decomposition. OpenRubrics uses Contrastive Rubric Generation to derive hard rules and principles from chosen–rejected pairs, then trains a rubric generator and rubric-aware judge [2510.07743]. CDRRM extends this with multi-dimensional contrastive profiling, evidence anchoring, pruning, and consistency filtering, emphasizing compact and context-aware rubrics [2603.08035]. MRRG replaces the single evaluator with multiple complementary roles and shows that the resulting scorer can be used both for pairwise judging and for GRPO-style RLVR [2607.01830].

A fourth family uses modular rewards for coordination or planning. ReSo decomposes reasoning tasks into DAGs and assigns heterogeneous agents with CRM-guided fine-grained credit [2503.02390]. LCA generates agent-specific dense rewards from ego-centric language descriptions and learns per-agent potentials from multi-query LLM preferences [2502.03723]. ARMAP, RLAR, and PARM all attach modular rewards to candidate trajectories or pipeline stages, enabling reranking, Reflexion, MCTS, or stage-wise selection without fine-tuning closed policy models [2502.12130][2603.00724][2604.18327].

## 5. Empirical performance and evaluation regimes

Empirical results indicate that modularization is not merely interpretability scaffolding; it often changes optimization behavior materially. ReSo reports 79.1%, 56.2%, and 33.7% on easy, medium, and hard Math-MAS, and 67.3%, 51.3%, and 32.3% on SciBench-MAS, with other MAS frameworks largely failing at hard difficulty [2503.02390]. OpenRubrics reports that Rubric-RM-8B surpasses strong size-matched baselines by 6.8% on average across reward-modeling benchmarks, with voting@5 reaching a 71.2 average, and also transfers to DPO-trained policy improvements on instruction-following and biomedical evaluations [2510.07743]. CDRRM reports 88.3 average across RewardBench, RMBench, and RMB for the 14B SFT variant, while using only 3k samples for the rubric generator and 3k for the judge [2603.08035]. MRRG improves preference validation accuracy across backbones and raises a Qwen2.5-3B-Instruct base policy from 57.8 to 63.7 on BiGGen and from 25.6 to 32.1 on HealthBench-Hard under RLVR [2607.01830].

In RL and control, ProgressCounts reaches an average success rate of 0.59 over 20 Bi-DexHands tasks, 13% above human-designed dense rewards and 4% above Eureka, while using only 4 policy samples per task versus Eureka’s 80, a 20× reduction in reward-function sampling [2410.09187]. CARD matches or outperforms baselines across Meta-World and ManiSkill2, showing better or comparable performance to expert-designed rewards on 10 out of 12 tasks and surpassing the oracle on 3 tasks [2410.14660]. The highway driving framework reports a 22% higher average success rate than expert handcrafted rewards across three traffic configurations [2406.10540]. In cyber defense, LLM-designed blue-persona rewards change decoy deployment behavior and improve the 95th-percentile time to first impact against stealthy and aggressive red personas, with proactive-v2 reaching 18 against stealthy red and 15 against aggressive red, compared with 13 and 10 for the baseline blue policy [2511.16483].

In domains with hard external verification, hierarchical or stage-specific modular rewards are especially strong. ChipSeek-R1 achieves state-of-the-art functional correctness on VerilogEval and RTLLM v1.1, generates 27 RTLLM designs surpassing the original human-written PPA, and reports a mean 40.01% EDAP drop among testbench-pass designs [2507.04736]. PARM raises NL4Opt from ER 0.79 and SA 0.39 to ER 0.88 and SA 0.50, and improves GSM8K-stage pipeline accuracy from 0.927 to 0.963 while reaching ER 1.000 [2604.18327]. RLAR reports consistent gains ranging from 10 to 60 across mathematics, coding, translation, and dialogue tasks, and its RewardBench-V2 selector reaches 90.44% accuracy on a 400-instance subset [2603.00724].

These results collectively suggest that modular rewards are particularly effective when the target objective is structurally heterogeneous: multi-step reasoning, multi-agent coordination, tool-mediated verification, or open-ended evaluation.

## 6. Reliability, failure modes, and future directions

The literature also identifies recurring pathologies. Reward hacking is explicit in ReSo, where an unfine-tuned PRM yields inflated scores for certain agents, motivating task-specific CRM specialization [2503.02390]. OpenRubrics notes brittleness from noisy or inconsistent rubrics and introduces preference-label consistency filtering to remove rubrics that do not reproduce the original preference [2510.07743]. CDRRM emphasizes that even rubric-based systems can inherit verbosity and position bias unless criteria are evidence-anchored, atomic, and filtered for pair consistency [2603.08035]. MRRG frames dimensional blind spots as a distinct failure mode of single-role rubric generation and shows that broader coverage reduces rubric hacking by making it harder to optimize only the visible criteria [2607.01830].

In executable reward-code systems, the dominant risks are semantic misunderstanding, poor scaling, and overfitting to local behavior statistics. The diagnostic-driven refinement study on sparse structured RL identifies reward flooding and semantic/API misunderstanding as the two dominant one-shot failure modes, with large gains from taxonomy-guided debugging in DoorKey-8x8 and KeyCorridor, but also shows that success-based diagnostics can misfire in dense-reward locomotion [2605.28918]. ProgressCounts shows that asking the LLM to discretize numerically can fail catastrophically, whereas heuristic min/max normalization is robust [2410.09187]. ROS evolution identifies threshold sensitivity in switching between state-selection and operation-optimization phases [2504.07596]. RLAR exposes distinct operational risks, including incorrect tool selection, unreliable code execution, and misleading repository documentation, and therefore inserts verification gates before adding tools to its library [2603.00724].

Future directions in the corpus are consistent. ReSo points to hierarchical rewards, curriculum learning, dynamic graph adaptation, and joint optimization of CRM-guided coordination with agent-level RL [2503.02390]. OpenRubrics and CDRRM both imply that better calibration, adversarial filtering, and domain-specific QA of rubric criteria remain open problems [2510.07743][2603.08035]. MRRG identifies adaptive role selection as a natural extension beyond the fixed five-role pool [2607.01830]. PARM suggests deeper multi-stage pipelines and richer credit propagation across stages [2604.18327]. The endogenous reward perspective suggests a different path altogether: replacing separately trained reward models with prompt-conditioned extraction of latent rewards already present inside foundation models [2506.23235].

A plausible implication is that the field is converging on a layered view of reward generation. At the lowest layer are verifiable local checks, executors, and programmatic modules; above them are semantically explicit criteria, subtasks, and agent- or stage-specific credits; above them are routing, calibration, and selection policies that decide which modules matter for the current input. The central research question is no longer simply how to obtain a reward, but how to organize reward structure so that optimization, attribution, and auditing remain aligned under distribution shift.

Source: https://www.emergentmind.com/topics/llm-driven-modular-reward-generation-6c2ecf9a-d641-462c-9e5f-723f4fcc098b