Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learning

Abstract: Rubric-based rewards are increasingly used for open-ended LLM post-training, but criterion-level scores are often aggregated as independent utilities. This flat scalarization ignores rubric-specified prerequisite and activation relations among criteria, allowing reward or penalty to be counted even when the condition that licenses it is absent. We call this structural reward-aggregation failure \textbf{False Credit Propagation} (FCP). To address this limitation, we propose \ourname (\textbf{G}raphical \textbf{E}vent \textbf{A}ggregation for \textbf{R}ubric rewards), a probabilistic graphical framework for dependency-aware rubric aggregation. \ourname models each criterion outcome as a latent Bernoulli event in a typed rubric graph, propagates soft suppression from unsupported parent events to their children, and aggregates the resulting event probabilities into a normalized expected signed utility. This yields a linear-time reward computation that can be plugged into standard rubric-based RL pipelines without changing the outer optimization algorithm. Experiments on HealthBench, WritingBench, and PLawBench with two policy backbones show that \ourname consistently improves over flat aggregation and deterministic gating, achieving relative gains of up to 15.5\% over flat aggregation. FCP diagnostics further show that \ourname reduces leakage by 96.5\% relative to flat aggregation while preserving more licensed downstream utility than deterministic gating. Our code is publicly available at https://github.com/LvCan926/GEAR.

• The paper presents GEAR, a dependency-aware graphical framework that addresses false credit propagation in rubric-based reinforcement learning.
• Using probabilistic graphical modeling, GEAR constructs rubric-specific DAGs to integrate prerequisite and activation dependencies, enabling precise reward aggregation.
• Empirical results across benchmarks show significant performance gains and a 96.5% reduction in leakage compared to traditional flat aggregation methods.

Probabilistic Graphical Reward Aggregation for Structured Rubric-Based Reinforcement Learning

Problem Statement and Motivation

Rubric-based reward signals are central to reinforcement learning (RL) protocols for open-ended LLM (LM) post-training, enabling fine-grained evaluation of agent behavior when single-reference correctness is unattainable. Existing practice aggregates criterion-level decisions using flat weighted sums, thereby treating each rubric item as an independent source of utility. However, real-world rubrics often exhibit prerequisite (e.g., evidence use dependent on factual grounding) or activation (e.g., penalties that apply only in specific contexts) dependencies among criteria. Ignoring such structure by flat aggregation leads to False Credit Propagation (FCP), where rewards or penalties are accrued absent their licensing conditions. This phenomenon can introduce significant optimization error—reinforcing superficial satisfaction of rubric items while discouraging valid responses where context is misjudged.

Figure 1: Propagation of unsupported credit or penalty in flat aggregation, and suppression of unlicensed downstream utility in GEAR.

GEAR: Dependency-Aware Graphical Reward Aggregation

To address FCP, the paper proposes Graphical Event Aggregation for Rubric rewards (GEAR), a framework combining probabilistic graphical modeling with dependency-aware reward aggregation. Given a query, a candidate response, and a set of weighted rubric criteria, GEAR proceeds as follows:

1. Rubric Graph Construction: A query-specific, typed directed acyclic graph (DAG) over rubric items is produced, encoding prerequisite (weak, strong) and activation dependencies offline from the rubric, decoupled from candidate responses.
2. Latent Event Representation: Each rubric criterion is modeled as a latent Bernoulli variable, with its score interpreted as an event probability estimated by an LLM-as-a-judge model.
3. Monotone Dependency Inference: Child-event probabilities are multiplicatively suppressed according to parental support and the strength of corresponding edges, enforcing monotonicity in dependency retention.
4. Marginal Approximation: Efficient topological marginal computation infers dependency-suppressed event probabilities in linear time.
5. Expected-Utility Aggregation: The normalized, weighted sum of dependency-adjusted event marginal probabilities defines the scalar reward, compatible with prevailing RL fine-tuning methods such as PPO and GRPO.

   Figure 2: Overview of the GEAR framework: query/rubric-specific graph construction, monotone dependency-aware inference, and normalization for policy optimization.

GEAR interpolates between two extremes: (i) reducing to flat aggregation if all dependency retentions are set to one, and (ii) embodying deterministic gating when selected retentions are zero. Default retention factors in experiments are set to 0.6 (weak prerequisite), 0.2 (strong prerequisite), and 0.0 (activation), exposing a meaningful continuum.

Empirical Evaluation

Experimental Protocol

GEAR is evaluated against both flat aggregation and deterministic gating on three open-ended, rubric-structured RL benchmarks: HealthBench (medical dialogue), WritingBench (long-form writing), and PLawBench (case-based legal reasoning). Two backbones are considered: Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, with each method run over identical setups, differing solely in scalar reward aggregation. Additionally, GEAR replaces aggregation modules in existing RL pipelines such as RaR-Explicit, MeRF, and RuscaRL for plug-in assessment.

Main Results

GEAR demonstrates consistent and substantial improvements over flat aggregation and hard gating across all metrics and benchmarks. Notably, on HealthBench-500 validation, average scores increased from 66.4 to 72.6 (Qwen2.5-7B-Instruct) and from 66.6 to 73.9 (Llama-3.1-8B-Instruct), with gains up to 15.5% observed. As a plug-in, GEAR improved established RL pipelines by 5-8 points on average. The benefits are robust across stochastic initialization, as shown by tight standard deviations.

False Credit Propagation Diagnostics

To precisely quantify FCP, edge-level diagnostic metrics compare downstream utility leakage (reward/penalty assigned when parent criterion is violated) and preservation (valid utility retained when dependency is satisfied). GEAR achieves a 96.5% reduction in FCP leakage compared to flat aggregation while maintaining substantially more licensed downstream utility than deterministic gating, which is overly suppressive and discards valid partial credit.

Sensitivity and Ablations

GEAR's performance is robust across retention strengths: higher suppression ($\gamma$) further reduces FCP leakage but impairs performance due to over-suppression; an intermediate setting (default $\gamma=1$) achieves the best reward–leakage trade-off.

Figure 3: GEAR's sensitivity to global suppression strength $\gamma$; optimal point balances score with false credit suppression.

Ablations confirm that topological dependency structure and typed relation strengths are essential for GEAR's gains; random graph rewiring or untyped edges diminish effectiveness, and deterministic gating underperforms flat aggregation due to brittleness against judge noise.

Theoretical and Practical Implications

The results solidify that aggregation-layer modeling is pivotal for RL reward design in rubric-driven post-training. Incorporating semantic dependencies mitigates structural reward noise, improves training signals, and therefore enhances downstream model alignment, generalization, and robustness to judge calibration errors. The probabilistic, monotone propagation mechanism naturally handles uncertainty in local judgments—avoiding the brittleness and all-or-nothing flaws of hard gating, and the unlicensed credit/penalty leakage of flat scalarization.

This work positions GEAR as a principled, efficient, drop-in solution for RL pipelines relying on semantically rich, criterion-dependent rubrics. Offline rubric graph construction ensures scalability, while linear-time marginal inference makes repeated reward computations feasible under large computational budgets.

Future Directions

The principal axes for further research include:

• Edge-specific Retention Learning: Instead of fixed retention factors, one can optimize strengths per edge or per rubric type, using preference-elicitation or held-out calibration.
• Integration with Human Audits: Extending evaluation to human preferences and gold-standard audits would solidify policy improvements beyond criterion-level judge metrics.
• Applicability to Subjective or Causally Entangled Rubrics: Extension to settings with highly subjective criteria or long-range dependencies would probe the representational limits of the dependency model.
• Active Rubric Elicitation: Automating or interactively refining query-specific rubric DAGs may minimize annotation burden while preserving aggregation fidelity.

Conclusion

GEAR systematically addresses a fundamental failure mode—false credit propagation—in structured rubric-driven RL by encoding and operationalizing inter-criterion dependencies at the aggregation layer. Its graphical reward modeling yields statistically significant performance gains across domains and RL architectures, decisively outperforming both conventional flat and hard-dependency aggregation schemes. Practically, this framework can be immediately deployed in any RL/RLHF pipeline with rubrics exhibiting semantic dependencies, facilitating more reliable and interpretable post-training for open-ended LLMs.

---