Rubric-ERL: Empathetic Reinforcement Learning

• The paper introduces a rubric-based RL framework that replaces opaque scalar rewards with multi-dimensional, human-interpretable criteria for empathy and quality.
• It employs a staged training protocol, integrating supervised fine-tuning and policy gradient optimization with explicit rubric scoring for improved alignment.
• Experimental results demonstrate that Rubric-ERL enhances emotion accuracy, persona consistency, and overall interpretability compared to conventional RL methods.

Rubric-as-Judge Empathetic Reinforcement Learning (Rubric-ERL) is a reinforcement learning framework for aligning LLMs with highly structured, human-interpretable, and typically multi-dimensional criteria of empathy, emotional intelligence, and response quality. It is designed to produce dialog agents that reason transparently about user profiles, demonstrate emotionally supportive behavior, and synthesize psychological plausibility with verifiable alignment, as exemplified by the Kardia-R1 system (Yuan et al., 1 Dec 2025).

1. Conceptual Overview

Rubric-ERL systematically replaces coarse or opaque scalar rewards used in conventional RLHF with rubric-based evaluation. Here, a rubric is a formal, multi-dimensional checklist or scoring protocol—crafted by humans, synthesized by LLMs, or dynamically extended via model-to-model comparison—that specifies distinct axes of desired conversational behavior. For empathetic applications, core axes typically include user understanding, contextual reasoning, affect perception, validation, persona/fidelity, and safety.

This methodology addresses two primary limitations of prior work: (1) the inability to learn persistent, user-grounded empathy from situation-centric data, and (2) the lack of interpretability inherent to black-box scalar reward signals. By making reward computation explicit, multi-dimensional, and auditable, Rubric-ERL supports not only improved subjective and automatic alignment metrics, but also forensic inspection of failure modes and response rationales (Yuan et al., 1 Dec 2025, Agnihotri et al., 6 Jun 2025, Yuan et al., 9 Oct 2025).

2. Training Protocols and Pipeline Structure

Rubric-ERL is typically deployed in a multi-stage training regime. The Kardia-R1 pipeline (Yuan et al., 1 Dec 2025) proceeds as follows:

Stage 1: Supervised Fine-Tuning (SFT)

• Train on “easy” data where a synthetic rubric evaluator is confident.
• Input: current user profile and dialog history.
• Target: model outputs are split into <understanding>, <reasoning>, <emotion>, and <response> spans.
• Objective: maximize $\log\pi_\theta(y|c)$ over labeled samples, shaping initial perspective-taking ability.

Stage 2: Rubric-ERL Policy Optimization

• Focuses on “hard” samples requiring nuanced, multi-criteria evaluation.
• For each dialog context:
  • Sample $N$ candidate responses from $\pi_\theta(\cdot|c)$.
  • Each output $o_j$ is independently scored across dimensions: format adherence, emotion matching, and rubric-based judgment.
  • Aggregate reward $$r_j = \lambda_f r_j^{\mathrm{fmt}} + \lambda_e r_j^{\mathrm{emo}} + \lambda_r r_j^{\mathrm{rub}}$$; in Kardia-R1, $\lambda_f = \lambda_e = \lambda_r = 1/3$.
  • Groupwise advantage normalization per context.
  • Optimize a clipped-ratio PPO-style policy gradient objective with KL-anchoring to the SFT policy:

  $$J_{\mathrm{GRPO}}(\theta) = \mathbb{E}_c\left[\frac{1}{N}\sum_{j=1}^N \min\left(\rho_j A_j, \mathrm{clip}(\rho_j, 1-\epsilon, 1+\epsilon)A_j\right)\right] - \beta \mathbb{E}_c \mathrm{D_{KL}}(\pi_\theta(\cdot|c)\|\pi_{\theta_0}(\cdot|c)).$$

• This staged protocol tightly binds the learning objective to interpretable axes of empathetic quality.

3. Rubric Construction and Reward Calculation

Rubric-ERL systems rely on explicit, human-aligned rubrics. Each response is evaluated on axes such as:

1. Relevance to user intent & emotional need
2. Fluency & clarity
3. Empathy (acknowledgement, validation)
4. Persona consistency (tone, style)
5. Safety (harm avoidance)

Scoring is typically performed by a rubric LLM with outputs $S_1, ..., S_D$ on a Likert or categorical scale (e.g., 1–5 per dimension). The scores are aggregated (sum or average) and normalized; e.g., for five 1–5 axes with sum $x$,

$$r_j^{\mathrm{rub}} = \mathrm{Norm}(x) = \frac{x-5}{20} \in [0,1].$$

Empathetic extensions add axes such as emotional resonance, compassion, or active listening (Rezaei et al., 8 Oct 2025, Yuan et al., 1 Dec 2025). All weights and dimensions are transparent and modifiable, enabling both domain specialization and online adaptation.

4. Algorithmic and Modeling Variants

Rubric-ERL can be implemented using various mechanisms for rubric generation, aggregation, and online adaptation.

• Static rubric-based RL: Rubrics are fixed at training start, derived from manual annotation or synthetic LLM generation (Yuan et al., 1 Dec 2025, Liu et al., 9 Oct 2025).
• Contrastive Rubric Generation: Uses preferred/rejected response pairs and an LLM to surface discriminative criteria dynamically (Liu et al., 9 Oct 2025).
• Online Rubrics Elicitation: Rubrics are continuously expanded and re-weighted through pairwise preference learning over model responses, allowing the rubric to track emergent desirable behaviors (Rezaei et al., 8 Oct 2025).
• Multi-Judge Aggregation: Multiple rubric-axes or judge-instances are combined via a supervised aggregator, using either a Generalized Additive Model (GAM) or MLP to estimate a composite reward (Sprejer et al., 29 Oct 2025). This supports persona-based alignment and robustness to rubric drift and judge bias.
• Process-oriented Rubric Rewards: In non-dialog domains (e.g., mathematical reasoning), process-level rubrics evaluate reasoning step-by-step, directly penalizing failure modes such as Miracle Steps and providing credit for partial progress (Yuan et al., 9 Oct 2025).

Rubric-driven judges can be LLMs with minimal LoRA adapters and prompt engineering, yielding both cost-efficient and interpretable reward modeling (Agnihotri et al., 6 Jun 2025).

5. Experimental Performance and Ablation Findings

Experiments with Rubric-ERL consistently demonstrate:

• Improvements in emotion accuracy, empathy scores, persona consistency, and safety on held-out benchmarks (KardiaBench).
• Rubric-ERL yields consistent gains across both subjective (human expert) and objective (GPT-5-mini) judge metrics; for example, SFT-only models show strong factual affect recognition but limited subjective alignment, whereas Rubric-ERL achieves improvements on all five rubric axes, winning majority human preference (Yuan et al., 1 Dec 2025).
• Baselines with embedding-reward RL or scalar RLHF-reward RL produce improvements on fluency or safety at the cost of empathy/perceptual accuracy.
• Groupwise normalization via GRPO ensures learning stability, particularly on “hard” cases where simple reward signals are ambiguous or noisy.
• Multi-judge robust aggregation and online rubric elicitation mitigate sensitivity to individual judge bias or static criteria (Sprejer et al., 29 Oct 2025, Rezaei et al., 8 Oct 2025).
• In the mathematical domain, rubric-based, process-granular rewards reduce invalid reasoning artifacts by 71% and double verified solution rates by promoting sound intermediate reasoning steps (Yuan et al., 9 Oct 2025).

6. Interpretability, Auditability, and Psychological Plausibility

Rubric-ERL enhances interpretability at several levels:

• Axis-level feedback corresponds to discrete psychological or logical constructs—allowing for diagnosis and targeted improvement (e.g., low empathy vs. low persona vs. unsafe content) (Yuan et al., 1 Dec 2025, Sprejer et al., 29 Oct 2025).
• Structured span formats scaffold LLM chain-of-thought, supporting transparent reasoning and explicitly highlighting latent reasoning errors.
• Reward functions and judge rationales are human-readable and modifiable, facilitating post-hoc audits and forensic analysis (Agnihotri et al., 6 Jun 2025).
• OnlineRubrics and multi-judge aggregation ensure that rubrics evolve with user/case diversity and emergent desiderata, maintaining psychological plausibility across operational contexts (Rezaei et al., 8 Oct 2025, Sprejer et al., 29 Oct 2025).

This structure yields agents capable of not only highly accurate but psychologically plausible empathy, where warmth, understanding, and user-specific validation arise directly from the optimized objective.

7. Broader Implications and Extensions

Rubric-ERL generalizes to multiple domains:

• Empathetic dialog, emotional support, and user-grounded conversational agents (Yuan et al., 1 Dec 2025)
• Theory-of-mind, persona routing, and persona-adaptive alignment (Sprejer et al., 29 Oct 2025)
• Scientific QA and factual evaluation with rubric-calibrated optimism/pessimism mitigation (D'Souza et al., 20 May 2025)
• Process reasoning in mathematics and logic (Yuan et al., 9 Oct 2025)
• Contrastive, large-scale alignment with preference-label consistent rubric generation and ensemble judge sampling (Liu et al., 9 Oct 2025)

Key properties of this paradigm are its modularity (new axes/principles can be added online or through persona expansion) and interpretability. Rubric-as-Judge models outperform much larger black-box LLMs when evaluated on alignment, supportiveness, and accuracy, demonstrating that structured, multi-dimensional, human-aligned reward modeling is a critical axis of LLM evolution—not merely scale or data quantity.

A plausible implication is that continued advances will involve richer persona/rubric multiplexing, fine-grained online updating, and increasing fusion of psychological theory with technical rubric construction, with Rubric-ERL providing the foundational infrastructure for such developments.