Curr-RLCER:Curriculum Reinforcement Learning For Coherence Explainable Recommendation

Abstract: Explainable recommendation systems (RSs) are designed to explicitly uncover the rationale of each recommendation, thereby enhancing the transparency and credibility of RSs. Previous methods often jointly predicted ratings and generated explanations, but overlooked the incoherence of such two objectives. To address this issue, we propose Curr-RLCER, a reinforcement learning framework for explanation coherent recommendation with dynamic rating alignment. It employs curriculum learning, transitioning from basic predictions (i.e., click through rating-CTR, selection-based rating) to open-ended recommendation explanation generation. In particular, the rewards of each stage are designed for progressively enhancing the stability of RSs. Furthermore, a coherence-driven reward mechanism is also proposed to enforce the coherence between generated explanations and predicted ratings, supported by a specifically designed evaluation scheme. The extensive experimental results on three explainable recommendation datasets indicate that the proposed framework is effective. Codes and datasets are available at https://github.com/pxcstart/Curr-RLCER.

• The paper introduces Curr-RLCER, a reinforcement learning framework employing curriculum learning to improve coherence in explainable recommendation systems.
• It combines staged tasks for CTR prediction, rating estimation, and explanation generation using advanced reward mechanisms like DPO and GRPO.
• Empirical results demonstrate enhanced accuracy, coherence, and robustness, setting a new benchmark in multi-objective explainable recommendation.

Curriculum Reinforcement Learning for Coherence-Explainable Recommendation: The Curr-RLCER Approach

Introduction

Coherence in explainable recommendation systems presents a persistent challenge, particularly in aligning quantitative ratings with the natural language explanations that accompany them. "Curr-RLCER: Curriculum Reinforcement Learning For Coherence Explainable Recommendation" (2604.05341) confronts this challenge by proposing a reinforcement learning (RL) driven training scheme for LLMs, specifically structured through curriculum learning (CL), to systematically enhance both the quality and the coherence of recommendations and generated explanations. The study’s technical emphasis includes the integration of reward mechanisms and staged RL curricula, culminating in a joint improvement in recommendation accuracy, explainability, and consistency.

Methodological Framework

Curr-RLCER decomposes explainable recommendation into three hierarchically staged tasks, reflecting increasing complexity and generative flexibility for LLMs: (1) click-through rate (CTR) binary classification, (2) rating prediction as multi-class classification, and (3) open-ended natural language explanation generation. To orchestrate stable and efficient multi-objective training, the framework combines advancements in RL from human feedback (RLHF)—Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO)—with curriculum learning principles, ensuring progressive competence build-up in the LLM backbone.

The system’s pipeline is illustrated in (Figure 1).

Figure 1: Overview of the Curr-RLCER framework with three progressively complex RL stages targeting CTR prediction, rating estimation, and explanation generation.

Stage 1 introduces the LLM to binary, discriminative CTR prediction using simple reward signals for correct positive/negative feedback. Stage 2 introduces multi-class discriminative structure for rating prediction, incentivizing proximity to ground-truth using a linear reward function. Stage 3 transitions to the generative task of explanation generation, with hybridized reward functions encompassing conciseness, semantic faithfulness, and—critically—coherence with predicted ratings.

DPO is first invoked to optimize explanation generation via preference pairs (GPT-3.5 generated vs. reference LLM responses), leading to globally more reasonable explanations. Subsequently, GRPO trains the model with group-based feedback, incorporating the coherence-driven reward. For coherence measurement during both training and evaluation, a BERT-based sentiment classifier estimates the polarity and magnitude of the explanation, and the output is compared against both the model-predicted rating and ground-truth.

Coherence Reward and Evaluation Mechanisms

A significant technical novel contribution resides in the explicit modeling of coherence as a reward term. The coherence reward assesses the alignment between the sentiment-imputed rating from the generated explanation and the ground-truth rating, penalizing deviations and thus providing direct RL signal to enforce explanation-rating consistency. Furthermore, the evaluation protocol for coherence is multi-faceted and robust: besides the automated BERT-based classifier, it involves LLM-based (GPT-3.5) rating and human annotation, with aggregated metrics designed to capture both accuracy and mutual consistency.

Experimental Results

Experiments are conducted on three Amazon Review benchmark subsets (Baby, Clothing, Sports). Curr-RLCER is compared against traditional deep explainable recommendation baselines (NRT, Att2Seq, PETER, CER), and the recent LLM-based XRec method leveraging GNNs for collaborative filtering signal.

Explanation Quality and Stability

Curr-RLCER demonstrates consistently superior explainability and stability metrics, measured via GPTScore, BERTScore, BARTScore, BLEURT, and USR. Except for recall in certain BERTScore metrics (where collaborative signals in XRec offer marginal benefit), Curr-RLCER outperforms all baselines in both mean and standard deviation, substantiating its capacity to generate high-quality, semantically faithful explanations.

Rating Prediction

Strong numerical improvements are observed in rating regression: Curr-RLCER achieves RMSE and MAE improvements ranging from 27.1% to 42.5% over the best baseline models, indicating that the staged RL curriculum bolsters not only generative (explanation) alignment but also the core predictive capability of the model.

Coherence Analysis

Coherence evaluation, leveraging the proposed tripartite assessment methodology, shows that Curr-RLCER yields the highest possible consistency between rating and explanation, with coherence metrics (across GPT, classifier, and human annotations) approximately 0.85–0.92—markedly above all competitors and baseline LLMs. Notably, these improvements do not entail any degradation in rating or explanation quality, overriding the typical trade-off encountered in multi-objective architectures.

Robustness to Noise

A robustness study injecting synthetic noise (entity masking) into the input space reveals that Curr-RLCER maintains stable explanation quality and coherence well beyond the operating points of other models, including those utilizing additional collaborative signals. As illustrated in (Figure 2), XRec's performance decays rapidly with high input noise, while Curr-RLCER remains resilient, underscoring the benefit of reward-driven CL RL for generalization.

Figure 2: Robustness analysis—Curr-RLCER maintains explainability and stability across increasing input noise ratios compared to XRec.

Ablation Studies

Excluding DPO training or the coherence reward results in ablation variants with recognizable drops in explanation quality and alignment. The most significant loss arises from removing the DPO component, indicating its central role in generating high-quality text; removing coherence reward leads to higher explanation quality but reduced alignment, confirming the necessity of the multi-objective approach.

Implications and Future Research

Curr-RLCER establishes a technically substantial precedent for staged RLHF with curriculum learning in multi-objective recommender settings, demonstrating that direct RL-based curriculum training can yield coherent, high-quality explanations while increasing predictive accuracy and stability, even with small LLM backbones (as low as 3B parameters). The explicit reward modeling for coherence, combined with a robust, multi-faceted evaluation protocol, provides a practical blueprint for subsequent research on explainable AI systems that must balance accuracy, transparency, and robustness.

Practical implications include improved user trust and system audibility in deployed recommendation engines, increased resilience to input perturbations, and reduced reliance on high-quality supervised data. Theoretically, the paper encourages further exploration of (1) advanced multi-level reward shaping, (2) cross-domain generalization under CL and RLHF, and (3) scaling towards more capable or domain-specialized LLMs under resource constraints.

Conclusion

Curr-RLCER demonstrates that curriculum reinforcement learning, leveraging DPO, GRPO, and a carefully structured coherence reward, resolves a persistent challenge in explainable recommendation: synthesizing high-quality, semantically plausible, and rating-aligned explanations without sacrificing predictive performance. The resulting approach is robust, data-efficient, and extensible, setting a new benchmark for joint reasoning in recommendation-oriented LLMs.