Explanation Quality Assessment as Ranking with Listwise Rewards

Abstract: We reformulate explanation quality assessment as a ranking problem rather than a generation problem. Instead of optimizing models to produce a single "best" explanation token-by-token, we train reward models to discriminate among multiple candidate explanations and learn their relative quality. Concretely, we construct per-instance candidate sets with graded quality levels and train listwise and pairwise ranking models (ListNet, LambdaRank, RankNet) to preserve ordinal structure and avoid score compression typical of pointwise regression or binary preference objectives. We observe three findings: First, ranking losses consistently outperform regression on score separation across all domains tested. Second, the optimal ranking loss depends on data characteristics: listwise objectives excel with well-separated quality tiers, while pairwise methods are more robust to noisy natural annotations. Third, when trained on carefully curated and well-structured data, small encoder models can match models that are orders of magnitude larger, suggesting that data quality matters more than model scale. Finally, when used as rewards in policy optimization, ranking-based scores enable stable convergence in settings where regression-based rewards fail entirely. Code and data are available at: https://github.com/Tankiit/PPO_Learning_to_rank

• The paper introduces a ranking-based paradigm for quality assessment that overcomes score compression in regression methods, yielding stronger PPO gradients.
• It constructs graded candidate sets with five quality tiers and employs listwise objectives like ListNet to achieve near-optimal NDCG and efficient training.
• The integration with RLHF pipelines demonstrates that well-curated, smaller encoder models can outperform larger LLMs through robust ranking fidelity and reduced compute costs.

Explanation Quality Assessment as Ranking with Listwise Rewards

Reformulation of Explanation Evaluation

The paper "Explanation Quality Assessment as Ranking with Listwise Rewards" (2604.24176) redefines the task of evaluating free-text explanations, shifting from traditional generation or regression-based approaches to a ranking-centric framework. Explanation quality, inherently graded, is often compressed or lost when using pointwise regression or binary objectives. The authors argue that learning-to-rank models, which optimize for the ordinal relation among candidate explanations, retain richer quality signals and better serve as reward models in RLHF pipelines.

Methodological Innovations

A central methodological advance is the construction of per-instance candidate sets graded across five levels (gold, good, fair, poor, nonsense), using both human-written gold explanations (from e-SNLI with high inter-annotator agreement) and deterministic templates for synthetic tiers. Critically, overlapping score ranges are introduced to capture realistic ambiguity and emulate annotation uncertainty found in datasets like ChaosNLI.

Quality assessment is formalized as a ranking problem: given a query $q$ and candidate set $\mathcal{E} = \{e_1, ..., e_n\}$, a scoring function $f_\theta(q, e)$ induces a quality ordering. Dense ranking loss functions (ListNet, RankNet, LambdaRank, ApproxNDCG) are employed over full explanation sets, rather than over binary pairs or regressed values.

The integration with PPO is direct: ranking scores yield advantage estimates that preserve quality gap structure, supporting stable policy optimization.

Figure 1: The ranking-based pipeline retains the full quality signal during training, yielding substantially stronger PPO and superior convergence compared to the generation-centric approach.

Experimental Findings

Reward Modeling and Score Separation

The authors demonstrate that regression and binary preference models suffer from score compression—differences between high and low quality explanations become negligible, undermining PPO gradient signals and leading to ineffective policy updates. By contrast, ranking objectives (especially ListNet) show robust score separation, with separation ratios (e.g., $\sigma(\hat{s})/\sigma(s)$) far exceeding those of regression. Quantitatively, ListNet achieves NDCG@5 $\approx$ 1.0 and separation ratios of 0.92, compared to 0.08 for regression.

Objective Selection and Data Quality

Objective type interacts strongly with the quality and structure of training data. Listwise objectives excel when candidate sets have well-separated quality tiers, while pairwise methods like RankNet cope better with noisy or ambiguous annotations. Notably, the importance of data curation overshadows model scale: small encoder models trained on carefully graded data achieve perfect discrimination (Figure 2), outperforming much larger decoder-only LLMs trained on less diverse data.

Figure 2: Heuristic-based and Graded-Delta methods achieve perfect discrimination (all metrics = 1.0), validating that synthetic quality-graded data captures learnable features.

Efficiency and Model Scale

End-to-end evaluation reveals that ListNet is not only superior in ranking fidelity and PPO convergence, but also more efficient overall. The strong learning signal reduces required epochs and PPO steps, yielding faster training. Encoder models operate at lower compute budgets and process longer contexts, achieving near-optimal NDCG and score separation regardless of parameter count. This indicates that well-structured ranking data allows smaller, cheaper models to perform on par with state-of-the-art LLMs.

Error Analysis and Generalization

Manual error analysis attributes remaining misrankings primarily to ambiguity induced by overlapping graded ranges, subtle reasoning gaps, domain-specific knowledge shortcomings, and annotation error. Cross-dataset validation (on MultiNLI, WinoWhy, Delta-NLI) verifies that ranking-based rewards consistently improve explanation quality and PPO convergence across domains, highlighting the robustness of the approach.

Practical and Theoretical Implications

The findings have several implications:

• Reward modeling for RLHF: Dense listwise rewards are critical for stable and effective policy optimization. Score separation is a necessary condition for meaningful advantage estimation, and ranking objectives reliably achieve this when trained on quality-diverse data.
• Data curation over model scaling: Well-graded, structured candidate sets are a stronger determinant of reward model performance than backbone size, especially for explanation ranking.
• Synthetic grading with realistic ambiguity: Overlapping score ranges and deterministic scoring heuristics allow for scalable construction of quality-diverse datasets, facilitating fine-grained evaluation and training.
• Deployment and compute costs: Efficient encoder architectures trained with listwise ranking objectives offer practical deployment benefits, using less compute while achieving competitive ranking fidelity.
• Generalization of ranking paradigms: The methodology is domain-agnostic and could extend to multi-aspect quality evaluation and continuous feedback signals, not limited to explanation tasks.

Future Developments

Future work should investigate scalable hybrid losses for step-wise reasoning reward modeling, validate synthetic scores against broader human judgments, and extend ranking-based approaches to multi-objective settings (e.g., coherence, informativeness, correctness). Efficient distillation and quantization of ranking reward models will be crucial for real-world RLHF pipelines. Integrating ranking signals into more complex policy optimization strategies and applying them to specialized domains (math, science) constitutes a logical next step.

Conclusion

The ranking-centric paradigm for explanation quality assessment establishes a robust, principled alternative to regression and binary classifications, directly addressing reward signal weakness in RLHF. The combination of listwise ranking objectives and quality-graded candidate sets yields superior ranking fidelity, stronger PPO gradients, and efficient training across architectures. The approach bridges information retrieval and explanation evaluation, laying the groundwork for more nuanced, effective training and assessment pipelines in NLP.