Rank, Don't Generate: Statement-level Ranking for Explainable Recommendation

Abstract: Textual explanations, generated with LLMs, are increasingly used to justify recommendations. Yet, evaluating these explanations remains a critical challenge. We advocate a shift in objective: rank, don't generate. We formalize explainable recommendation as a statement-level ranking problem, where systems rank candidate explanatory statements derived from reviews and return the top-k as explanation. This formulation mitigates hallucination by construction and enables fine-grained factual analysis. It also models factor importance through relevance scores and supports standardized, reproducible evaluation with established ranking metrics. Meaningful assessment, however, requires each statement to be explanatory (item facts affecting user experience), atomic (one opinion about one aspect), and unique (paraphrases consolidated), which is challenging to obtain from noisy reviews. We address this with (i) an LLM-based extraction pipeline producing explanatory and atomic statements, and (ii) a scalable, semantic clustering method consolidating paraphrases to enforce uniqueness. Building on this pipeline, we introduce StaR, a benchmark for statement ranking in explainable recommendation, constructed from four Amazon Reviews 2014 product categories. We evaluate popularity-based baselines and state-of-the-art models under global-level (all statements) and item-level (target item statements) ranking. Popularity baselines are competitive in global-level ranking but outperform state-of-the-art models on average in item-level ranking, exposing critical limitations in personalized explanation ranking.

• The paper proposes a ranking-based framework that replaces generative explanations with evidence-grounded statement selection.
• It introduces a robust two-stage LLM extraction and semantic clustering pipeline achieving over 92% explainatoriness and 80% atomicity.
• Empirical results reveal that simple user-history models can rival neural methods, emphasizing a personalization gap in explainable recommendations.

Statement-Level Ranking for Explainable Recommendation: Reformulating Explanation via Ranking Rather than Generation

Motivation and Framework

The prevalent paradigm for explainable recommendation has involved free-form text generation, commonly relying on LLMs to produce natural language rationales. However, such generative approaches face notable challenges in evaluation, reproducibility, and factual accuracy. Common metrics such as n-gram overlap (e.g., BLEU, ROUGE) inadequately address paraphrase robustness, semantic metrics incentivize spurious similarity ungrounded in evidence, and LLM-based evaluation is prompt- and system-dependent, hindering replicability. Furthermore, generative explanation models tend to hallucinate claims not substantiated by source data or regress to generic rationales, diminishing both faithfulness and personalization.

This work advances a decisive reframing: rather than generating explanations, the system should rank candidate explanatory statements derived from user reviews and select the most relevant $k$ as the explanation for each user–item interaction. This paradigm shift (i) constrains explanations to evidence-grounded, atomic, and unique statements, disallowing hallucination and enabling verifiable analysis, (ii) provides relevance scores reflecting factor salience per interaction, and (iii) supports standardized, reproducible evaluation using established ranking metrics.

Extraction and Clustering Pipeline

Achieving high-quality statement-level ranking necessitates a dataset in which all candidate statements are (a) explanatory (item facts likely to affect user experience), (b) atomic (a single opinion on one aspect), and (c) unique (no duplicate paraphrases). Addressing the deficiencies of existing heuristics, the paper introduces a robust LLM-driven, two-stage extraction plus scalable semantic clustering pipeline.

Figure 1: The LLM-based extraction and verification pipeline filters raw reviews to retain only explanatory, atomic, and non-redundant statements.

In the first stage, an LLM extracts candidate statements annotated with polarity. In a subsequent verification stage, another LLM filters non-explanatory, non-atomic, or redundant statements. This two-stage process achieves verified output quality, with human/LLM-based audits confirming $>92\%$ explainatoriness and $>80\%$ atomicity, significantly outperforming prior heuristics.

Figure 2: Scalable semantic paraphrase clustering employs (1) dense embedding retrieval, (2) filtering with a cross-encoder for high-precision matches, and (3) graph-based refinement for intra-cluster cohesion.

For uniqueness, the pipeline groups paraphrases using ANN search with text embeddings (Qwen3-Embedding-0.6B), high-threshold cosine similarity, and cross-encoder filtering (Qwen3-Reranker-0.6B), followed by graph-based connected components and intra-cluster cohesion refinement. This approach consolidates $40\%-55\%$ of unique statements relative to pre-clustering counts, far surpassing n-gram LSH baselines (which reduce by less than $0.2\%$). Embedding-based evaluation confirms both tighter clusters and better semantic separation.

StaR Benchmark: Dataset Construction and Protocol

Leveraging the extraction and clustering pipeline, the StaR benchmark is constructed from four Amazon Reviews 2014 categories (Toys, Clothes, Beauty, Sports), covering 115K–294K interactions and 718K–1.3M user–item–statement triplets post-clustering.

Evaluation is defined in two regimes:

• Global-level ranking: Candidates are drawn from the full statement universe for each interaction, allowing maximal scope for retrieval at the cost of potential item-agnostic outputs.
• Item-level ranking: Candidates are restricted to the statements associated with the target item, isolating the ability to rank the most salient explanation for a specific item-user context.

Standard IR metrics (Precision@$k$, Recall@$k$, NDCG@$k$) are used for evaluation.

Empirical Findings: Models, Baselines, and Personalization Gap

The study conducts comprehensive evaluation comparing state-of-the-art explanation ranking models (BPER+, ExpGCN) and multiple popularity-based baselines (UserPop, ItemPop, GlobalPop), across both ranking regimes.

In global-level ranking, ExpGCN achieves the strongest results, with an NDCG@10 improvement up to $+0.01481$ over the best popularity baseline. Frequency-based baselines (ItemPop, GlobalPop) remain highly competitive, in some cases rivaling neural models, highlighting the enduring influence of item priors in coarse retrieval.

In item-level ranking, a notable and contradictory result emerges: simple user-history popularity (UserPop) matches or even outperforms ExpGCN and BPER+ on average, particularly in the Toys and Sports domains, with a gap up to $−0.08381$ NDCG@10 in favor of UserPop. This exposes a clear personalization gap—current models do not effectively leverage user-specific signals beyond what is captured by historical frequency. Importantly, ItemPop collapses in this setting, confirming that mere item priors no longer suffice once candidates are already item-specific.

Hyperparameter studies further support this observation. In BPER+, global-level performance peaks at intermediate user–item weighting, while item-level ranking strictly favors the user-side as $>92\%$0; for ExpGCN, deeper graphs hurt performance globally but have little impact in the item-level regime.

Figure 3: Performance sensitivity of BPER+ to the $>92\%$1 parameter, with distinct profiles under global- and item-level ranking.

Figure 4: Impact of the number of ExpGCN layers on performance, indicating over-propagation is detrimental globally but less so in item-level tasks.

Implications and Perspective

This research foregrounds statement-level ranking as a reproducible and robust alternative for explainable recommendation, providing clarity and factorization absent from paragraph-based generations. By construction, hallucination is precluded, and explanations become decomposed, auditable, and directly comparable via ranking metrics. The LLM-driven extraction and semantic clustering pipeline is shown to enforce high-quality statement properties required for meaningful benchmarking.

The empirical finding that even sophisticated graph-based models struggle to surpass simple user-history heuristics at the item-specific ranking task has major implications. It suggests that existing models do not sufficiently model fine-grained user preferences or interaction-specific explanatory factors. There is substantial headroom for developing architectures that explicitly integrate personalized explanatory signals, perhaps through attention mechanisms on historical rationale or multi-view user modeling.

The pipeline also lays groundwork for extending benchmarks to graded relevance, incorporating explanation diversity and coverage, and curating higher-order or aspect-level factors. These directions are critical for capturing the multifaceted nature of explanation as perceived by end users.

Conclusion

The paper formalizes and operationalizes statement-level ranking as the preferred formulation for explainable recommendation, arguing for and empirically demonstrating its advantages over generation-based paradigms. The StaR benchmark, supported by robust extraction and clustering methodology, offers a reproducible, granular platform for evaluating explanation models under both retrieval and personalization regimes. The observed performance gap of neural models in item-level ranking underscores the necessity for deeper modeling of user-specific explanatory relevance. This agenda will likely drive future progress in personalized, faithful, and interpretable recommendation systems.

Reference: "Rank, Don't Generate: Statement-level Ranking for Explainable Recommendation" (2604.03724)