EviRank: Evidence-Based Confidence Estimation for LLM-Based Ranking

Abstract: LLMs show promise for recommendation, but they raise reliability concerns due to limited domain coverage and inherent stochasticity. Existing uncertainty quantification methods persist two fundamental challenges: (1) the global confidence score designed for question answering fails to reveal which positions are unreliable in ranking list; (2) fine-grained confidence extracted from model internals exhibits uniformly low values across all positions, making it impossible to filter unreliable predictions. To tackle the challenges, we propose an evidence-based confidence estimation for LLM-based ranking (EviRank). We extract three complementary evidences from a single forward pass and aggregate them via reliable opinion aggregation. Furthermore, we recognize that ranking positions are inherently unequal, and introduce a position-aware calibration. Lastly, the calibrated confidence guides ranking optimization. Experiments on three datasets demonstrate that our method achieves state-of-the-art performance on both recommendation and uncertainty quantification.

• The paper introduces an evidence-based framework that uses multi-source confidence signals to calibrate and rerank LLM outputs.
• It leverages semantic, attention, and output evidence to create position-aware belief distributions aligned with NDCG weights.
• Empirical evaluations on real-world datasets demonstrate state-of-the-art improvements in ranking metrics and uncertainty quantification.

Evidence-Based Confidence Estimation for LLM-Based Ranking in Recommendation Systems

Motivation and Problem Formulation

Contemporary LLMs provide a unified paradigm for recommender systems by semantically encoding users, items, and contextual histories. However, these models, predominantly trained on general corpora, are prone to reliability issues when exposed to domain-specific or unseen items and exhibit stochasticity across repeated inferences. Existing uncertainty quantification methods, typically designed for question answering, fail to capture the nuanced, position-dependent risks inherent in ranking tasks. For instance, a global confidence score obscures the distinction between highly risky top ranks versus uncertainty-laden lower ranks, which can lead to misleading risk assessments.

Figure 1: Two ranked lists can share an identical global confidence score while embodying drastically different risk profiles, especially with uncertainty concentrated at top positions.

Further, fine-grained metrics leveraging internal LLM activations yield uniformly low and uninformative confidence values across ranks, demonstrating severe calibration issues and minimal discrimination power, even for highly relevant items at top positions.

Figure 2: Empirical label probability distribution across ranking positions evidences underconfidence and lack of discriminative power for ground-truth positions.

The task is thus to develop position-aware, reliable confidence estimation at the ranking level for LLM-based recommenders, supporting both trustworthy delivery and uncertainty-aware downstream processing.

EviRank Framework

The EviRank model introduces an evidence aggregation and calibration framework for LLM-based ranking. The architecture is constructed around three major components: evidence-based position-level confidence estimation, a position-aware calibration schema, and a confidence-guided reranking objective.

Figure 3: EviRank architecture comprises (1) multi-source evidence extraction and belief aggregation, (2) position-aware calibration, and (3) confidence-guided reranking.

Evidence Extraction and Belief Aggregation

EviRank extracts three complementary evidences from a single forward pass:

• Semantic Evidence: Measures the margin in similarity between consecutive items (hidden state embeddings) relative to user preference context, intuitively reflecting the distinction in model certainty between adjacent positions.
• Attention Evidence: Quantifies decision-making focus via normalized attention entropy at each prediction step; low entropy indicates reliance on localized, informative contexts yielding greater reliability.
• Output Evidence: Captures the difference in maximum output token probability between consecutive positions, offering a direct—but contextually normalized—signal of model decisiveness.

Each evidence type is normalized into a Dirichlet-based belief mass, representing the positional reliability and the residual uncertainty (via subjective logic). Reliable opinion aggregation then fuses the three evidences into a unified belief distribution, explicitly weighting sources by their internal reliability (lower uncertainty sources are trusted more). The resulting position-level belief mass forms the foundation for final calibration and downstream use.

Position-Aware Calibration

Recognizing that ranking positions are ordinally asymmetric and require non-uniform treatment, EviRank applies a position-aware sigmoid calibration function modulated by the standard NDCG discount factor. Calibration loss is computed between the calibrated confidence score and the position's ideal NDCG weight for the ground-truth item, explicitly optimizing for relative positional reliability rather than marginal classification accuracy. This step crucially corrects the severe underconfidence and uniformity observed in baseline methods.

Figure 4: Position-aware calibration shifts and sharpens the confidence distribution, increasing its alignment with NDCG-based position importance.

Confidence-Guided Reranking

The final step integrates the calibrated positional confidence (as a reliability weight) with the original LLM ranking score (softmaxed logits), producing an optimized score for each position. Reranking is then supervised through a Bayesian Personalized Ranking loss, encouraging ground-truth promotion among candidates. Joint optimization with the calibration loss (controlled by a λ hyperparameter) ensures that EviRank balances both point-wise calibration fidelity and global ranking efficacy.

Empirical Evaluation and Analysis

EviRank was evaluated on three real-world datasets (MovieLens 1M, Amazon Grocery, Steam) and with multiple LLM backbones (Llama3, Mistral, Qwen2.5). Key findings:

• State-of-the-art ranking metrics (R@5/N@5, R@20/N@20): EviRank consistently surpasses prominent baselines—including LLM4Rerank—across all datasets and LLM backbones, with maximal gains for top-of-list metrics, evidencing the efficacy of position-aware calibration.
• Superior uncertainty quantification: EviRank achieves the highest Kendall’s $\tau$ and Concordance Index (C@K) scores amongst competitive uncertainty quantification baselines, demonstrating a robust correlation between confidence estimates and actual ranking quality.
• Ablation results: Removal of any evidence source, belief aggregation, or calibration module resulted in reduced recommendation and uncertainty quantification performance, corroborating the necessity of all three evidence streams and the aggregation mechanism.
• Calibration effectiveness: EviRank successfully corrects label underconfidence and produces discriminative confidence distributions closely tracking ideal NDCG-related reliability.

Hyperparameter and Efficiency Analysis

Parameter sensitivity analysis for λ shows a predictable tradeoff: higher weight on calibration loss improves discriminative power up to an optimal point, after which ranking performance declines due to over-regularized confidence estimates.

Figure 5: $\lambda$ controls the trade-off between calibration and reranking, with performance peaking at a balanced value.

In terms of computational cost, EviRank adds negligible overhead versus conventional reranking approaches, extracting all evidence in a single forward pass and employing lightweight post-processing. It is also faster than previous multi-pass reranking systems like LLM4Rerank.

Implications and Future Directions

Practically, EviRank delivers more interpretable, robust, and trustworthy rankings from LLM-based recommenders without incurring meaningful latency, a key requirement for production deployment in e-commerce or entertainment platforms managing reliability risk. The evidence aggregation and subjective logic-based fusion approach generalizes well, suggesting applicability to other structured prediction tasks requiring fine-grained uncertainty quantification.

The calibrational objective, anchored to NDCG discounting, is theoretically sound and demonstrates that position-aware confidence estimation outperforms classic binning or pointwise accuracy approaches—aligning the uncertainty model with the actual objectives of recommendation systems.

Future work could extend EviRank to enhanced cold-start handling, domain adaptation, or reinforcement learning paradigms where confidence-driven exploration is essential. Additionally, integrating external knowledge graphs or user feedback mechanisms with the evidence extraction module could further improve reliability and transparency.

Conclusion

EviRank establishes an effective and interpretable framework for position-level confidence estimation in LLM-based ranking. By extracting multi-source evidence, reliably aggregating beliefs, and performing position-aware calibration, it improves both recommendation quality and the fidelity of uncertainty quantification. Its modular structure, computational efficiency, and empirical dominance position it as a standard methodology for trustworthy LLM-powered recommender systems (2606.04727).