A Theoretical Analysis of NDCG Type Ranking Measures

Abstract: A central problem in ranking is to design a ranking measure for evaluation of ranking functions. In this paper we study, from a theoretical perspective, the widely used Normalized Discounted Cumulative Gain (NDCG)-type ranking measures. Although there are extensive empirical studies of NDCG, little is known about its theoretical properties. We first show that, whatever the ranking function is, the standard NDCG which adopts a logarithmic discount, converges to 1 as the number of items to rank goes to infinity. On the first sight, this result is very surprising. It seems to imply that NDCG cannot differentiate good and bad ranking functions, contradicting to the empirical success of NDCG in many applications. In order to have a deeper understanding of ranking measures in general, we propose a notion referred to as consistent distinguishability. This notion captures the intuition that a ranking measure should have such a property: For every pair of substantially different ranking functions, the ranking measure can decide which one is better in a consistent manner on almost all datasets. We show that NDCG with logarithmic discount has consistent distinguishability although it converges to the same limit for all ranking functions. We next characterize the set of all feasible discount functions for NDCG according to the concept of consistent distinguishability. Specifically we show that whether NDCG has consistent distinguishability depends on how fast the discount decays, and 1/r is a critical point. We then turn to the cut-off version of NDCG, i.e., NDCG@k. We analyze the distinguishability of NDCG@k for various choices of k and the discount functions. Experimental results on real Web search datasets agree well with the theory.

• The paper examines NDCG's convergence behavior and shows that standard logarithmic discounting asymptotically approaches 1 with large item sets.
• The paper introduces the concept of consistent distinguishability, proving that NDCG can reliably differentiate ranking functions despite its convergence properties.
• The paper extends its analysis to polynomial discounts and cut-off methods, identifying key thresholds that improve ranking evaluation in practice.

Theoretical Analysis of NDCG-Type Ranking Measures

The paper presents a comprehensive theoretical investigation into Normalized Discounted Cumulative Gain (NDCG), a prevalent metric used for ranking performance evaluation, particularly in web search and information retrieval contexts. Despite its widespread empirical success, the theoretical understanding of NDCG has remained limited. This paper seeks to bridge that gap by analyzing NDCG's convergence behavior, and its capacity to distinguish between different ranking functions.

Key findings illustrate that traditional NDCG, employing a logarithmic discount function, asymptotically converges to 1 as the number of items increases indefinitely, irrespective of the ranking function employed. This indicates that while NDCG can fail to discern distinctions between highly different ranking systems for vast datasets, it does not inherently lack distinguishing power. The authors introduce the concept of "consistent distinguishability," suggesting that an ideal ranking measure should reliably rank distinct functions across diverse datasets. The paper proves that, notwithstanding its convergence, standard NDCG exhibits consistent distinguishability.

The exploration doesn't stop at the standard logarithmic discount. The paper extends its insights into variations of discount functions, notably polynomial discounts, and delineates the set of feasible discount functions through the lens of consistent distinguishability. The analysis reveals $r^{-1}$ as a critical threshold—functioning as a demarcation line where discounts faster than $r^{-1}$ undermine convergence and distinguishability.

For polynomial discounts slower than this threshold (e.g., $r^{-\beta}$ for $\beta \in (0,1)$), NDCG converges to distinct limits, implying it can differentiate between ranking functions more effectively than the standard version. The findings further extend to cut-off approaches, such as NDCG@k, underscoring that a combination of logarithmic decay and a cut-off creates a potent evaluation function by countering both slow and excessively rapid discounting's pitfalls.

The implications of these findings are significant both theoretically and practically. By elucidating the conditions and characteristics that underpin effective ranking measures, the research offers designers of search engines and recommendation systems a nuanced understanding of the trade-offs involved in selecting or developing ranking metrics. The results emphasize a pivotal balance; while a measure must be sensitive enough to pick up differences in function performance, it should also remain computationally feasible and possessing adequate convergence properties.

Future research should focus on combining these theoretical insights with user interface evaluation to holistically assess ranking measures' effectiveness within real-world applications. Additionally, extending this framework to other prevalent ranking measures could potentially fortify and broaden the understanding of ranking evaluation in data-rich environments.

Overall, this paper significantly advances theoretical comprehension of NDCG and provides a foundation for future explorations of ranking measure design and evaluation. The marriage of empirical observations with rigorous theoretical analysis ensures that the outcomes hold considerable utility across multiple domains where ranking is paramount.