Learning a Deep Listwise Context Model for Ranking Refinement (1804.05936v2)

Abstract: Learning to rank has been intensively studied and widely applied in information retrieval. Typically, a global ranking function is learned from a set of labeled data, which can achieve good performance on average but may be suboptimal for individual queries by ignoring the fact that relevant documents for different queries may have different distributions in the feature space. Inspired by the idea of pseudo relevance feedback where top ranked documents, which we refer as the \textit{local ranking context}, can provide important information about the query's characteristics, we propose to use the inherent feature distributions of the top results to learn a Deep Listwise Context Model that helps us fine tune the initial ranked list. Specifically, we employ a recurrent neural network to sequentially encode the top results using their feature vectors, learn a local context model and use it to re-rank the top results. There are three merits with our model: (1) Our model can capture the local ranking context based on the complex interactions between top results using a deep neural network; (2) Our model can be built upon existing learning-to-rank methods by directly using their extracted feature vectors; (3) Our model is trained with an attention-based loss function, which is more effective and efficient than many existing listwise methods. Experimental results show that the proposed model can significantly improve the state-of-the-art learning to rank methods on benchmark retrieval corpora.

Learning a Deep Listwise Context Model for Ranking Refinement

This paper addresses the challenge of refining ranking functions in the domain of information retrieval by proposing a novel Deep Listwise Context Model (DLCM). Conventional learning-to-rank frameworks typically apply a uniform global ranking function trained over extensive datasets, often resulting in suboptimal performance for specific query contexts. This is primarily because such approaches overlook the nuanced variations in feature distributions pertinent to individual queries.

The introduction of the DLCM constitutes an innovative approach to incorporating local ranking context, derived from top-ranked documents, into the ranking refinement process. Specifically, the model leverages recurrent neural networks (RNNs) with gated recurrent units (GRUs) to encode these local contexts. By feeding feature vectors of the top results from a global learning-to-rank model into the RNN sequentially and in reversed order, an embedded local context is efficiently captured, which is then used to upgrade the relevance predictions of the initially ranked list.

The merits of this model include its compatibility with existing learning-to-rank models through direct use of previously extracted feature vectors, its employment of a sophisticated deep neural network architecture to account for inter-document relationships, and its enhanced efficacy via an attention-based listwise loss function that outperforms many existing methods.

Empirical validations of the DLCM were conducted on large-scale datasets, including Microsoft 30k, Microsoft 10k, and the Yahoo! Webscope dataset. The experimental results consistently demonstrate that DLCM can significantly enhance state-of-the-art learning-to-rank methods. It achieved marked improvements in crucial ranking metrics such as NDCG and ERR, particularly at higher positions in the ranked lists. This underscores the aptitude of the DLCM in discerning the most relevant documents amid competitive results, an attribute paramount in high-stakes ranking scenarios.

While the model demonstrates robust performance across different settings, its capability varies slightly with dataset characteristics. For instance, the Yahoo! Webscope dataset, sanitized of less predictive features, showed lesser gains due to the reduced scope for improvement by leveraging local context amid already high-performing global features.

The research implicates significant theoretical and practical advancements. Theoretically, the model shifts the paradigm from global-only ranking functions to a composite approach integrating local queries-adjusted contexts, marking a significant departure in ranking methodologies. Practically, the implications extend to various IR applications, including web search result optimization and personalized recommendations, promising enhancements in relevance and user experience.

Future research directions could explore optimizing the DLCM for increased diversity in ranked outputs or adapting it to more complex, heterogeneous data inputs. Additionally, examining alternative architectures for the recurrent components and loss functions could yield further improvements in ranking effectiveness and computational efficiency.