A Deep Relevance Matching Model for Ad-hoc Retrieval (1711.08611v1)

Abstract: In recent years, deep neural networks have led to exciting breakthroughs in speech recognition, computer vision, and NLP tasks. However, there have been few positive results of deep models on ad-hoc retrieval tasks. This is partially due to the fact that many important characteristics of the ad-hoc retrieval task have not been well addressed in deep models yet. Typically, the ad-hoc retrieval task is formalized as a matching problem between two pieces of text in existing work using deep models, and treated equivalent to many NLP tasks such as paraphrase identification, question answering and automatic conversation. However, we argue that the ad-hoc retrieval task is mainly about relevance matching while most NLP matching tasks concern semantic matching, and there are some fundamental differences between these two matching tasks. Successful relevance matching requires proper handling of the exact matching signals, query term importance, and diverse matching requirements. In this paper, we propose a novel deep relevance matching model (DRMM) for ad-hoc retrieval. Specifically, our model employs a joint deep architecture at the query term level for relevance matching. By using matching histogram mapping, a feed forward matching network, and a term gating network, we can effectively deal with the three relevance matching factors mentioned above. Experimental results on two representative benchmark collections show that our model can significantly outperform some well-known retrieval models as well as state-of-the-art deep matching models.

A Deep Relevance Matching Model for Ad-hoc Retrieval

The paper by Guo et al., presented at CIKM'16, introduces a novel deep neural network-based approach to tackle ad-hoc retrieval tasks, named the Deep Relevance Matching Model (DRMM). This work critically differentiates relevance matching from semantic matching tasks prevalent in NLP, such as paraphrase identification or question answering, and demonstrates how these differences necessitate fundamentally different modeling techniques. The primary focus of this research is to optimize ad-hoc retrieval through explicit modeling of query term importance, exact matching signals, and heterogeneous document lengths.

Key Contributions

The paper makes several significant contributions to the field of Information Retrieval (IR):

1. Discrimination Between Semantic and Relevance Matching:
   • This paper elucidates the core differences between semantic matching and relevance matching. It identifies that relevance matching relies heavily on exact matching signals, the importance of query terms, and can vary greatly due to the diverse lengths of documents. These distinctions are critical as DRMM is specifically tailored to meet the unique requirements of relevance matching.
2. Introduction of a Novel Model Architecture (DRMM):
   • The proposed DRMM employs a joint deep neural network architecture at the query term level and contains three main components: matching histogram mapping, a feed forward matching network, and a term gating network. This architecture collaboratively focuses on exact matching signals, evaluates term importance, and accommodates diverse matching requirements.
3. Experimental Validation:
   • The paper validates the model on two ad-hoc retrieval benchmark collections: Robust04 and ClueWeb-09-Cat-B. Results demonstrate that DRMM significantly outperforms both traditional retrieval methods such as BM25 and several state-of-the-art deep matching models.

Technical Approach and Methodology

Model Components

• Matching Histogram Mapping:

The DRMM uses a novel representation to map variable-length local interactions into fixed-length matching histograms. This technique circumvents the limitations of earlier models that either rely on zero-padding or are position-sensitive. It employs bins to discretize interaction strengths, distinguishing exact matching from similarity-based signals.

• Feed Forward Matching Network:

This component is designed to extract hierarchical matching patterns from matching histograms. Unlike previous interaction-focused models which inadvertently treat all signs equally, this network allows the DRMM to learn the implicit importance of exact matching over semantic similarity.

• Term Gating Network:

The term gating network determines the final relevance score by aggregating individual term scores based on their importance. It uses a softmax function to weigh the contributions of different terms and is evaluated both with term vector importance (TV) and inverse document frequency (IDF).

Experimental Setup

Evaluation metrics included MAP, nDCG@20, and P@20. The effectiveness of DRMM was assessed against traditional retrieval models (QL and BM25) and both representation-focused (DSSM, C-DSSM, ARC-I) and interaction-focused deep matching models (ARC-II and MatchPyramid).

Results

The DRMM displayed superior performance over all baselines, highlighting its efficacy in different metrics on both collections, marking a significant step forward in handling both semantic similarity and term specificity.

• On the Robust04 collection, DRMM$_{LCH\times IDF}$ exhibited a relative improvement in MAP of up to 11.9% over BM25, broadening the utility of DRMM in scenarios where exact matching and relevance ranking are paramount.
• For the ClueWeb-09-Cat-B dataset, DRMM equally demonstrated outstanding performance improvements, underpinning the robustness and scalability of the proposed approach across varying document lengths and genres.

Implications and Future Work

The DRMM's innovative architecture and its rigorous validation imply a substantial advancement in the field of ad-hoc retrieval. Practically, it promises improved relevancy in search results, critical for tasks such as information discovery and content recommendation. Theoretically, the clear distinction and tailored modeling for relevance matching enrich the understanding of text matching dynamics.

Future directions could involve:

• Leveraging larger-scale click-through data to train deeper DRMMs, benefiting from real-world search behavior inclusion.
• Exploring the integration of phrase embeddings to encapsulate better the semantic coherence of query terms, enhancing the relevance of search results.

In summary, this paper by Guo et al. significantly advances the field of IR by proposing and validating the DRMM, providing a substantial leap in understanding and efficient modeling of relevance matching for ad-hoc retrieval tasks.