Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models

Abstract: Is neural IR mostly hype? In a recent SIGIR Forum article, Lin expressed skepticism that neural ranking models were actually improving ad hoc retrieval effectiveness in limited data scenarios. He provided anecdotal evidence that authors of neural IR papers demonstrate "wins" by comparing against weak baselines. This paper provides a rigorous evaluation of those claims in two ways: First, we conducted a meta-analysis of papers that have reported experimental results on the TREC Robust04 test collection. We do not find evidence of an upward trend in effectiveness over time. In fact, the best reported results are from a decade ago and no recent neural approach comes close. Second, we applied five recent neural models to rerank the strong baselines that Lin used to make his arguments. A significant improvement was observed for one of the models, demonstrating additivity in gains. While there appears to be merit to neural IR approaches, at least some of the gains reported in the literature appear illusory.

A Critical Examination of Neural Ranking Models in Information Retrieval

The paper "Critically Examining the 'Neural Hype': Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models" provides a rigorous analysis of the efficacy of neural ranking models in the field of Information Retrieval (IR), particularly focusing on their performance compared to traditional models. This study explores claims of inflated or misleading improvements reported in the literature due to comparisons with weak baselines.

The paper's contributions are twofold: a meta-analysis of past IR literature and an empirical evaluation of several neural ranking models. The meta-analysis encompasses 109 papers published between 2005 and 2018 that report results using the TREC 2004 Robust Track test collection. A significant finding from this analysis is the persistent issue of weak baselines; 33% of the papers analyzed used baselines performing below the TREC median, and only a small fraction reported results surpassing the best-known results from a decade ago. This lack of upward trend in effectiveness calls into question the substantive advances attributed to neural models over time.

Turning to the empirical evaluation, the study applies five neural models from the MatchZoo toolkit to rerank strong baselines established using the Anserini toolkit. Among the models—DSSM, CDSSM, DRMM, KNRM, and DUET—only DRMM, a model leveraging deep relevance matching, significantly outperformed the strong RM3 baseline in both two-fold and five-fold cross-validation settings. This suggests that while neural networks can offer improvements, such gains are not universally realized across all models.

The paper puts forth several pertinent implications. The enduring comparison to weak baselines can lead to improvements that lack generalizability. Thus, there is a pressing need for the community to adopt more rigorous standards in baseline selection to validate the actual effectiveness of new models. Additionally, the lack of additivity in gain from neural models underscores the critical need for developing models that can harness unique signals or complement existing methods rather than simply mimic existing successful approaches. Furthermore, it emphasizes the importance of transparency and reproducibility in empirical research, as demonstrated by the open availability of the data and code associated with the study.

Looking forward, addressing the challenges outlined by this study could refine the evaluation standards in IR and ensure that reported advances genuinely contribute to the field's state-of-the-art. The findings suggest potential directions for future research, such as exploring innovative combinations of neural and traditional methods that can exploit the complementary strengths of different approaches. Moreover, understanding the data conditions under which neural methods excel could drive the design of more effective models that align with real-world data availability constraints.

In summary, this paper provides a critical reflection on the practices within the IR community regarding baseline selection and effectiveness claims of neural ranking models. By grounding their claims in a systematic meta-analysis and empirical evaluation, the authors contribute to a deeper understanding of the challenges and opportunities in leveraging neural methods for information retrieval tasks.