From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective (2205.04733v2)

Abstract: Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -- while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -- a sparse expansion-based retriever -- and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained LLM initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, leading to state-of-the-art results in both scenarios for sufficiently expressive models.

Enhancing Sparse Neural Information Retrieval Models through Advanced Training Techniques

Introduction

Sparse representation learning for Information Retrieval (IR) has gained renewed interest in the era of deep learning, aiming to leverage the efficiency of traditional inverted index mechanisms while benefiting from the representational power of neural models. In this context, the paper titled "From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective" presents an extensive empirical paper on improving the effectiveness of sparse neural IR models, focusing on the SPLADE model. By adopting advanced training techniques originally developed for dense models, including knowledge distillation and hard negative mining, the authors investigate the potential of these strategies to enhance sparse representation-based IR models.

Sparse Neural IR Models and SPLADE

Sparse representation learning for IR primarily focuses on generating high-dimensional, yet sparse, term representations that can be efficiently indexed using inverted indices. This approach maintains the explicit lexical matching capabilities of traditional IR systems while introducing the nuanced semantic understanding enabled by neural architectures. SPLADE, a model standing at the core of this paper, epitomizes this line of work by generating sparse document and query representations through a distillation of pre-trained LLMs, specifically leveraging the Masked LLMing (MLM) prediction head for term expansion and importance weighting.

Methodological Enhancements

The paper delineates several augmentations to the basic SPLADE training regime aimed at exploring the full potential of sparse neural IR models. These include:

• Knowledge Distillation: Leveraging MarginMSE loss derived from a teacher cross-encoder model to guide the sparse student model towards more effective representations.
• Hard Negative Mining: Implementing both self and ensemble mining strategies to identify more challenging and informative negative samples during training.
• Enhanced Pre-training: Utilizing a retrieval-oriented pre-trained checkpoint, CoCondenser, as an initialization point for the SPLADE model, potentially imbuing it with richer semantic knowledge beneficial for retrieval tasks.

Experimental Setup and Evaluation

The experiments conducted offer a comprehensive view of the impact of the proposed enhancements across various scenarios, tested on prominent datasets such as MS MARCO, TREC DL 2019, and the BEIR benchmark for zero-shot evaluation. Models were assessed based on their retrieval effectiveness and efficiency (measured in FLOPS), offering insights into the trade-offs between model complexity and performance.

Findings and Implications

The findings reveal significant improvements in both effectiveness and generalization capabilities of the SPLADE model when augmented with the proposed training techniques. Notably, the combination of knowledge distillation, hard negative mining, and advanced pre-training (CoCondenser-EnsembleDistil scenario) achieved state-of-the-art performance across evaluated datasets. These results underscore the potential for sparse neural IR models to benefit from complex training strategies, challenging the prevailing notion that such enhancements are exclusive to dense representation models.

Future Prospects in AI and IR

This paper not only establishes a new benchmark for sparse neural IR models but also opens avenues for further research in merging traditional sparse retrieval mechanisms with advanced neural architectures. The demonstrated scalability and effectiveness of SPLADE, when equipped with modern training techniques, hint at a promising direction for developing efficient and powerful retrieval systems that do not compromise on the interpretability afforded by sparse representations.

Conclusion

The paper's rigorous exploration of advanced training techniques for improving sparse neural IR models exemplifies a significant stride towards harmonizing the depth of neural approaches with the efficiency of sparse representations. As the IR community continues to navigate the challenges of indexing and retrieval in increasingly large information spaces, contributions such as this not only broaden our toolkit but also deepen our understanding of the synergistic potential between old and new.