Injecting Domain Adaptation with Learning-to-hash for Effective and Efficient Zero-shot Dense Retrieval (2205.11498v2)

Abstract: Dense retrieval overcome the lexical gap and has shown great success in ad-hoc information retrieval (IR). Despite their success, dense retrievers are expensive to serve across practical use cases. For use cases requiring to search from millions of documents, the dense index becomes bulky and requires high memory usage for storing the index. More recently, learning-to-hash (LTH) techniques, for e.g., BPR and JPQ, produce binary document vectors, thereby reducing the memory requirement to efficiently store the dense index. LTH techniques are supervised and finetune the retriever using a ranking loss. They outperform their counterparts, i.e., traditional out-of-the-box vector compression techniques such as PCA or PQ. A missing piece from prior work is that existing techniques have been evaluated only in-domain, i.e., on a single dataset such as MS MARCO. In our work, we evaluate LTH and vector compression techniques for improving the downstream zero-shot retrieval accuracy of the TAS-B dense retriever while maintaining efficiency at inference. Our results demonstrate that, unlike prior work, LTH strategies when applied naively can underperform the zero-shot TAS-B dense retriever on average by up to 14% nDCG@10 on the BEIR benchmark. To solve this limitation, in our work, we propose an easy yet effective solution of injecting domain adaptation with existing supervised LTH techniques. We experiment with two well-known unsupervised domain adaptation techniques: GenQ and GPL. Our domain adaptation injection technique can improve the downstream zero-shot retrieval effectiveness for both BPR and JPQ variants of the TAS-B model by on average 11.5% and 8.2% nDCG@10 while both maintaining 32$\times$ memory efficiency and 14$\times$ and 2$\times$ speedup respectively in CPU retrieval latency on BEIR. All our code, models, and data are publicly available at https://github.com/thakur-nandan/income.

Injecting Domain Adaptation with Learning-to-hash for Effective and Efficient Zero-shot Dense Retrieval

This paper addresses a significant challenge in neural information retrieval: achieving efficient and effective retrieval across diverse and unseen domains. Traditional dense retrieval models excel at overcoming the lexical gap but incur high memory usage, especially when handling vast document corpora. This paper explores innovative approaches to compress dense representations using learning-to-hash (LTH) techniques, particularly focusing on zero-shot retrieval.

Overview and Methods

Dense retrieval models, while effective, often face scalability issues due to large memory requirements when storing dense indexes. LTH techniques, such as BPR and JPQ, offer solutions by converting dense embeddings into compact binary vectors, thus reducing memory consumption significantly. However, previous efforts have primarily focused on in-domain retrieval, lacking a comprehensive evaluation across multiple domains.

This paper evaluates the downstream zero-shot retrieval performance of the TAS-B dense retriever complemented by LTH strategies on the BEIR benchmark. Notably, the paper identifies a performance gap when LTH techniques are applied naively, observing up to a 14% drop in nDCG@10 compared to the TAS-B dense retriever. To address this shortfall, the authors propose incorporating domain adaptation techniques into the LTH framework.

Two unsupervised domain adaptation strategies, GenQ and GPL, are explored. By integrating these with existing LTH techniques, the paper demonstrates improved retrieval effectiveness. Specifically, the domain adaptation injects significant improvements in nDCG@10: boosting BPR by 11.5% and JPQ by 8.2%, while maintaining substantial memory efficiency and speedup in retrieval latency.

Experimental Results

The experimental results underscore the limitations of applying LTH strategies without domain adaptation. On average, BPR underperformed the zero-shot TAS-B retriever by 5.8 nDCG@10 points when domain adaptation was not used. Conversely, with domain adaptation, both BPR and JPQ achieved remarkable improvements in retrieval accuracy, effectively closing the gap with the zero-shot baseline.

Significantly, the GPL method within the JPQ framework not only improved the model's performance but also surpassed the original TAS-B retriever in a zero-shot setting by 2.0 nDCG@10 points on the BEIR benchmark. This indicates the potential of leveraging domain adaptation to enhance the robustness and applicability of LTH strategies beyond commonly evaluated scenarios.

Practical and Theoretical Implications

The paper's findings highlight the crucial role of domain adaptation in deploying efficient and accurate retrieval systems across varied domains. By effectively injecting domain adaptation into LTH strategies, it becomes feasible to achieve high retrieval quality without compromising on memory efficiency. This has practical implications for real-world applications where memory and computation resources are often constrained.

Theoretically, the paper suggests avenues for future research exploring deeper integration of domain adaptation and LTH methodologies. As AI continues to evolve, fostering generalizations across heterogeneous domains while maintaining efficiency becomes increasingly essential.

Conclusion

This paper provides critical insights into enhancing zero-shot dense retrieval systems through innovative domain adaptation techniques in combination with LTH. By achieving significant improvements in both retrieval accuracy and resource efficiency, the research offers a promising direction for advancing neural information retrieval capabilities. Future developments may explore more robust models and additional compression algorithms to further optimize performance across diverse retrieval tasks.