Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
124 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DREditor: An Time-efficient Approach for Building a Domain-specific Dense Retrieval Model (2401.12540v1)

Published 23 Jan 2024 in cs.IR and cs.CL

Abstract: Deploying dense retrieval models efficiently is becoming increasingly important across various industries. This is especially true for enterprise search services, where customizing search engines to meet the time demands of different enterprises in different domains is crucial. Motivated by this, we develop a time-efficient approach called DREditor to edit the matching rule of an off-the-shelf dense retrieval model to suit a specific domain. This is achieved by directly calibrating the output embeddings of the model using an efficient and effective linear mapping. This mapping is powered by an edit operator that is obtained by solving a specially constructed least squares problem. Compared to implicit rule modification via long-time finetuning, our experimental results show that DREditor provides significant advantages on different domain-specific datasets, dataset sources, retrieval models, and computing devices. It consistently enhances time efficiency by 100-300 times while maintaining comparable or even superior retrieval performance. In a broader context, we take the first step to introduce a novel embedding calibration approach for the retrieval task, filling the technical blank in the current field of embedding calibration. This approach also paves the way for building domain-specific dense retrieval models efficiently and inexpensively.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Predicting efficiency/effectiveness trade-offs for dense vs. sparse retrieval strategy selection. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2862–2866.
  2. Rewriting a deep generative model. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16. Springer, 351–369.
  3. Search and discovery in personal email collections. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 1617–1619.
  4. A study of barriers and benefits of artificial intelligence adoption in small and medium enterprise. Academy of Marketing Studies Journal 26 (2022), 1–6.
  5. A full-text learning to rank dataset for medical information retrieval. In Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20–23, 2016. Proceedings 38. Springer, 716–722.
  6. Yuhao Chen and Farhana Zulkernine. 2021. BIRD-QA: a BERT-based information retrieval approach to domain specific question answering. In 2021 IEEE International Conference On Big Data (Big Data). IEEE, 3503–3510.
  7. Editing Language Model-based Knowledge Graph Embeddings. arXiv preprint arXiv:2301.10405 (2023).
  8. Language model with Plug-in Knowldge Memory. (2023). https://openreview.net/forum?id=Plr5l7r0jY6
  9. Calibrating Factual Knowledge in Pretrained Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2022. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 5937–5947. https://aclanthology.org/2022.findings-emnlp.438
  10. Scheduling beyond CPUs for HPC. In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing. 97–108.
  11. Retrofitting Word Vectors to Semantic Lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1606–1615.
  12. Paolo Ferragina and Ugo Scaiella. 2010. Tagme: on-the-fly annotation of short text fragments (by wikipedia entities). In Proceedings of the 19th ACM international conference on Information and knowledge management. 1625–1628.
  13. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 12799–12807.
  14. Luyu Gao and Jamie Callan. 2021. Condenser: a Pre-training Architecture for Dense Retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 981–993. https://doi.org/10.18653/v1/2021.emnlp-main.75
  15. Goran Glavaš and Ivan Vulić. 2018. Explicit retrofitting of distributional word vectors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 34–45.
  16. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463 (2020).
  17. Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors. In NeurIPS 2022 Workshop on Robustness in Sequence Modeling.
  18. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR, 2790–2799.
  19. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
  20. Unsupervised Dense Information Retrieval with Contrastive Learning. Transactions on Machine Learning Research (2022). https://openreview.net/forum?id=jKN1pXi7b0
  21. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6769–6781. https://doi.org/10.18653/v1/2020.emnlp-main.550
  22. Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for Computing Machinery, New York, NY, USA, 39–48. https://doi.org/10.1145/3397271.3401075
  23. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics 7 (2019), 452–466. https://doi.org/10.1162/tacl_a_00276
  24. Retrofitting Distributional Embeddings to Knowledge Graphs with Functional Relations. In Proceedings of the 27th International Conference on Computational Linguistics. 2423–2436.
  25. WWW’18 Open Challenge: Financial Opinion Mining and Question Answering. In Companion Proceedings of the The Web Conference 2018 (Lyon, France) (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1941–1942. https://doi.org/10.1145/3184558.3192301
  26. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229 (2022).
  27. Memory-based model editing at scale. In International Conference on Machine Learning. PMLR, 15817–15831.
  28. Counter-fitting Word Vectors to Linguistic Constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, 142–148. https://doi.org/10.18653/v1/N16-1018
  29. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the association for Computational Linguistics 5 (2017), 309–324.
  30. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005 (2022).
  31. Costs to consider in adopting NLP for your business. arXiv preprint arXiv:2012.08958 (2020).
  32. Parameter-Efficient Sparse Retrievers and Rerankers Using Adapters. In European Conference on Information Retrieval. Springer, 16–31.
  33. Language Models as Knowledge Bases?. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2463–2473.
  34. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. https://arxiv.org/abs/1908.10084
  35. A thorough examination on zero-shot dense retrieval. arXiv preprint arXiv:2204.12755 (2022).
  36. Injecting logical background knowledge into embeddings for relation extraction. In Proceedings of the 2015 conference of the north American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1119–1129.
  37. Daniel E Rose. 2012. CloudSearch and the democratization of information retrieval. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. 1022–1023.
  38. MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2471–2486.
  39. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3715–3734.
  40. Editing a classifier by rewriting its prediction rules. Advances in Neural Information Processing Systems 34 (2021), 23359–23373.
  41. Virtual Knowledge Graph Construction for Zero-Shot Domain-Specific Document Retrieval. In Proceedings of the 29th International Conference on Computational Linguistics. 1169–1178.
  42. Emotional embeddings: Refining word embeddings to capture emotional content of words. arXiv preprint arXiv:1906.00112 (2019).
  43. A retrofitting model for incorporating semantic relations into word embeddings. In Proceedings of the 28th International Conference on Computational Linguistics. 1292–1298.
  44. Affective Retrofitted Word Embeddings. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. 550–561.
  45. Parameter-efficient prompt tuning makes generalized and calibrated neural text retrievers. arXiv preprint arXiv:2207.07087 (2022).
  46. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=wCu6T5xFjeJ
  47. Domain adaptation for enterprise email search. In Proceedings of the 42nd international ACM SIGIR Conference on Research and Development in Information Retrieval. 25–34.
  48. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).
  49. Fact or Fiction: Verifying Scientific Claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 7534–7550. https://doi.org/10.18653/v1/2020.emnlp-main.609
  50. On calibration and out-of-domain generalization. Advances in neural information processing systems 34 (2021), 2215–2227.
  51. Language models are open knowledge graphs. arXiv preprint arXiv:2010.11967 (2020).
  52. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In International Conference on Learning Representations.
  53. Kformer: Knowledge injection in transformer feed-forward layers. In CCF International Conference on Natural Language Processing and Chinese Computing. Springer, 131–143.
  54. COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 1462–1479. https://aclanthology.org/2022.emnlp-main.95
  55. On Refining BERT Contextualized Embeddings using Semantic Lexicons. In ECML PKDD 2021-Machine Learning with Symbolic Methods and Knowledge Graphs co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases.
  56. Enhancing Zero-Shot Stance Detection via Targeted Background Knowledge. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2070–2075.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now