Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Search-Adaptor: Embedding Customization for Information Retrieval (2310.08750v3)

Published 12 Oct 2023 in cs.LG

Abstract: Embeddings extracted by pre-trained LLMs have significant potential to improve information retrieval and search. Beyond the zero-shot setup in which they are being conventionally used, being able to take advantage of the information from the relevant query-corpus paired data can further boost the LLM capabilities. In this paper, we propose a novel method, Search-Adaptor, for customizing LLMs for information retrieval in an efficient and robust way. Search-Adaptor modifies the embeddings generated by pre-trained LLMs, and can be integrated with any LLM, including those only available via prediction APIs. On multiple English, multilingual, and multimodal retrieval datasets, we show consistent and significant performance benefits for Search-Adaptor -- e.g., more than 5% improvements for Google Embedding APIs in nDCG@10 averaged over 14 BEIR datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Beir data repository. https://github.com/beir-cellar/beir.
  2. Google cloud text embedding. https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings.
  3. Miracl data repository. https://project-miracl.github.io/.
  4. Openai text embedding. https://platform.openai.com/docs/guides/embeddings/embeddings.
  5. Task-aware retrieval with instructions. arXiv preprint arXiv:2211.09260.
  6. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  7. An analysis of the softmax cross entropy loss for learning-to-rank with binary relevance. In Proceedings of the 2019 ACM SIGIR international conference on theory of information retrieval, pages 75–78.
  8. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89–96.
  9. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  10. Promptagator: Few-shot dense retrieval from 8 examples. arXiv preprint arXiv:2209.11755.
  11. Luyu Gao and Jamie Callan. 2021. Condenser: a pre-training architecture for dense retrieval. arXiv preprint arXiv:2104.08253.
  12. Automatic spatially-aware fashion concept discovery. In Proceedings of the IEEE international conference on computer vision, pages 1463–1471.
  13. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR.
  14. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146.
  15. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  16. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128.
  17. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.
  18. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.
  19. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446.
  20. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
  21. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
  22. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  23. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.06256.
  24. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965.
  25. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.08877.
  26. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899.
  27. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  28. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334.
  29. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.
  30. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918.
  31. One embedder, any task: Instruction-finetuned text embeddings. arXiv preprint arXiv:2212.09741.
  32. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663.
  33. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
  34. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199.
  35. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  36. Making a miracl: Multilingual information retrieval across a continuum of languages. arXiv preprint arXiv:2210.09984.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now