Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2402.03216v4)

Published 5 Feb 2024 in cs.CL, cs.AI, and cs.LG

Abstract: In this paper, we present a new embedding model, called M3-Embedding, which is distinguished for its versatility in Multi-Linguality, Multi-Functionality, and Multi-Granularity. It can support more than 100 working languages, leading to new state-of-the-art performances on multi-lingual and cross-lingual retrieval tasks. It can simultaneously perform the three common retrieval functionalities of embedding model: dense retrieval, multi-vector retrieval, and sparse retrieval, which provides a unified model foundation for real-world IR applications. It is able to process inputs of different granularities, spanning from short sentences to long documents of up to 8192 tokens. The effective training of M3-Embedding involves the following technical contributions. We propose a novel self-knowledge distillation approach, where the relevance scores from different retrieval functionalities can be integrated as the teacher signal to enhance the training quality. We also optimize the batching strategy, enabling a large batch size and high training throughput to ensure the discriminativeness of embeddings. To the best of our knowledge, M3-Embedding is the first embedding model which realizes such a strong versatility. The model and code will be publicly available at https://github.com/FlagOpen/FlagEmbedding.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. mmarco: A multilingual version of the ms marco passage ranking dataset. arXiv preprint arXiv:2108.13897.
  2. Peter Bühlmann. 2012. Bagging, boosting and ensemble methods. Handbook of computational statistics: Concepts and methods, pages 985–1022.
  3. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174.
  4. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.
  5. Zhuyun Dai and Jamie Callan. 2020. Context-aware term weighting for first stage passage retrieval. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 1533–1536.
  6. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.
  7. Coil: Revisit exact lexical match in information retrieval with contextualized inverted list. arXiv preprint arXiv:2104.07186.
  8. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821.
  9. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents.
  10. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, pages 218–223.
  11. Dureader: a chinese machine reading comprehension dataset from real-world applications. arXiv preprint arXiv:1711.05073.
  12. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 113–122.
  13. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  14. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022.
  15. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146.
  16. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551.
  17. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.
  18. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39–48.
  19. Coliee 2022 summary: Methods for legal document retrieval and entailment. In JSAI International Symposium on Artificial Intelligence, pages 51–67. Springer.
  20. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.
  21. Lecardv2: A large-scale chinese legal case retrieval dataset. arXiv preprint arXiv:2310.17609.
  22. Jimmy Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. arXiv preprint arXiv:2106.14807.
  23. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356–2362.
  24. Pretrained transformers for text ranking: Bert and beyond. Springer Nature.
  25. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics.
  26. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. Trans. Assoc. Comput. Linguistics, 9:1389–1406.
  27. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics, 9:329–345.
  28. Niklas Muennighoff. 2022. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904.
  29. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786.
  30. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005.
  31. Ms marco: A human generated machine reading comprehension dataset. choice, 2640:660.
  32. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.08877.
  33. No language left behind: Scaling human-centered machine translation.
  34. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502.
  35. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2010.08191.
  36. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints.
  37. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, page arXiv:1606.05250.
  38. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
  39. Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking. arXiv preprint arXiv:2110.07367.
  40. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.
  41. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, TBD:TBD.
  42. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6490–6500, Online. Association for Computational Linguistics.
  43. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  44. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
  45. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.
  46. RetroMAE: Pre-training retrieval-oriented language models via masked auto-encoder. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 538–548, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  47. C-pack: Packaged resources to advance general chinese embedding.
  48. T2ranking: A large-scale chinese benchmark for passage ranking. arXiv preprint arXiv:2304.03679.
  49. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808.
  50. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.
  51. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
  52. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. AI Open, 2:65–68.
  53. Adversarial retriever-ranker for dense text retrieval. arXiv preprint arXiv:2110.03611.
  54. Retrieve anything to augment large language models.
  55. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access, 6:74061–74071.
  56. Mr. tydi: A multi-lingual benchmark for dense retrieval. arXiv preprint arXiv:2108.08787.
  57. Toward best practices for training multilingual dense retrieval models. ACM Transactions on Information Systems, 42(2):1–33.
  58. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. Transactions of the Association for Computational Linguistics, 11:1114–1131.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Jianlv Chen (1 paper)
  2. Shitao Xiao (38 papers)
  3. Peitian Zhang (23 papers)
  4. Kun Luo (31 papers)
  5. Defu Lian (142 papers)
  6. Zheng Liu (312 papers)
Citations (153)
View on Semantic Scholar

Summary

  • The paper introduces a self-knowledge distillation strategy that aligns dense, lexical, and multi-vector retrieval methods.
  • The model employs an optimized batching strategy to support both short and long texts across 100 languages.
  • Empirical results show that M3-Embedding outperforms state-of-the-art methods, including BM25 and popular dense retrieval models.

Introduction

Embedding models are integral to natural language processing, serving to convert text into a format that deep neural networks (DNNs) can interpret. The aim is to represent texts in a latent space that captures their semantic meanings, which is crucial for information retrieval (IR) tasks. Traditional embedding models face limitations: they are often language-specific, tailored to single functionalities, and struggle with long inputs. The novel text embedding model, M3-Embedding, addresses these issues head-on by supporting multilingual inputs, unifying numerous retrieval functionalities, and processing input granularities that encompass both short and long documents.

Technical Innovations

M3-Embedding owes much of its proficiency to several key technical innovations. A novel self-knowledge distillation approach is introduced to align multiple retrieval functionalities—dense retrieval, lexical (sparse) retrieval, and multi-vector retrieval—during training. This harmonization allows different retrieval functionalities to mutually enhance each other instead of working in isolation, which is a common pitfall in multi-objective optimization.

Another area of contribution is the batching strategy applied during model training. This technique ensures large batch sizes for embedding training, without compromising on the inclusion of long-sequence data. By grouping and sampling training data based on sequence length, reducing sequence padding, and employing gradient-checkpointing, a higher training throughput is achieved, improving the model's ability to discriminate and produce high-quality embeddings.

Empirical Evidence

Empirical results provide solid evidence for M3-Embedding's superior capabilities. The model not only demonstrates state-of-the-art performance on widely recognized multi-lingual and cross-lingual benchmarks but also maintains this exceptional performance across a notable span of 100 languages. Notably, the model outperforms existing methods, including BM25 as well as several popular dense and multi-vector retrieval methods.

Additionally, M3-Embedding displays outstanding versatility in retrieval tasks. When using different retrieval methods in unison, the model's re-ranking results show further improvement compared to individual functionalities. This underscores the effectiveness of the integrated approach championed by M3-Embedding.

Conclusion

M3-Embedding stands out as a trailblazing model, establishing the first-of-its-kind versatility in text embeddings. By bridging the gap in language support, retrieval functionalities, and input granularity, it sets a new benchmark for IR applications. The extensive dataset curation, innovative self-knowledge distillation framework, and optimized batching strategy underscore the ingenuity behind M3-Embedding’s success.

The shared open-source model and code pave the way for further developments in the text embedding paradigm. Its exemplary IR quality across diverse languages and robust handling of various document lengths position M3-Embedding as a pivotal resource for future research and real-world applications in text-based retrieval systems.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - FlagOpen/FlagEmbedding: Retrieval and Retrieval-augmented LLMs (7,436 stars)