Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

pEBR: A Probabilistic Approach to Embedding Based Retrieval (2410.19349v1)

Published 25 Oct 2024 in cs.IR and cs.AI

Abstract: Embedding retrieval aims to learn a shared semantic representation space for both queries and items, thus enabling efficient and effective item retrieval using approximate nearest neighbor (ANN) algorithms. In current industrial practice, retrieval systems typically retrieve a fixed number of items for different queries, which actually leads to insufficient retrieval (low recall) for head queries and irrelevant retrieval (low precision) for tail queries. Mostly due to the trend of frequentist approach to loss function designs, till now there is no satisfactory solution to holistically address this challenge in the industry. In this paper, we move away from the frequentist approach, and take a novel \textbf{p}robabilistic approach to \textbf{e}mbedding \textbf{b}ased \textbf{r}etrieval (namely \textbf{pEBR}) by learning the item distribution for different queries, which enables a dynamic cosine similarity threshold calculated by the probabilistic cumulative distribution function (CDF) value. The experimental results show that our approach improves both the retrieval precision and recall significantly. Ablation studies also illustrate how the probabilistic approach is able to capture the differences between head and tail queries.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Jeffrey Dean. Challenges in building large-scale information retrieval systems. In WSDM, volume 10, 2009.
  2. Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85–117, 2015.
  3. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  4. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  5. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019.
  6. Towards personalized and semantic retrieval: An end-to-endsolution for e-commerce search via embedding learning. In SIGIR, pages 2407–2416, 2020.
  7. Embedding-based retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2553–2561, 2020.
  8. Deep neural networks for youtube recommendations. In RecSys, pages 191–198, 2016.
  9. Learning tree-based deep model for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1079–1088, 2018.
  10. Joint optimization of tree-based index and deep model for recommender systems. Advances in Neural Information Processing Systems, 32, 2019.
  11. Deep retrieval: Learning a retrievable structure for large-scale recommendations. arXiv preprint arXiv:2007.07203, 2020.
  12. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391–407, 1990.
  13. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 2333–2338, 2013.
  14. Learning semantic representations using convolutional neural networks for web search. In Proceedings of the 23rd international conference on world wide web, pages 373–374, 2014.
  15. Joint learning of deep retrieval model and product quantization based embedding index. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1718–1722, 2021.
  16. Jointly optimizing query encoder and product quantization to improve retrieval performance. 2021.
  17. Pre-training tasks for user intent detection and embedding retrieval in e-commerce search. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 4424–4428, 2022.
  18. Pre-trained language model-based retrieval and ranking for web search. ACM Transactions on the Web, 17(1):1–36, 2022.
  19. Transformer memory as a differentiable search index. Advances in Neural Information Processing Systems, 35:21831–21843, 2022.
  20. Generative retrieval with preference optimization for e-commerce search. arXiv preprint arXiv:2407.19829, 2024.
  21. A neural corpus indexer for document retrieval. Advances in Neural Information Processing Systems, 35:25600–25614, 2022.
  22. Query rewriting in taobao search. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 3262–3271, 2022.
  23. Query rewriting via cycle-consistent translation for e-commerce search. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 2435–2446. IEEE, 2021.
  24. Differentiable retrieval augmentation via generative language modeling for e-commerce query intent classification. arXiv preprint arXiv:2308.09308, 2023.
  25. Embedding-based product retrieval in taobao search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3181–3189, 2021.
  26. Modeling user behavior with graph convolution for personalized product search. In Proceedings of the ACM Web Conference 2022, pages 203–212, 2022.
  27. Graph contrastive learning with multi-objective for personalized product retrieval in taobao search. arXiv preprint arXiv:2307.04322, 2023.
  28. Contrastive learning for user sequence representation in personalized product search. 2023.
  29. Semantic-enhanced modality-asymmetric retrieval for online e-commerce search. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3405–3409, 2023.
  30. Modality-balanced embedding for video retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2578–2582, 2022.
  31. Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
  32. Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning. The MIT Press, 2009.
  33. A course in time series analysis, volume 409. Wiley Online Library, 2001.
  34. G.D. Forney. The viterbi algorithm. Proceedings of the IEEE, 61(3):268–278, 1973.
  35. Andrew McCallum. Hidden markov models baum welch algorithm. Introduction to Natural Language Processing CS585; University of Massachusetts Amherst: Massachusetts, USA, 2004.
  36. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.
  37. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022, 2003.
  38. A probabilistic model for learning multi-prototype word embeddings. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 151–160, 2014.
  39. A mixture model for learning multi-sense word embeddings. arXiv preprint arXiv:1706.05111, 2017.
  40. Andrey Malinin. Uncertainty estimation in deep learning with application to spoken language assessment. PhD thesis, University of Cambridge, 2019.
  41. A probabilistic u-net for segmentation of ambiguous images. Advances in neural information processing systems, 31, 2018.
  42. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407, 1990.
  43. Thomas Hofmann. Probabilistic latent semantic indexing. In SIGIR, pages 50–57, 1999.
  44. Semantic hashing. International Journal of Approximate Reasoning, 50(7):969–978, July 2009.
  45. Mixed Negative Sampling for Learning Two-tower Neural Networks in Recommendations. In WWW Companion, pages 441–447, 2020.
  46. Sampling-bias-corrected neural modeling for large corpus item recommendations. In RecSys, pages 269–277, 2019.
  47. Symmetric metric learning with adaptive margin for recommendation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4634–4641, 2020.
  48. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169–174, 2018.
  49. Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652, 2017.
  50. Adaptive hyper-parameter learning for deep semantic retrieval. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 775–782, 2023.
  51. Learning query-aware embedding index for improving e-commerce dense retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3265–3269, 2023.
  52. Learning multi-stage multi-grained semantic embeddings for e-commerce search. In Companion Proceedings of the ACM Web Conference 2023, pages 411–415, 2023.
  53. Multi-interest network with dynamic routing for recommendation at tmall. In CIKM, page 2615–2623, 2019.
  54. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry, pages 253–262, 2004.
  55. Erik Bernhardsson. Annoy: Approximate Nearest Neighbors in C++/Python, 2018.
  56. Accelerating large-scale inference with anisotropic vector quantization. In ICML, pages 3887–3896, 2020.
  57. Deep quantization network for efficient image retrieval. AAAI Press, 2016.
  58. Joint learning of deep retrieval model and product quantization based embedding index. In SIGIR, 2021.
  59. Word representations via gaussian embedding. In International Conference on Learning Representations, 2014.
  60. Probabilistic face embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6902–6911, 2019.
  61. Data uncertainty learning in face recognition. IEEE, 2020.
  62. Modeling uncertainty with hedged instance embedding. 2018.
  63. Stochastic prototype embeddings. 2019.
  64. View-invariant probabilistic embedding for human pose. 2019.
  65. Probabilistic embeddings for speaker diarization, 2020.
  66. Auto-encoding variational bayes. arXiv.org, 2014.
  67. Probabilistic embeddings for cross-modal retrieval. 2021.
  68. Sphereface: Deep hypersphere embedding for face recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6738–6746, 2017.
  69. Cosface: Large margin cosine loss for deep face recognition. pages 5265–5274, 2018.
  70. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  71. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets