Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unleashing Graph Partitioning for Large-Scale Nearest Neighbor Search (2403.01797v1)

Published 4 Mar 2024 in cs.DS and cs.IR

Abstract: We consider the fundamental problem of decomposing a large-scale approximate nearest neighbor search (ANNS) problem into smaller sub-problems. The goal is to partition the input points into neighborhood-preserving shards, so that the nearest neighbors of any point are contained in only a few shards. When a query arrives, a routing algorithm is used to identify the shards which should be searched for its nearest neighbors. This approach forms the backbone of distributed ANNS, where the dataset is so large that it must be split across multiple machines. In this paper, we design simple and highly efficient routing methods, and prove strong theoretical guarantees on their performance. A crucial characteristic of our routing algorithms is that they are inherently modular, and can be used with any partitioning method. This addresses a key drawback of prior approaches, where the routing algorithms are inextricably linked to their associated partitioning method. In particular, our new routing methods enable the use of balanced graph partitioning, which is a high-quality partitioning method without a naturally associated routing algorithm. Thus, we provide the first methods for routing using balanced graph partitioning that are extremely fast to train, admit low latency, and achieve high recall. We provide a comprehensive evaluation of our full partitioning and routing pipeline on billion-scale datasets, where it outperforms existing scalable partitioning methods by significant margins, achieving up to 2.14x higher QPS at 90% recall$@10$ than the best competitor.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Optimal hashing-based time-space trade-offs for approximate near neighbors. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, pp.  47–66. SIAM, 2017.
  2. Approximate nearest neighbor search in high dimensions. In Proceedings of the International Congress of Mathematicians: Rio de Janeiro 2018, pp.  3287–3318. World Scientific, 2018.
  3. Efficient indexing of billion-scale datasets of deep descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  2055–2063, 2016.
  4. Benchmarks for billion-scale similarity search. Webpage, 2021. URL https://research.yandex.com/blog/benchmarks-for-billion-scale- similarity-search.
  5. Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighbors. In Computer Vision - ECCV 2018, volume 11216 of Lecture Notes in Computer Science, pp.  209–224. Springer, 2018. doi: 10.1007/978-3-030-01258-8_13. URL https://doi.org/10.1007/978-3-030-01258-8_13.
  6. LSH Forest: Self-tuning Indexes for Similarity Search. In Ellis, A. and Hagino, T. (eds.), Proceedings of the 14th international conference on World Wide Web, WWW 2005, pp.  651–660. ACM, 2005. doi: 10.1145/1060745.1060840. URL https://doi.org/10.1145/1060745.1060840.
  7. Bruch, S. Foundations of Vector Retrieval. arXiv preprint arXiv:2401.09350, 2024.
  8. More Recent Advances in (Hyper)Graph Partitioning. ACM Computing Surveys, 55(12):253:1–253:38, 2023. doi: 10.1145/3571808.
  9. SPANN: Highly-efficient Billion-scale Approximate Nearest Neighborhood Search. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: NeurIPS 2021, pp.  5199–5212, 2021a. URL https://proceedings.neurips.cc/paper/2021/hash/299dc35e747eb77177d9cea10a802da2- Abstract.html.
  10. SPANN paper discussion. https://openreview.net/forum?id=-1rrzmJCp4&noteId=zhMe9y8w25b, 2021b. Accessed: 2023-04-14.
  11. New streaming algorithms for high dimensional emd and mst. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, pp.  222–233, 2022.
  12. Balanced k-means Revisited. Applied Computing and Intelligence, 3(2):145–179, 2023.
  13. Pyramid: A General Framework for Distributed Similarity Search on Large-scale Datasets. In Baru, C. K., Huan, J., Khan, L., Hu, X., Ak, R., Tian, Y., Barga, R. S., Zaniolo, C., Lee, K., and Ye, Y. F. (eds.), 2019 IEEE International Conference on Big Data (IEEE BigData). IEEE, 2019a. doi: 10.1109/BigData47090.2019.9006219. URL https://doi.org/10.1109/BigData47090.2019.9006219.
  14. Pyramid: A General Framework for Distributed Similarity Search. CoRR, abs/1906.10602, 2019b. URL http://arxiv.org/abs/1906.10602.
  15. Concept decompositions for large sparse text data using clustering. Machine Learning, 42(1/2):143–175, 2001. doi: 10.1023/A:1007612920971.
  16. Learning space partitions for nearest neighbor search. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=rkenmREFDr.
  17. The faiss library. arXiv preprint arXiv:2401.08281, 2024.
  18. Unsupervised Space Partitioning for Nearest Neighbor Search. CoRR, abs/2206.08091, 2022. doi: 10.48550/arXiv.2206.08091. URL https://doi.org/10.48550/arXiv.2206.08091.
  19. A linear-time heuristic for improving network partitions. In Proceedings of the 19th Design Automation Conference, DAC 1982, pp.  175–181. ACM/IEEE, 1982. doi: 10.1145/800263.809204. URL https://doi.org/10.1145/800263.809204.
  20. Deep Multilevel Graph Partitioning. In 29th Annual European Symposium on Algorithms, ESA 2021, 2021. doi: 10.4230/LIPIcs.ESA.2021.48. URL https://doi.org/10.4230/LIPIcs.ESA.2021.48.
  21. Accelerating Large-Scale Inference with Anisotropic Vector Quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Learning Research, pp.  3887–3896. PMLR, 2020. URL http://proceedings.mlr.press/v119/guo20h.html.
  22. BLISS: A Billion scale Index using Iterative Re-partitioning. In Zhang, A. and Rangwala, H. (eds.), KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.  486–495. ACM, 2022. doi: 10.1145/3534678.3539414. URL https://doi.org/10.1145/3534678.3539414.
  23. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pp.  604–613, 1998.
  24. OOD-DiskANN: Efficient and Scalable Graph ANNS for Out-of-Distribution Queries. CoRR, abs/2211.12850, 2022. doi: 10.48550/arXiv.2211.12850. URL https://doi.org/10.48550/arXiv.2211.12850.
  25. Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell., 33(1):117–128, 2011a.
  26. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128, 2011b. doi: 10.1109/TPAMI.2010.57. URL https://doi.org/10.1109/TPAMI.2010.57.
  27. Searching in one billion vectors: Re-rank with source coding. In Proceedings of the IEEE International Conference on Acoustics (ICASSP), pp.  861–864. IEEE, 2011c.
  28. Billion-Scale Similarity Search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2021. doi: 10.1109/TBDATA.2019.2921572. URL https://doi.org/10.1109/TBDATA.2019.2921572.
  29. Approximate nearest neighbor search on high dimensional data—experiments, analyses, and improvement. IEEE Transactions on Knowledge and Data Engineering, 32(8):1475–1488, 2019.
  30. Lloyd, S. P. Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2):129–136, 1982. doi: 10.1109/TIT.1982.1056489.
  31. Multi-Probe LSH: Efficient Indexing for High-Dimensional Similarity Search. In Proceedings of the 33rd International Conference on Very Large Data Bases (VLDB) 2007, pp.  950–961. ACM, 2007. URL http://www.vldb.org/conf/2007/papers/research/p950-lv.pdf.
  32. Balanced k-means for clustering. In Structural, Syntactic, and Statistical Pattern Recognition - Joint IAPR International Workshop, S+SSPR 2014, Joensuu, Finland, August 20-22, 2014. Proceedings, volume 8621 of Lecture Notes in Computer Science, pp.  32–41. Springer, 2014. doi: 10.1007/978-3-662-44415-3_4.
  33. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836, 2020. doi: 10.1109/TPAMI.2018.2889473. URL https://doi.org/10.1109/TPAMI.2018.2889473.
  34. Scalable Nearest Neighbor Algorithms for High Dimensional Data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(11):2227–2240, 2014. doi: 10.1109/TPAMI.2014.2321376. URL https://doi.org/10.1109/TPAMI.2014.2321376.
  35. Scalable Recognition with a Vocabulary Tree. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), pp.  2161–2168. IEEE Computer Society, 2006. doi: 10.1109/CVPR.2006.264. URL https://doi.org/10.1109/CVPR.2006.264.
  36. Paz, O. InfiniBand essentials every HPC expert must know. HPC Advisory Council Switzerland Conference, 2014. URL https://www.hpcadvisorycouncil.com/events/2014/swiss-workshop/presos/Day_1/1_Mellanox.pdf.
  37. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp.  1532–1543, 2014.
  38. Artificial Intelligence: a modern approach. Pearson, 3 edition, 2009.
  39. Nearest-neighbor methods in learning and vision. IEEE Trans. Neural Networks, 19(2):377, 2008.
  40. Results of the NeurIPS’21 Challenge on Billion-Scale Approximate Nearest Neighbor Search. In Kiela, D., Ciccone, M., and Caputo, B. (eds.), NeurIPS 2021 Competitions and Demonstrations Track, volume 176 of Proceedings of Machine Learning Research, pp.  177–189. PMLR, 2021. URL https://proceedings.mlr.press/v176/simhadri22a.html.
  41. DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, pp.  13748–13758, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/09853c7fb1d3f8ee67a61b6bf4a7f8e6- Abstract.html.
  42. Soar: Improved indexing for approximate nearest neighbor search. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Citations (1)
View on Semantic Scholar

Summary

  • The paper introduces fast, modular graph partitioning and routing methods that achieve up to 2.14x higher queries per second at a 90% recall rate.
  • It presents novel routing techniques, kRt and hRt, with strong theoretical guarantees, enabling efficient adaptation to various partitioning strategies.
  • Empirical evaluations demonstrate significant performance improvements in distributed approximate nearest neighbor search for billion-scale datasets.

Unleashing Graph Partitioning for Enhanced Large-Scale Nearest Neighbor Search Performance

Introduction to the Research Problem

Nearest neighbor search (NNS) is a computational routine pivotal in a spectrum of applications across computer vision, information retrieval, and machine learning domains. Given the challenges associated with performing NNS in high-dimensional spaces, particularly the curse of dimensionality, approximate nearest neighbor search (ANNS) has emerged as a practical alternative. Traditionally, techniques such as quantization and the employment of various index data structures have been the focus of ANNS solutions. However, the distributed execution of ANNS, necessary for handling datasets of a billion-scale magnitude, introduces unique challenges. This paper addresses the decomposition of a large-scale ANNS problem into smaller sub-problems, utilizing graph partitioning for the efficient and effective distribution of data.

Advancements in Partitioning and Routing

The paper marks a significant step forward in the deployment of graph partitioning methods for ANNS by devising highly efficient routing methods accompanied by strong theoretical performance guarantees. The methods introduced are distinguished by their modularity, allowing for their application with any partitioning methodology.

  1. Fast, Inexact Graph Building: Demonstrates that an approximate k-nearest neighbor graph, despite being built through a simplified, inexact process, can effectively partition the dataset without compromising query performance.
  2. Fast, Accurate, and Modular Combinatorial Routing: Introduces two novel routing methods, named kRt (based on hierarchical k-means) and hRt (based on locality sensitive hashing), which are both efficient and adaptable to various partitioning strategies.
  3. Theoretical Guarantees for Routing: Provides the first analytical exploration of routing performance, presenting theoretical guarantees reaffirming the effectiveness of the hRt method.
  4. Empirical Evaluation: Conducts a thorough empirical analysis revealing that partitions achieved via balanced graph partitioning procured a marked improvement in throughput, surpassing existing methods significantly with up to 2.14x higher queries per second (QPS) at a recall rate of 90%.

Theoretical and Practical Implications

The theoretical contributions elucidate the robust foundations of the proposed routing strategies, notably the hRt variant, by establishing guarantees of routing effectiveness. On a practical level, the empirical results provide convincing evidence of the superiority of graph partitioning, particularly in scenarios characterized by extremely large datasets. Notably, the combinatorial routing methods not only deliver high recall but also demonstrate remarkable speed in training compared to neural network approaches, showcasing a significant reduction in computational resource requirements.

Future Directions

The research opens several avenues for further exploration. More sophisticated routing accuracy and efficiency improvements remain a promising area of research. Investigating routing index compression through quantization could yield more space-efficient solutions without sacrificing performance. Additionally, exploring partitioning cost functions optimized for ANNS beyond the first shard could offer more nuanced shard retrieval strategies. Another area ripe for investigation is the application of these graph partitioning and routing strategies across heterogeneous computing environments, including GPUs, which could further enhance the scalability and efficiency of distributed ANNS systems.

Conclusion

By addressing the limitations of prior work and introducing groundbreaking routing methodologies with strong theoretical underpinnings, this paper significantly advances the field of distributed ANNS. The combined use of balanced graph partitioning with the modular, highly efficient routing methods proposed herein sets a new benchmark for large-scale ANNS performance, potentially reshaping future approaches to handling billion-scale datasets in high-dimensional spaces.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets