Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Curator: Efficient Indexing for Multi-Tenant Vector Databases (2401.07119v1)

Published 13 Jan 2024 in cs.DB, cs.DC, cs.IR, and cs.LG

Abstract: Vector databases have emerged as key enablers for bridging intelligent applications with unstructured data, providing generic search and management support for embedding vectors extracted from the raw unstructured data. As multiple data users can share the same database infrastructure, multi-tenancy support for vector databases is increasingly desirable. This hinges on an efficient filtered search operation, i.e., only querying the vectors accessible to a particular tenant. Multi-tenancy in vector databases is currently achieved by building either a single, shared index among all tenants, or a per-tenant index. The former optimizes for memory efficiency at the expense of search performance, while the latter does the opposite. Instead, this paper presents Curator, an in-memory vector index design tailored for multi-tenant queries that simultaneously achieves the two conflicting goals, low memory overhead and high performance for queries, vector insertion, and deletion. Curator indexes each tenant's vectors with a tenant-specific clustering tree and encodes these trees compactly as sub-trees of a shared clustering tree. Each tenant's clustering tree adapts dynamically to its unique vector distribution, while maintaining a low per-tenant memory footprint. Our evaluation, based on two widely used data sets, confirms that Curator delivers search performance on par with per-tenant indexing, while maintaining memory consumption at the same level as metadata filtering on a single, shared index.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Practical and optimal LSH for angular distance. Advances in neural information processing systems 28 (2015).
  2. Multi-tenant databases for software as a service: schema-mapping techniques. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data. 1195–1206.
  3. A comparison of flexible schemas for software as a service. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data. 881–888.
  4. Extensibility and data sharing in evolving multi-tenant databases. In 2011 IEEE 27th international conference on data engineering. IEEE, 99–110.
  5. Artem Babenko and Victor Lempitsky. 2014. The inverted multi-index. IEEE transactions on pattern analysis and machine intelligence 37, 6 (2014), 1247–1260.
  6. Revisiting the inverted indices for billion-scale approximate nearest neighbors. In Proceedings of the European Conference on Computer Vision (ECCV). 202–216.
  7. Jon Louis Bentley. 1975. Multidimensional binary search trees used for associative searching. Commun. ACM 18, 9 (1975), 509–517.
  8. Parallel programming in OpenMP. Morgan kaufmann.
  9. Learning to index for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence 42, 8 (2019), 1942–1956.
  10. On the use of arxiv as a dataset. arXiv preprint arXiv:1905.00075 (2019).
  11. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198.
  12. Sanjoy Dasgupta and Kaushik Sinha. 2013. Randomized partition trees for exact nearest neighbor search. In Conference on learning theory. PMLR, 317–337.
  13. Cong Fu and Deng Cai. 2016. Efanna: An extremely fast approximate nearest neighbor search algorithm based on knn graph. arXiv preprint arXiv:1609.07228 (2016).
  14. Fast approximate nearest neighbor search with the navigating spreading-out graph. Proceedings of the VLDB Endowment 12, 5 (2019), 461–474.
  15. Similarity search in high dimensions via hashing. In Vldb, Vol. 99. 518–529.
  16. Filtered-DiskANN: Graph Algorithms for Approximate Nearest Neighbor Search with Filters. In Proceedings of the ACM Web Conference 2023. 3406–3416.
  17. Ben Harwood and Tom Drummond. 2016. Fanng: Fast approximate nearest neighbour graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5713–5722.
  18. Hugging Face. 2021. sentence-transformers/all-MiniLM-L6-v2 - Hugging Face. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2. Accessed: 2023-11-26.
  19. Supporting database applications as a service. In 2009 IEEE 25th International Conference on Data Engineering. IEEE, 832–843.
  20. Masajiro Iwasaki and Daisuke Miyazaki. 2018. Optimization of indexing based on k-nearest neighbor graph for proximity search in high-dimensional data. arXiv preprint arXiv:1810.07355 (2018).
  21. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in Neural Information Processing Systems 32 (2019).
  22. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence 33, 1 (2010), 117–128.
  23. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 7, 3 (2019), 535–547.
  24. Timothy King. 2021. 80 percent of your data will be unstructured in five years. https://solutionsreview.com/data-management/80-percent-of-your-data-will-be-unstructured-in-five-years/
  25. Sid Lamrous and Mounira Taileb. 2006. Divisive hierarchical k-means. In 2006 International Conference on Computational Inteligence for Modelling Control and Automation and International Conference on Intelligent Agents Web Technologies and International Commerce (CIMCA’06). IEEE, 18–18.
  26. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning. PMLR, 1188–1196.
  27. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–9474.
  28. Dynamic authenticated index structures for outsourced databases. In Proceedings of the 2006 ACM SIGMOD international conference on Management of data. 121–132.
  29. James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, Vol. 1. Oakland, CA, USA, 281–297.
  30. Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824–836.
  31. Marius Muja and David G Lowe. 2009. Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP (1) 2, 331-340 (2009), 2.
  32. Marius Muja and David G Lowe. 2014. Scalable nearest neighbor algorithms for high dimensional data. IEEE transactions on pattern analysis and machine intelligence 36, 11 (2014), 2227–2240.
  33. graph2vec: Learning distributed representations of graphs. arXiv preprint arXiv:1707.05005 (2017).
  34. Deep learning recommendation model for personalization and recommendation systems. arXiv preprint arXiv:1906.00091 (2019).
  35. Pandu Nayak. 2019. Understanding searches better than ever before. https://blog.google/products/search/search-language-understanding-bert/ Accessed: 2023-11-26.
  36. Behnam Neyshabur and Nathan Srebro. 2015. On symmetric and asymmetric lshs for inner product search. In International Conference on Machine Learning. PMLR, 1926–1934.
  37. Stephen M Omohundro. 1989. Five balltree construction algorithms. International Computer Science Institute Berkeley.
  38. OpenAI. 2023. ChatGPT Retrieval Plugin. https://github.com/openai/chatgpt-retrieval-plugin. Accessed: 2023-11-26.
  39. Arash Partow. 2019. C++ Bloom Filter Library. https://github.com/ArashPartow/bloom. Accessed: 2023-11-26.
  40. Pinecone. 2023. Pinecone. https://www.pinecone.io/. Accessed: 2023-11-26.
  41. Qdrant. 2023. Qdrant. https://github.com/qdrant/qdrant. Accessed: 2023-11-26.
  42. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  43. Anshumali Shrivastava and Ping Li. 2014a. Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS). Advances in neural information processing systems 27 (2014).
  44. Anshumali Shrivastava and Ping Li. 2014b. Improved asymmetric locality sensitive hashing (ALSH) for maximum inner product search (MIPS). arXiv preprint arXiv:1410.5410 (2014).
  45. FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search. arXiv preprint arXiv:2105.09613 (2021).
  46. Radu Sion. 2005. Query execution assurance for outsourced databases. In Proceedings of the 31st international conference on Very large data bases. 601–612.
  47. Spotify. 2013. Annoy (Approximate Nearest Neighbors Oh Yeah). https://github.com/spotify/annoy. Accessed: 2023-11-26.
  48. Streaming similarity search over one billion tweets using parallel locality-sensitive hashing. Proceedings of the VLDB Endowment 6, 14 (2013), 1930–1941.
  49. YFCC100M: The new data in multimedia research. Commun. ACM 59, 2 (2016), 64–73.
  50. Milvus: A purpose-built vector data management system. In Proceedings of the 2021 International Conference on Management of Data. 2614–2627.
  51. Weaviate. 2023. Weaviate. https://github.com/weaviate/weaviate. Accessed: 2023-11-26.
  52. Analyticdb-v: A hybrid analytical engine towards query fusion for structured and unstructured data. Proceedings of the VLDB Endowment 13, 12 (2020), 3152–3165.
  53. Craig D Weissman and Steve Bobrowski. 2009. The design of the force. com multitenant internet application development platform. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data. 889–896.
  54. HQANN: Efficient and Robust Similarity Search for Hybrid Queries with Structured and Unstructured Constraints. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 4580–4584.
  55. Complementary hashing for approximate nearest neighbor search. In 2011 International Conference on Computer Vision. IEEE, 1631–1638.
  56. Malkov Yury et al. 2018. hnswlib. https://github.com/nmslib/hnswlib. Accessed: 2023-11-26.
  57. Belinda Zeng. 2022. Go beyond the search box: Introducing multisearch. https://blog.google/products/search/multisearch/
  58. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR) 52, 1 (2019), 1–38.
  59. Composite quantization for approximate nearest neighbor search. In International Conference on Machine Learning. PMLR, 838–846.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets