2000 character limit reached
Minerva: Decentralized Collaborative Query Processing over InterPlanetary File System
Published 6 Oct 2023 in cs.DB and cs.NI | (2310.04342v2)
Abstract: Data silos create barriers in accessing and utilizing data dispersed over networks. Directly sharing data easily suffers from the long downloading time, the single point failure and the untraceable data usage. In this paper, we present Minerva, a peer-to-peer cross-cluster data query system based on InterPlanetary File System (IPFS). Minerva makes use of the distributed Hash table (DHT) lookup to pinpoint the locations that store content chunks. We theoretically model the DHT query delay and introduce the fat Merkle tree structure as well as the DHT caching to reduce it. We design the query plan for read and write operations on top of Apache Drill that enables the collaborative query with decentralized workers. We conduct comprehensive experiments on Minerva, and the results show that Minerva achieves up to query performance acceleration compared to the original IPFS data query, and could complete data analysis queries on the Internet-like environments within an average latency of $0.615$ second. With collaborative query, Minerva could perform up to performance acceleration than centralized query with raw data shipment.
- Y. Tong, X. Pan, Y. Zeng, Y. Shi, C. Xue, Z. Zhou, X. Zhang, L. Chen, Y. Xu, K. Xu, and W. Lv, “Hu-fu: Efficient and secure spatial queries over data federation,” Proc. VLDB Endow., vol. 15, no. 6, p. 1159–1172, jun 2022. [Online]. Available: https://doi.org/10.14778/3514061.3514064
- Y. Zheng, X. Xie, and W.-Y. Ma, “Geolife: A collaborative social networking service among user, location and trajectory,” IEEE Data(base) Engineering Bulletin, June 2010.
- A. Bahmani, K. Ferriter, V. Krishnan, A. Alavi, A. Alavi, P. Tsao, M. Snyder, and C. Pan, “Swarm: A federated cloud framework for large-scale variant analysis,” PLOS Computational Biology, vol. 17, p. e1008977, 05 2021.
- P. Bellavista, L. Foschini, and A. Mora, “Decentralised learning in federated deployment environments: A system-level survey,” ACM Comput. Surv., vol. 54, no. 1, feb 2021. [Online]. Available: https://doi.org/10.1145/3429252
- R. Sethi, M. Traverso, D. Sundstrom, D. Phillips, W. Xie, Y. Sun, N. Yegitbasi, H. Jin, E. Hwang, N. Shingte, and C. Berner, “Presto: Sql on everything,” 04 2019, pp. 1802–1813.
- Z. Luo, L. Niu, V. Korukanti, Y. Sun, M. Basmanova, Y. He, B. Wang, D. Agrawal, H. Luo, C. Tang, A. Singh, Y. Li, P. Du, G. Baliga, and M. Fu, “From batch processing to real time analytics: Running presto® at scale,” 05 2022, pp. 1598–1609.
- V. Raghavan, A. Denissov, F. Guerrero, O. Albertini, D. Bhargov, L. Owen, S. Mani, and L. Jain, “Platform extension framework (pxf): Enabling parallelquery processing over heterogeneous data sources in greenplum,” 2019.
- S. Shukla, “Secure multi-party computation (smc): A review,” 12 2012.
- J. Benet, “IPFS - content addressed, versioned, P2P file system,” CoRR, vol. abs/1407.3561, 2014. [Online]. Available: http://arxiv.org/abs/1407.3561
- M. Hausenblas and J. Nadeau, “Apache drill: Interactive ad-hoc analysis at scale,” Big Data, vol. 1, pp. 100–104, 06 2013.
- S. Babu and H. Herodotou, 2013.
- S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis, “Dremel: Interactive analysis of web-scale datasets,” Commun. ACM, vol. 54, no. 6, p. 114–123, jun 2011. [Online]. Available: https://doi.org/10.1145/1953122.1953148
- K. Shvachko, H. Kuang, S. Radia, and R. Chansler, “The hadoop distributed file system,” in 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), 2010, pp. 1–10.
- H. Krishnan, M. Elayidom, and T. Santhanakrishnan, “Mongodb – a comparison with nosql databases,” International Journal of Scientific and Engineering Research, vol. 7, pp. 1035–1037, 05 2016.
- M. Kornacker, A. Behm, V. Bittorf, T. Bobrovytsky, C. Ching, A. Choi, J. Erickson, M. Grund, D. Hecht, M. Jacobs, I. Joshi, L. Kuff, D. Kumar, A. Leblang, N. Li, I. Pandis, H. Robinson, D. Rorke, S. V. Rus, J. Russell, D. Tsirogiannis, S. Wanderman-Milne, and M. Yoder, “Impala: A modern, open-source sql engine for hadoop,” in CIDR, 2015.
- E. Daniel and F. Tschorsch, “Ipfs and friends: A qualitative comparison of next generation peer-to-peer data networks,” IEEE Communications Surveys Tutorials, vol. 24, no. 1, pp. 31–52, 2022.
- R. C. Merkle, “A digital signature based on a conventional encryption function,” in Advances in Cryptology — CRYPTO ’87, C. Pomerance, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 1988, pp. 369–378.
- A. Sinha, Z. Shen, Y. Song, H. Ma, D. Eide, B.-J. Hsu, and K. Wang, “An overview of microsoft academic service (mas) and applications,” 05 2015, pp. 243–246.
- R. Nambiar and M. Poess, “The making of tpc-ds,” 01 2006, pp. 1049–1058.
- P. Hunt, M. Konar, F. P. Junqueira, and B. Reed, “Zookeeper: Wait-free coordination for internet-scale systems,” in Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference, ser. USENIXATC’10. USA: USENIX Association, 2010, p. 11.
- M. Steiner, D. Carra, and E. W. Biersack, “Faster content access in kad,” in International Conference on Peer-to-peer Computing, 2008.
- X. S. Cai and L. Devroye, “A probabilistic analysis of kademlia networks,” in Algorithms and Computation. Springer Berlin Heidelberg, 2013, pp. 711–721. [Online]. Available: https://doi.org/10.1007%2F978-3-642-45030-3_66
- D. Trautwein, A. Raman, G. Tyson, I. Castro, W. Scott, M. Schubotz, B. Gipp, and Y. Psaras, “Design and evaluation of ipfs: A storage layer for the decentralized web,” 08 2022.
- E. Daniel and F. Tschorsch, “Ipfs and friends: A qualitative comparison of next generation peer-to-peer data networks,” IEEE Communications Surveys & Tutorials, vol. PP, pp. 1–1, 01 2022.
- L. Balduf, S. A. Henningsen, M. Florian, S. Rust, and B. Scheuermann, “Monitoring data requests in decentralized data storage systems: A case study of IPFS,” CoRR, vol. abs/2104.09202, 2021. [Online]. Available: https://arxiv.org/abs/2104.09202
- E. A. Inc., “Pc games,” https://www.origin.com/hkg/zh-tw/, 2023-07-04.
- Chainers, “steepshot-mobile,” https://github.com/Chainers/steepshot-mobile, 2023-07-04.
- “Mediachain,” http://www.mediachain.io/, 2023-07-04.
- “Openbazaar,” https://github.com/OpenBazaar, 2023-07-04.
- D. Wang, S. Shi, Y. Zhu, and Z. Han, “Federated analytics: Opportunities and challenges,” IEEE Network, vol. 36, no. 1, pp. 151–158, 2022.
Paper Prompts
Sign up for free to create and run prompts on this paper.