Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learned spatial data partitioning (2306.04846v2)

Published 8 Jun 2023 in cs.DB and cs.AI

Abstract: Due to the significant increase in the size of spatial data, it is essential to use distributed parallel processing systems to efficiently analyze spatial data. In this paper, we first study learned spatial data partitioning, which effectively assigns groups of big spatial data to computers based on locations of data by using machine learning techniques. We formalize spatial data partitioning in the context of reinforcement learning and develop a novel deep reinforcement learning algorithm. Our learning algorithm leverages features of spatial data partitioning and prunes ineffective learning processes to find optimal partitions efficiently. Our experimental study, which uses Apache Sedona and real-world spatial data, demonstrates that our method efficiently finds partitions for accelerating distance join queries and reduces the workload run time by up to 59.4%.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Hadoop-GIS: a high performance spatial data warehousing system over mapreduce. PVLDB 6, 11 (2013), 1009–1020.
  2. Aqwa: adaptive query workload aware partitioning of big spatial data. PVLDB 8, 13 (2015), 2062–2073.
  3. Apache Sedona. 2021. https://sedona.apache.org/.
  4. Multi-step Reinforcement Learning: A Unifying Algorithm. arXiv (2018).
  5. Tsunami: A learned multi-dimensional index for correlated data and skewed workloads. PVLDB (2020), 74–86.
  6. Spatial partitioning techniques in SpatialHadoop. PVLDB 8, 12 (2015), 1602–1605.
  7. Ahmed Eldawy and Mohamed F. Mokbel. 2015. SpatialHadoop: A MapReduce framework for spatial data. In ICDE. 1352–1363.
  8. R. A. Finkel and J. L. Bentley. 1974. Quad Trees a Data Structure for Retrieval on Composite Keys. The Acta Informatica 4, 1 (1974), 1–9.
  9. A Reinforcement Learning Based R-Tree for Spatial Data Indexing in Dynamic Environments.
  10. Deep Q-learning From Demonstrations. In AAAI. 3223–3230.
  11. Learning a Partitioning Advisor for Cloud Databases. In SIGMOD. 143–157.
  12. Kyle D Julian and Mykel J Kochenderfer. 2019. Distributed wildfire surveillance with autonomous aircraft using deep reinforcement learning. Journal of Guidance, Control, and Dynamics 42, 8 (2019), 1768–1778.
  13. Reinforcement learning: A survey. JAIR 4 (1996), 237–285.
  14. An Index Advisor Using Deep Reinforcement Learning. In CIKM. 2105–2108.
  15. LISA: A learned index structure for spatial data. In SIGMOD. 2119–2133.
  16. Yuxi Li. 2017. Deep reinforcement learning: An overview. arXiv (2017).
  17. Geo-ALM: POI Recommendation by Fusing Geographical Information and Adversarial Learning Mechanism.. In IJCAI, Vol. 7. 1807–1813.
  18. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529–533.
  19. Massively Parallel Methods for Deep Reinforcement Learning. arXiv (2015).
  20. Learning multi-dimensional indexes. In SIGMOD. 985–1000.
  21. A deep reinforcement learning framework for rebalancing dockless bike sharing systems. In AAAI, Vol. 33. 1393–1400.
  22. Intercity Simulation of Human Mobility at Rare Events via Reinforcement Learning. In SIGSPATIAL. 293–302.
  23. Effectively learning spatial indices. PVLDB (2020), 2341–2354.
  24. John T. Robinson. 1981. The K-D-B-Tree: A Search Structure for Large Multidimensional Dynamic Indexes. In SIGMOD. 10–18.
  25. Yuya Sasaki. 2021. A Survey on IoT Big Data Analytic Systems: Current and Future. IEEE Internet of Things Journal (2021).
  26. Sequenced route query with semantic hierarchy. In EDBT. 37–48.
  27. Prioritized Experience Replay. arXiv (2016). arXiv:1511.05952
  28. A brief introduction to geospatial big data analytics with apache AsterixDB. In International Workshop on APIs and Libraries for Geospatial Data Science. 1–2.
  29. Locationspark: A distributed in-memory data management system for big spatial data. PVLDB 9, 13 (2016), 1565–1568.
  30. Sebastian Thrun and Michael L Littman. 2000. Reinforcement learning: an introduction. AI Magazine 21, 1 (2000), 103–103.
  31. A Learned Query Optimizer for Spatial Join. In SIGSPATIAL. 458–467.
  32. Using Deep Learning for Big Spatial Data Partitioning. TSAS 7 (08 2020), 1–37.
  33. Incremental partitioning for efficient spatial data analytics. PVLDB 15, 3 (2021), 713–726.
  34. Christopher JCH Watkins and Peter Dayan. 2004. Technical Note: Q-Learning. The Machine Learning 8 (2004), 279–292.
  35. Intellilight: A reinforcement learning approach for intelligent traffic light control. In SIGKDD. 2496–2505.
  36. Simba: Efficient in-memory spatial analytics. In Proceedings of the SIGMOD. 1071–1085.
  37. Qd-tree: Learning data layouts for big data analytics. In SIGMOD. 193–208.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now