Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approach (2401.09455v1)

Published 23 Dec 2023 in cs.NI, cs.AI, cs.LG, cs.SY, and eess.SY

Abstract: The integrated satellite-terrestrial network (ISTN) system has experienced significant growth, offering seamless communication services in remote areas with limited terrestrial infrastructure. However, designing a routing scheme for ISTN is exceedingly difficult, primarily due to the heightened complexity resulting from the inclusion of additional ground stations, along with the requirement to satisfy various constraints related to satellite service quality. To address these challenges, we study packet routing with ground stations and satellites working jointly to transmit packets, while prioritizing fast communication and meeting energy efficiency and packet loss requirements. Specifically, we formulate the problem of packet routing with constraints as a max-min problem using the Lagrange method. Then we propose a novel constrained Multi-Agent reinforcement learning (MARL) dynamic routing algorithm named CMADR, which efficiently balances objective improvement and constraint satisfaction during the updating of policy and Lagrange multipliers. Finally, we conduct extensive experiments and an ablation study using the OneWeb and Telesat mega-constellations. Results demonstrate that CMADR reduces the packet delay by a minimum of 21% and 15%, while meeting stringent energy consumption and packet loss rate constraints, outperforming several baseline algorithms.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. TelesatCanada, “Sat-mpl-20200526-00053.pending approval.” 2020.
  2. W. S. Limited, “Sat-mpl-20210112-00007.pending approval.” 2021.
  3. Y. Yang, M. Xu, D. Wang, and Y. Wang, “Towards energy-efficient routing in satellite networks,” IEEE Journal on Selected Areas in Communications, vol. 34, no. 12, pp. 3869–3886, 2016.
  4. M. Jia, S. Zhu, L. Wang, Q. Guo, H. Wang, and Z. Liu, “Routing algorithm with virtual topology toward to huge numbers of leo mobile satellite network based on sdn,” Mobile Networks and Applications, vol. 23, p. 285–300, 2018.
  5. Z. Yaoying, W. Qian, L. Zeqi, and L. Hewu, “Enabling low-latency-capable satellite-ground topology for emerging leo satellite networks,” in IEEE INFOCOM 2022 - IEEE Conference on Computer Communications, 2022, pp. 1329–1338.
  6. Y. Huang, S. Wu, Z. Kand, Z. Mu, H. Huang, X. Wu, A. J. Tang, and X. Cheng, “Reinforcement learning based dynamic distributed routing scheme for mega leo satellite networks,” Chinese Journal of Aeronautics, vol. 36, no. 2, pp. 284–291, 2023.
  7. L. Jiahao, Z. Baokang, X. Qin, S. Jinshu, and O. Wei, “Drl-er: An intelligent energy-aware routing protocol with guaranteed delay bounds in satellite mega-constellations,” IEEE Transactions on Network Science and Engineering, vol. 8, no. 4, pp. 2872–2884, 2021.
  8. R. Mauger and C. Rosenberg, “Qos guarantees for multimedia services on a tdma-based satellite network,” IEEE Communications Magazine, vol. 35, no. 7, pp. 56–65, 1997.
  9. E. Ekic, I. Akyildiz, and M. Bender, “A distributed routing algorithm for datagram traffic in leo satellite networks,” IEEE/ACM Transactions on networking, vol. 9, no. 2, pp. 137–147, 2001.
  10. M. Werner, C. Delucchi, H.-J. Vogel, G. Maral, and J.-J. De Ridder, “Atm-based routing in leo/meo satellite networks with intersatellite links,” IEEE Journal on Selected Areas in Communications, vol. 15, no. 1, pp. 69–82, 1997.
  11. Y. Lu, F. Sun, and Y. Zhao, “Virtual topology for leo satellite networks based on earth-fixed footprint mode,” IEEE communications letters, vol. 17, no. 2, pp. 357–360, 2013.
  12. J. Wang, L. Li, and M. Zhou, “Topological dynamics characterization for leo satellite networks,” Computer Networks, vol. 51, no. 1, pp. 43–53, 2007.
  13. W. Zhaofeng, H. Guyu, Y. Seyedi, and J. Fenglin, “A simple real-time handover management in the mobile satellite communication networks,” in 2015 17th Asia-Pacific Network Operations and Management Symposium (APNOMS), 2015, pp. 175–179.
  14. E. Papapetrou, S. Karapantazis, G. Dimitriadis, and F.-N. Pavlidou, “Satellite handover techniques for leo networks,” International Journal of Satellite Communications and Networking, vol. 22, no. 2, pp. 231–245, 2004.
  15. X. Cao, Y. Li, X. Xiong, and J. Wang, “Dynamic routings in satellite networks: An overview,” Sensors, vol. 22, no. 12, p. 4552, 2022.
  16. C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, pp. 279–292, 1992.
  17. H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1, Mar. 2016.
  18. G. Xu, Y. Zhao, Y. Ran, R. Zhao, and J. Luo, “Spatial location aided fully-distributed dynamic routing for large-scale leo satellite networks,” IEEE Communications Letters, vol. 26, no. 12, pp. 3034–3038, 2022.
  19. H. Zhang, H. Tang, Y. Hu, X. Wei, C. Wu, W. Ding, and X.-P. Zhang, “Heterogeneous mean-field multi-agent reinforcement learning for communication routing selection in sagi-net,” in 2022 IEEE 96th Vehicular Technology Conference (VTC2022-Fall), 2022, pp. 1–5.
  20. D. Chen, Q. Qi, Z. Zhuang, J. Wang, J. Liao, and Z. Han, “Mean field deep reinforcement learning for fair and efficient uav control,” IEEE Internet of Things Journal, vol. 8, no. 2, pp. 813–828, 2021.
  21. Q. Chen, G. Giambene, L. Yang, C. Fan, and X. Chen, “Analysis of inter-satellite link paths for leo mega-constellation networks,” IEEE Transactions on Vehicular Technology, vol. 70, no. 3, pp. 2743–2755, 2021.
  22. A. Donner, M. Berioli, and M. Werner, “Mpls-based satellite constellation networks,” IEEE Journal on Selected Areas in Communications, vol. 22, no. 3, pp. 438–448, 2004.
  23. C. Zhou, W. Wu, H. He, P. Yang, F. Lyu, N. Cheng, and X. Shen, “Deep reinforcement learning for delay-oriented iot task scheduling in sagin,” IEEE Transactions on Wireless Communications, vol. 20, no. 2, pp. 911–925, 2021.
  24. J.-H. Lee, J. Park, M. Bennis, and Y.-C. Ko, “Integrating leo satellites and multi-uav reinforcement learning for hybrid fso/rf non-terrestrial networks,” IEEE Transactions on Vehicular Technology, vol. 72, no. 3, pp. 3647–3662, 2023.
  25. Q. Yang, D. I. Laurenson, and J. A. Barria, “On the use of leo satellite constellation for active network management in power distribution networks,” IEEE Transactions on Smart Grid, vol. 3, no. 3, pp. 1371–1381, 2012.
  26. M. Guelman, A. Kogan, A. Kazarian, A. Livne, M. Orenstein, and H. Michalik, “Acquisition and pointing control for inter-satellite laser communications,” IEEE Transactions on Aerospace and Electronic Systems, vol. 40, no. 4, pp. 1239–1248, 2004.
  27. J. Mulholland and S. Cadogan, “Intersatellite laser crosslinks,” IEEE Transactions on Aerospace and Electronic Systems, vol. 32, no. 3, pp. 1011–1020, 1996.
  28. Y. Zeng and R. Zhang, “Energy-efficient uav communication with trajectory optimization,” IEEE Transactions on Wireless Communications, vol. 16, no. 6, pp. 3747–3760, 2017.
  29. Z. Qu, G. Zhang, H. Cao, and J. Xie, “Leo satellite constellation for internet of things,” IEEE Access, vol. 5, pp. 18 391–18 401, 2017.
  30. Z. Song, Y. Hao, Y. Liu, and X. Sun, “Energy-efficient multiaccess edge computing for terrestrial-satellite internet of things,” IEEE Internet of Things Journal, vol. 8, no. 18, pp. 14 202–14 218, 2021.
  31. N. Saeed, A. Elzanaty, H. Almorad, H. Dahrouj, T. Y. Al-Naffouri, and M.-S. Alouini, “Cubesat communications: Recent advances and future challenges,” IEEE Communications Surveys Tutorials, vol. 22, no. 3, pp. 1839–1862, 2020.
  32. S. Fu, J. Gao, and L. Zhao, “Integrated resource management for terrestrial-satellite systems,” IEEE Transactions on Vehicular Technology, vol. 69, no. 3, pp. 3256–3266, 2020.
  33. I.-R. and P.1814, “Prediction methods required for the design of terrestrial free-space optical links,” International Telecommunication Union, p. 1–12, 2007.
  34. A. Lapidoth, S. M. Moser, and M. A. Wigger, “On the capacity of free-space optical intensity channels,” IEEE Transactions on Information Theory, vol. 55, no. 10, pp. 4449–4461, 2009.
  35. A. Elhakeem, S. Bohm, M. Hachicha, T. Le-Ngoc, and H. Mouftah, “Analysis of a new multiaccess/switching technique for multibeam satellites in a prioritized isdn environment,” IEEE Journal on Selected Areas in Communications, vol. 10, no. 2, pp. 378–390, 1992.
  36. C. Lin, “Admission control in time-slotted multihop mobile networks,” IEEE Journal on Selected Areas in Communications, vol. 19, no. 10, pp. 1974–1983, 2001.
  37. R. Caceres and L. Iftode, “Improving the performance of reliable transport protocols in mobile computing environments,” IEEE Journal on Selected Areas in Communications, vol. 13, no. 5, pp. 850–857, 1995.
  38. C. A. Frans A Oliehoek, “A concise introduction to decentralized pomdps,” Springer, vol. 1, no. 10, 2016.
  39. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” PMLR, p. 1889–1897, 2015.
  40. L. Zhang, L. Shen, L. Yang, S. Chen, X. Wang, B. Yuan, and D. Tao, “Penalized proximal policy optimization for safe reinforcement learning,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI, 2022, pp. 3744–3750.
  41. J. Songsiri, J. Dahl, and L. Vandenberghe, “Convex optimization in signal processing and communications: Graphical models of autoregressive processes,” 2009.
  42. S. Gu, J. G. Kuba, M. Wen, R. Chen, Z. Wang, Z. Tian, J. Wang, A. Knoll, and Y. Yang, “Multi-agent constrained policy optimisation,” ArXiv, vol. abs/2110.02793, 2021.
  43. J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” In International Conference on Machine Learning, PMLR, pp. 22–31, 2017.
  44. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” ArXiv, vol. abs/1707.06347, 2017.
  45. J. G. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang, “Trust region policy optimisation in multi-agent reinforcement learning,” ArXiv, vol. abs/2109.11251, 2021.
  46. T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 7234–7284, 2020.
  47. J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018.
  48. A. M, “Spacex non-geostationary satellite system: Attachment a technical information to supplement schedules,” US Fed. Commun. Comm., Washington, DC, USA, Rep. SAT-OA-20161115-00118, 2016.
  49. S. Gu, J. Grudzien Kuba, Y. Chen, Y. Du, L. Yang, A. Knoll, and Y. Yang, “Safe multi-agent reinforcement learning for multi-robot control,” Artificial Intelligence, vol. 319, p. 103905, 2023.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now