Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Bayesian Framework of Deep Reinforcement Learning for Joint O-RAN/MEC Orchestration (2312.16142v1)

Published 26 Dec 2023 in cs.NI and cs.LG

Abstract: Multi-access Edge Computing (MEC) can be implemented together with Open Radio Access Network (O-RAN) over commodity platforms to offer low-cost deployment and bring the services closer to end-users. In this paper, a joint O-RAN/MEC orchestration using a Bayesian deep reinforcement learning (RL)-based framework is proposed that jointly controls the O-RAN functional splits, the allocated resources and hosting locations of the O-RAN/MEC services across geo-distributed platforms, and the routing for each O-RAN/MEC data flow. The goal is to minimize the long-term overall network operation cost and maximize the MEC performance criterion while adapting possibly time-varying O-RAN/MEC demands and resource availability. This orchestration problem is formulated as Markov decision process (MDP). However, the system consists of multiple BSs that share the same resources and serve heterogeneous demands, where their parameters have non-trivial relations. Consequently, finding the exact model of the underlying system is impractical, and the formulated MDP renders in a large state space with multi-dimensional discrete action. To address such modeling and dimensionality issues, a novel model-free RL agent is proposed for our solution framework. The agent is built from Double Deep Q-network (DDQN) that tackles the large state space and is then incorporated with action branching, an action decomposition method that effectively addresses the multi-dimensional discrete action with linear increase complexity. Further, an efficient exploration-exploitation strategy under a Bayesian framework using Thomson sampling is proposed to improve the learning performance and expedite its convergence. Trace-driven simulations are performed using an O-RAN-compliant model. The results show that our approach is data-efficient (i.e., converges faster) and increases the returned reward by 32\% than its non-Bayesian version.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. “5G immersive service opportunities with Edge Cloud and Cloud RAN (White Paper),” Nokia, Tech. Rep., 2019.
  2. “Open and Virtualized - The Future Radio Access Network,” NEC, Tech. Rep., 01 2020.
  3. L. Bonati et al., “Open, Programmable, and Virtualized 5G Networks: State-of-the-Art and the Road Ahead,” Computer Networks, vol. 182, p. 107516, 2020.
  4. “O-RAN-WG1-O-RAN Architecture Description v01.00.00,” O-RAN Alliance, Technical Specification (TS), 02 2020.
  5. ETSI, “Cloud RAN and MEC:A Perfect Pairing ,” European Telecommunications Standards Institute (ETSI), White Paper First edition, 2018.
  6. F. W. Murti, J. A. Ayala-Romero, A. Garcia-Saavedra, X. Costa-Pérez, and G. Iosifidis, “An Optimal Deployment Framework for Multi-Cloud Virtualized Radio Access Networks,” IEEE Transactions on Wireless Communications, vol. 20, no. 4, pp. 2251–2265, 2021.
  7. N. Makris, V. Passas, T. Korakis, and L. Tassiulas, “Employing mec in the cloud-ran: An experimental analysis,” in Proceedings of the 2018 on Technologies for the Wireless Edge Workshop, ser. WirelessEdge ’18.   Association for Computing Machinery, 2018, p. 15–19.
  8. V. Passas, N. Makris, C. Nanis, and T. Korakis, “Mec service placement over the fronthaul of 5g cloud-rans,” in 2019 IEEE International Symposium on Local and Metropolitan Area Networks (LANMAN), 2019, pp. 1–2.
  9. A. Garcia-Saavedra et al., “Joint Optimization of Edge Computing Architectures and Radio Access Networks,” IEEE Journal on Selected Areas in Communications, vol. 36, no. 11, 2018.
  10. L. d. S. Fraga, G. M. Almeida, S. Correa, C. Both, L. Pinto, and K. Cardoso, “Efficient allocation of disaggregated ran functions and multi-access edge computing services,” in GLOBECOM 2022 - 2022 IEEE Global Communications Conference, 2022, pp. 191–196.
  11. J. A. Ayala-Romero et al., “vrAIn: Deep Learning based Orchestration for Computing and Radio Resources in vRANs,” IEEE Transactions on Mobile Computing, pp. 1–1, 2020.
  12. J. A. Ayala-Romero, A. Garcia-Saavedra, X. Costa-Pérez, and G. Iosifidis, “EdgeBOL: A Bayesian Learning Approach for the Joint Orchestration of vRANs and Mobile Edge AI,” IEEE/ACM Transactions on Networking, pp. 1–0, 2023.
  13. H. v. Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” ser. AAAI’16.   AAAI Press, 2016, p. 2094–2100.
  14. A. Tavakoli, F. Pardo, and P. Kormushev, “Action Branching Architectures for Deep Reinforcement Learning,” in AAAI Conference on Artificial Intelligence, 2018, pp. 4131–4138.
  15. K. Azizzadenesheli, E. Brunskill, and A. Anandkumar, “Efficient exploration through bayesian deep q-networks,” in 2018 Information Theory and Applications Workshop (ITA), 2018, pp. 1–9.
  16. C. Riquelme, G. Tucker, and J. Snoek, “Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling,” International Conference on Machine Learning (ICML), 2018.
  17. A. M. Alba, S. Janardhanan, and W. Kellerer, “Enabling Dynamically Centralized RAN Architectures in 5G and Beyond,” IEEE Transactions on Network and Service Management, vol. 18, no. 3, pp. 3509–3526, 2021.
  18. A. M. Alba and W. Kellerer, “Dynamic Functional Split Adaptation in Next-Generation Radio Access Networks,” IEEE Transactions on Network and Service Management, 2022.
  19. D. Harutyunyan and R. Riggio, “Flex5G: Flexible Functional Split in 5G Networks,” IEEE Transactions on Network and Service Management, vol. 15, no. 3, pp. 961–975, 2018.
  20. D. Bega, A. Banchs, M. Gramaglia, X. Costa-Pérez, and P. Rost, “CARES: Computation-Aware Scheduling in Virtualized Radio Access Networks,” IEEE Transactions on Wireless Communications, vol. 17, no. 12, pp. 7993–8006, 2018.
  21. S. Matoussi et al., “Deep Learning based User Slice Allocation in 5G Radio Access Networks,” in 2020 IEEE 45th Conference on Local Computer Networks (LCN), 2020, pp. 286–296.
  22. J. A. Ayala-Romero, A. Garcia-Saavedra, X. Costa-Perez, and G. Iosifidis, “Orchestrating energy-efficient vrans: Bayesian learning and experimental results,” IEEE Transactions on Mobile Computing, pp. 1–1, 2021.
  23. T. Pamuklu, M. Erol-Kantarci, and C. Ersoy, “Reinforcement Learning Based Dynamic Function Splitting in Disaggregated Green Open RANs,” in ICC 2021 - IEEE International Conference on Communications, 2021, pp. 1–6.
  24. M. Polese, L. Bonati, S. D’Oro, S. Basagni, and T. Melodia, “ColO-RAN: Developing Machine Learning-based xApps for Open RAN Closed-loop Control on Programmable Experimental Platforms,” IEEE Transactions on Mobile Computing, pp. 1–14, 2022.
  25. F. W. Murti, S. Ali, G. Iosifidis, and M. Latva-Aho, “Learning-Based Orchestration for Dynamic Functional Split and Resource Allocation in vRANs,” in 2022 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit), 2022, pp. 243–248.
  26. F. W. Murti, S. Ali, G. Iosifidis, and M. Latva-aho, “Deep reinforcement learning for orchestrating cost-aware reconfigurations of vrans,” IEEE Transactions on Network and Service Management, pp. 1–1, 2023.
  27. S. Kukliński, L. Tomaszewski, and R. Kołakowski, “On O-RAN, MEC, SON and Network Slicing integration,” in 2020 IEEE Globecom Workshops (GC Wkshps, 2020, pp. 1–6.
  28. B. Ojaghi, F. Adelantado, and C. Verikoukis, “So-ran: Dynamic ran slicing via joint functional splitting and mec placement,” IEEE Transactions on Vehicular Technology, vol. 72, no. 2, pp. 1925–1939, 2023.
  29. S. D’Oro, L. Bonati, F. Restuccia, M. Polese, M. Zorzi, and T. Melodia, “Sl-Edge: Network Slicing at the Edge,” in Proceedings of the Twenty-First International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, ser. Mobihoc ’20.   New York, NY, USA: Association for Computing Machinery, 2020, p. 1–10. [Online]. Available: https://doi.org/10.1145/3397166.3409133
  30. X. Foukas and B. Radunovic, “Concordia: Teaching the 5g vran to share compute,” in Proceedings of the 2021 ACM SIGCOMM 2021 Conference, ser. SIGCOMM ’21.   New York, NY, USA: Association for Computing Machinery, 2021, p. 580–596. [Online]. Available: https://doi.org/10.1145/3452296.3472894
  31. J. Chuai, Z. Chen, G. Liu, X. Guo, X. Wang, X. Liu, C. Zhu, and F. Shen, “A Collaborative Learning Based Approach for Parameter Configuration of Cellular Networks,” in IEEE INFOCOM 2019 - IEEE Conference on Computer Communications, 2019, pp. 1396–1404.
  32. F. Wei, G. Feng, Y. Sun, Y. Wang, S. Qin, and Y.-C. Liang, “Network slice reconfiguration by exploiting deep reinforcement learning with large action space,” IEEE Transactions on Network and Service Management, vol. 17, no. 4, pp. 2197–2211, 2020.
  33. M. K. Abdel-Aziz, C. Perfecto, S. Samarakoon, M. Bennis, and W. Saad, “Vehicular cooperative perception through action branching and federated reinforcement learning,” IEEE Transactions on Communications, vol. 70, no. 2, pp. 891–903, 2022.
  34. L. Bonati, S. D’Oro, M. Polese, S. Basagni, and T. Melodia, “Intelligence and Learning in O-RAN for Data-Driven NextG Cellular Networks,” IEEE Communications Magazine, vol. 59, no. 10, pp. 21–27, 2021.
  35. 3GPP, “Architecture description (Release 16),” 3rd Generation Partnership Project (3GPP), Technical Specification Group Radio Access Network (NG-RAN) 38.401, 03 2020, version 16.1.0.
  36. D. Bega, M. Gramaglia, M. Fiore, A. Banchs, and X. Costa-Perez, “AZTEC: Anticipatory Capacity Allocation for Zero-Touch Network Slicing,” in IEEE INFOCOM 2020 - IEEE Conference on Computer Communications, 2020, pp. 794–803.
  37. V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, 2015.
  38. B. Xiang, J. Elias, F. Martignon, and E. Di Nitto, “A dataset for mobile edge computing network topologies,” Data in Brief, vol. 39, p. 107557, 2021.
  39. T. Italia, “Telecommunications - SMS, Call, Internet - MI,” 2015. [Online]. Available: https://doi.org/10.7910/DVN/EGZHFV
  40. B. Waxman, “Routing of multipoint connections,” IEEE Journal on Selected Areas in Communications, vol. 6, no. 9, pp. 1617–1622, 1988.
  41. I. Gomez-Miguelez et al., “SrsLTE: An Open-Source Platform for LTE Evolution and Experimentation,” in Proceedings of the Tenth ACM International Workshop on Wireless Network Testbeds, Experimental Evaluation, and Characterization, 2016, p. 25–32.
  42. F. Z. Morais et al., “PlaceRAN: optimal placement of virtualized network functions in Beyond 5G radio access networks,” IEEE Transactions on Mobile Computing, pp. 1–1, 2022.
  43. P. Rost, S. Talarico, and M. C. Valenti, “The complexity–rate tradeoff of centralized radio access networks,” IEEE Transactions on Wireless Communications, vol. 14, no. 11, pp. 6164–6176, 2015.
  44. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, 2015.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now