To RL or not to RL? An Algorithmic Cheat-Sheet for AI-Based Radio Resource Management (2405.19045v2)
Abstract: Several Radio Resource Management (RRM) use cases can be framed as sequential decision planning problems, where an agent (the base station, typically) makes decisions that influence the network utility and state. While Reinforcement Learning (RL) in its general form can address this scenario, it is known to be sample inefficient. Following the principle of Occam's razor, we argue that the choice of the solution technique for RRM should be guided by questions such as, "Is it a short or long-term planning problem?", "Is the underlying model known or does it need to be learned?", "Can we solve the problem analytically?" or "Is an expert-designed policy available?". A wide range of techniques exists to address these questions, including static and stochastic optimization, bandits, model predictive control (MPC) and, indeed, RL. We review some of these techniques that have already been successfully applied to RRM, and we believe that others, such as MPC, may present exciting research opportunities for the future.
- J. Hoydis, F. A. Aoudia, A. Valcarce, and H. Viswanathan, “Toward a 6G AI-Native Air Interface,” IEEE Communications Magazine, vol. 59, no. 5, pp. 76–81, 2021.
- G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester, “Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis,” Machine Learning, vol. 110, no. 9, pp. 2419–2468, 2021.
- M. J. Sobel, “Myopic solutions of Markov decision processes and stochastic games,” Operations Research, vol. 29, no. 5, pp. 995–1009, 1981.
- A. M. Elbir, K. V. Mishra, S. A. Vorobyov, and R. W. Heath, “Twenty-five years of advances in beamforming: From convex and nonconvex optimization to learning techniques,” IEEE Signal Processing Magazine, vol. 40, no. 4, pp. 118–131, 2023.
- L. Maggi, A. Valcarce, and J. Hoydis, “Bayesian optimization for radio resource management: Open loop power control,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 7, pp. 1858–1871, 2021.
- L. Maggi, A. R. Koblitz, Q. Zhu, and M. Andrews, “Tracking the Best Beam for a Mobile User via Bayesian Optimization,” in 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring). IEEE, 2023, pp. 1–7.
- J. A. Ayala-Romero, A. Garcia-Saavedra, and X. Costa-Perez, “Risk-Aware Continuous Control with Neural Contextual Bandits,” arXiv preprint arXiv:2312.09961, 2023.
- V. Saxena, H. Tullberg, and J. Jaldén, “Reinforcement learning for efficient and tuning-free link adaptation,” IEEE Transactions on Wireless Communications, vol. 21, no. 2, pp. 768–780, 2021.
- A. L. Stolyar, “Maximizing queueing network utility subject to stability: Greedy primal-dual algorithm,” Queueing Systems, vol. 50, pp. 401–457, 2005.
- M. J. Neely, “Energy optimal control for time-varying wireless networks,” IEEE transactions on Information Theory, vol. 52, no. 7, pp. 2915–2934, 2006.
- E. Altman, “Applications of Markov decision processes in communication networks,” in Handbook of Markov Decision Processes: Methods and Applications. Springer, 2002, pp. 489–536.
- S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643, 2020.
- L. Maggi, C. Mihailescu, Q. Cao, A. Tetich, S. Khan, S. Aaltonen, R. Koblitz, M. Holma, S. Macchi, M. E. Ruggieri et al., “Energy savings under performance constraints via carrier shutdown with Bayesian learning,” in 2023 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit). IEEE, 2023, pp. 1–6.
- P. Hu, L. Pan, Y. Chen, Z. Fang, and L. Huang, “Effective multi-user delay-constrained scheduling with deep recurrent reinforcement learning,” in Proceedings of the Twenty-Third International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, 2022, pp. 1–10.
- F. H. C. Neto, D. C. Araújo, M. P. Mota, T. F. Maciel, and A. L. de Almeida, “Uplink power control framework based on reinforcement learning for 5G networks,” IEEE Transactions on Vehicular Technology, vol. 70, no. 6, pp. 5734–5748, 2021.
- S. K. Dehkordi, M. Kobayashi, and G. Caire, “Adaptive beam tracking based on recurrent neural networks for mmwave channels,” in 2021 IEEE 22nd International Workshop on Signal Processing Advances in Wireless Communications (SPAWC). IEEE, 2021, pp. 1–5.