Defining Problem from Solutions: Inverse Reinforcement Learning (IRL) and Its Applications for Next-Generation Networking (2404.01583v1)
Abstract: Performance optimization is a critical concern in networking, on which Deep Reinforcement Learning (DRL) has achieved great success. Nonetheless, DRL training relies on precisely defined reward functions, which formulate the optimization objective and indicate the positive/negative progress towards the optimal. With the ever-increasing environmental complexity and human participation in Next-Generation Networking (NGN), defining appropriate reward functions become challenging. In this article, we explore the applications of Inverse Reinforcement Learning (IRL) in NGN. Particularly, if DRL aims to find optimal solutions to the problem, IRL finds a problem from the optimal solutions, where the optimal solutions are collected from experts, and the problem is defined by reward inference. Specifically, we first formally introduce the IRL technique, including its fundamentals, workflow, and difference from DRL. Afterward, we present the motivations of IRL applications in NGN and survey existing studies. Furthermore, to demonstrate the process of applying IRL in NGN, we perform a case study about human-centric prompt engineering in Generative AI-enabled networks. We demonstrate the effectiveness of using both DRL and IRL techniques and prove the superiority of IRL.
- H. Du, R. Zhang, Y. Liu, J. Wang, Y. Lin, Z. Li, D. Niyato, J. Kang, Z. Xiong, S. Cui, B. Ai, H. Zhou, and D. I. Kim, “Beyond deep reinforcement learning: A tutorial on generative diffusion models in network optimization,” ArXiv preprint: ArXiv:2308.05384, 2023.
- G. Wang, P. Cheng, Z. Chen, W. Xiang, B. Vucetic, and Y. Li, “Inverse reinforcement learning with graph neural networks for iot resource allocation,” in ICASSP, 2023, pp. 1–5.
- J. Parras, A. Almodóvar, P. A. Apellániz, and S. Zazo, “Inverse reinforcement learning: A new framework to mitigate an intelligent backoff attack,” IEEE Internet of Things Journal, vol. 9, no. 24, pp. 24 790–24 799, 2022.
- S. Arora and P. Doshi, “A survey of inverse reinforcement learning: Challenges, methods and progress,” Artificial Intelligence, vol. 297, pp. 103 500–103 548, 2021.
- L. Snow, V. Krishnamurthy, and B. M. Sadler, “Identifying coordination in a cognitive radar network - a multi-objective inverse reinforcement learning approach,” in ICASSP, 2023, pp. 1–5.
- R. Zhang, K. Xiong, Y. Lu, P. Fan, D. W. K. Ng, and K. B. Letaief, “Energy efficiency maximization in ris-assisted swipt networks with rsma: A ppo-based approach,” IEEE Journal on Selected Areas in Communications, vol. 41, no. 5, pp. 1413–1430, 2023.
- V. S. Vaishnavi, Y. M. Roopa, and P. L. Srinivasa Murthy, “A survey on next generation networks,” in ICCNCT, 2020, pp. 162–171.
- G. Wang, P. Cheng, Z. Chen, B. Vucetic, and Y. Li, “Inverse reinforcement learning with graph neural networks for full-dimensional task offloading in edge computing,” IEEE Transactions on Mobile Computing, pp. 1–18, 2023.
- J. Li, H. Wu, Q. He, Y. Zhao, and X. Wang, “Dynamic qos prediction with intelligent route estimation via inverse reinforcement learning,” IEEE Transactions on Services Computing, pp. 1–18, 2023.
- A. Konar, D. Wu, Y. T. Xu, S. Jang, S. Liu, and G. Dudek, “Communication load balancing via efficient inverse reinforcement learning,” in ICC, 2023, pp. 472–478.
- A. Shamsoshoara, F. Lotfi, S. Mousavi, F. Afghah, and I. Guvenc, “Joint path planning and power allocation of a cellular-connected uav using apprenticeship learning via deep inverse reinforcement learning,” ArXiv preprint: ArXiv:2306.10071, 2023.
- R. Zhang, K. Xiong, X. Tian, Y. Lu, P. Fan, and K. B. Letaief, “Inverse reinforcement learning meets power allocation in multi-user cellular networks,” in INFOCOM WKSHPS, 2022, pp. 1–2.
- X. Tian, K. Xiong, R. Zhang, P. Fan, D. Niyato, and K. B. Letaief, “Sum rate maximization in multi-cell multi-user networks: An inverse reinforcement learning-based approach,” IEEE Wireless Communications Letters, vol. 13, no. 1, pp. 4–8, 2024.
- X. Tian, B. Gao, M. Liu, K. Xiong, P. Fan, and K. Ben Letaief, “Irl-pm: An inverse reinforcement learning-based power minimization in multi-user miso networks,” in ICCCS, 2023, pp. 332–336.
- Y. Liu, H. Du, D. Niyato, J. Kang, S. Cui, X. Shen, and P. Zhang, “Optimizing mobile-edge ai-generated everything (aigx) services by prompt engineering: Fundamental, framework, and case study,” IEEE Network, pp. 1–1, 2024.