Papers
Topics
Authors
Recent
2000 character limit reached

SINR-Aware Deep Reinforcement Learning for Distributed Dynamic Channel Allocation in Cognitive Interference Networks

Published 17 Feb 2024 in eess.SP and cs.LG | (2402.17773v1)

Abstract: We consider the problem of dynamic channel allocation (DCA) in cognitive communication networks with the goal of maximizing a global signal-to-interference-plus-noise ratio (SINR) measure under a specified target quality of service (QoS)-SINR for each network. The shared bandwidth is partitioned into K channels with frequency separation. In contrast to the majority of existing studies that assume perfect orthogonality or a one- to-one user-channel allocation mapping, this paper focuses on real-world systems experiencing inter-carrier interference (ICI) and channel reuse by multiple large-scale networks. This realistic scenario significantly increases the problem dimension, rendering existing algorithms inefficient. We propose a novel multi-agent reinforcement learning (RL) framework for distributed DCA, named Channel Allocation RL To Overlapped Networks (CARLTON). The CARLTON framework is based on the Centralized Training with Decentralized Execution (CTDE) paradigm, utilizing the DeepMellow value-based RL algorithm. To ensure robust performance in the interference-laden environment we address, CARLTON employs a low-dimensional representation of observations, generating a QoS-type measure while maximizing a global SINR measure and ensuring the target QoS-SINR for each network. Our results demonstrate exceptional performance and robust generalization, showcasing superior efficiency compared to alternative state-of-the-art methods, while achieving a marginally diminished performance relative to a fully centralized approach.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. K. Liu and Q. Zhao, “Cooperative game in dynamic spectrum access with unknown model and imperfect sensing,” IEEE Transactions on Wireless Communications, vol. 11, no. 4, pp. 1596–1604, 2012.
  2. C. Tekin and M. Liu, “Online learning of rested and restless bandits,” IEEE Transactions on Information Theory, vol. 58, no. 8, pp. 5588–5611, 2012.
  3. H. Liu, K. Liu, and Q. Zhao, “Learning in a changing world: Restless multiarmed bandit with unknown dynamics,” IEEE Transactions on Information Theory, vol. 59, no. 3, pp. 1902–1916, 2013.
  4. N. Nayyar, D. Kalathil, and R. Jain, “On regret-optimal learning in decentralized multiplayer multiarmed bandits,” IEEE Transactions on Control of Network Systems, vol. 5, no. 1, pp. 597–606, 2016.
  5. O. Avner and S. Mannor, “Multi-user lax communications: a multi-armed bandit approach,” in IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications, pp. 1–9, IEEE, 2016.
  6. I. Bistritz and A. Leshem, “Distributed multi-player bandits-a game of thrones approach,” in Advances in Neural Information Processing Systems, pp. 7222–7232, 2018.
  7. T. Gafni and K. Cohen, “Learning in restless multi-armed bandits via adaptive arm sequencing rules,” IEEE Transactions on Automatic Control, 2020.
  8. W. Wang, A. Leshem, D. Niyato, and Z. Han, “Decentralized learning for channel allocation in iot networks over unlicensed bandwidth as a contextual multi-player multi-armed bandit game,” IEEE Transactions on Wireless Communications, vol. 21, no. 5, pp. 3162–3178, 2021.
  9. T. Gafni, M. Yemini, and K. Cohen, “Learning in restless bandits under exogenous global markov process,” IEEE Transactions on Signal Processing, vol. 70, pp. 5679–5693, 2022.
  10. A. Leshem, E. Zehavi, and Y. Yaffe, “Multichannel opportunistic carrier sensing for stable channel access control in cognitive radio systems,” IEEE Journal on Selected Areas in Communications, vol. 30, no. 1, pp. 82–95, 2012.
  11. T. Gafni and K. Cohen, “Distributed learning over markovian fading channels for stable spectrum access,” IEEE Access, vol. 10, pp. 46652–46669, 2022.
  12. Z. Han, Z. Ji, and K. R. Liu, “Fair multiuser channel allocation for OFDMA networks using Nash bargaining solutions and coalitions,” IEEE Transactions on Communications, vol. 53, no. 8, pp. 1366–1376, 2005.
  13. I. Menache and N. Shimkin, “Rate-based equilibria in collision channels with fading,” IEEE Journal on Selected Areas in Communications, vol. 26, no. 7, pp. 1070–1077, 2008.
  14. I. Menache and A. Ozdaglar, “Network games: Theory, models, and dynamics,” Synthesis Lectures on Communication Networks, vol. 4, no. 1, pp. 1–159, 2011.
  15. L. M. Law, J. Huang, and M. Liu, “Price of anarchy for congestion games in cognitive radio networks,” IEEE Transactions on Wireless Communications, vol. 11, no. 10, pp. 3778–3787, 2012.
  16. K. Cohen, A. Leshem, and E. Zehavi, “Game theoretic aspects of the multi-channel ALOHA protocol in cognitive radio networks,” IEEE Journal on Selected Areas in Communications, vol. 31, pp. 2276–2288, 2013.
  17. H. Wu, C. Zhu, R. J. La, X. Liu, and Y. Zhang, “Fasa: Accelerated S-ALOHA using access history for event-driven M2M communications,” IEEE/ACM Transactions on Networking (TON), vol. 21, no. 6, pp. 1904–1917, 2013.
  18. C. Singh, A. Kumar, and R. Sundaresan, “Combined base station association and power control in multichannel cellular networks,” IEEE/ACM Transactions on Networking, vol. 24, no. 2, pp. 1065–1080, 2016.
  19. K. Cohen and A. Leshem, “Distributed game-theoretic optimization and management of multichannel aloha networks,” IEEE/ACM Transactions on Networking, vol. 24, no. 3, pp. 1718–1731, 2016.
  20. K. Cohen, A. Nedić, and R. Srikant, “Distributed learning algorithms for spectrum sharing in spatial random access wireless networks,” IEEE Transactions on Automatic Control, vol. 62, no. 6, pp. 2854–2869, 2017.
  21. D. Malachi and K. Cohen, “Queue and channel-based aloha algorithm in multichannel wireless networks,” IEEE Wireless Communications Letters, vol. 9, no. 8, pp. 1309–1313, 2020.
  22. O. Naparstek and K. Cohen, “Deep multi-user reinforcement learning for dynamic spectrum access in multichannel wireless networks,” in GLOBECOM 2017-2017 IEEE Global Communications Conference, pp. 1–7, IEEE, 2017.
  23. O. Naparstek and K. Cohen, “Deep multi-user reinforcement learning for distributed dynamic spectrum access,” IEEE transactions on wireless communications, vol. 18, no. 1, pp. 310–323, 2018.
  24. H.-H. Chang, H. Song, Y. Yi, J. Zhang, H. He, and L. Liu, “Distributive dynamic spectrum access through deep reinforcement learning: A reservoir computing-based approach,” IEEE Internet of Things Journal, vol. 6, no. 2, pp. 1938–1948, 2018.
  25. P. Yang, L. Li, J. Yin, H. Zhang, W. Liang, W. Chen, and Z. Han, “Dynamic spectrum access in cognitive radio networks using deep reinforcement learning and evolutionary game,” in 2018 IEEE/CIC International Conference on Communications in China (ICCC), pp. 405–409, IEEE, 2018.
  26. W. Su, J. Lin, K. Chen, L. Xiao, and C. En, “Reinforcement learning-based adaptive modulation and coding for efficient underwater communications,” IEEE access, vol. 7, pp. 67539–67550, 2019.
  27. Y. Zhang, X. Wang, and Y. Xu, “Energy-efficient resource allocation in uplink noma systems with deep reinforcement learning,” in 2019 11th international conference on wireless communications and signal processing (WCSP), pp. 1–6, IEEE, 2019.
  28. D. Lee, Y. G. Sun, S. H. Kim, I. Sim, Y. M. Hwang, Y. Shin, D. I. Kim, and J. Y. Kim, “Dqn-based adaptive modulation scheme over wireless communication channels,” IEEE Communications Letters, vol. 24, no. 6, pp. 1289–1293, 2020.
  29. W. Ning, X. Huang, K. Yang, F. Wu, and S. Leng, “Reinforcement learning enabled cooperative spectrum sensing in cognitive radio networks,” Journal of Communications and Networks, vol. 22, no. 1, pp. 12–22, 2020.
  30. J. Xu, H. Lou, W. Zhang, and G. Sang, “An intelligent anti-jamming scheme for cognitive radio based on deep reinforcement learning,” IEEE Access, vol. 8, pp. 202563–202572, 2020.
  31. F. Obite, A. D. Usman, and E. Okafor, “An overview of deep reinforcement learning for spectrum sensing in cognitive radio networks,” Digital Signal Processing, vol. 113, p. 103014, 2021.
  32. X. Du, T. Wang, Q. Feng, C. Ye, T. Tao, L. Wang, Y. Shi, and M. Chen, “Multi-agent reinforcement learning for dynamic resource management in 6g in-x subnetworks,” IEEE Transactions on Wireless Communications, vol. 22, no. 3, pp. 1900–1914, 2022.
  33. R. Adeogun and G. Berardinelli, “Multi-agent dynamic resource allocation in 6g in-x subnetworks with limited sensing information,” Sensors, vol. 22, no. 13, p. 5062, 2022.
  34. Y. Bokobza, R. Dabora, and K. Cohen, “Deep reinforcement learning for simultaneous sensing and channel access in cognitive networks,” IEEE Transactions on Wireless Communications, 2023.
  35. R. Adeogun and G. Berardinelli, “Distributed channel allocation for mobile 6g subnetworks via multi-agent deep q-learning,” in 2023 IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6, IEEE, 2023.
  36. R. Paul, K. Cohen, and G. Kedar, “Multi-flow transmission in wireless interference networks: A convergent graph learning approach,” IEEE Transactions on Wireless Communications, pp. 1–1, 2023.
  37. S. Wang, H. Liu, P. H. Gomes, and B. Krishnamachari, “Deep reinforcement learning for dynamic multichannel access in wireless networks,” IEEE Transactions on Cognitive Communications and Networking, vol. 4, no. 2, pp. 257–265, 2018.
  38. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013.
  39. M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, 2018.
  40. J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al., “Mastering atari, go, chess and shogi by planning with a learned model,” Nature, vol. 588, no. 7839, pp. 604–609, 2020.
  41. O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al., “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” Nature, vol. 575, no. 7782, pp. 350–354, 2019.
  42. Y. Li, “Deep reinforcement learning: An overview,” arXiv preprint arXiv:1701.07274, 2017.
  43. Y. Xing, R. Chandramouli, S. Mangold, and N. SS, “Dynamic spectrum access in open spectrum wireless networks,” IEEE journal on selected areas in communications, vol. 24, no. 3, pp. 626–637, 2006.
  44. H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 30, 2016.
  45. S. Kim, K. Asadi, M. Littman, and G. Konidaris, “Deepmellow: removing the need for a target network in deep q-learning,” in Proceedings of the twenty eighth international joint conference on artificial intelligence, 2019.
  46. C. Huang, V. J. Sorger, M. Miscuglio, M. Al-Qadasi, A. Mukherjee, L. Lampe, M. Nichols, A. N. Tait, T. Ferreira de Lima, B. A. Marquez, et al., “Prospects and applications of photonic neural networks,” Advances in Physics: X, vol. 7, no. 1, p. 1981155, 2022.
  47. J. J. Egli, “Radio propagation above 40 mc over irregular terrain,” Proceedings of the IRE, vol. 45, no. 10, pp. 1383–1391, 1957.
  48. A. Goldsmith, Wireless communications. Cambridge university press, 2005.
  49. P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire, “The nonstochastic multiarmed bandit problem,” SIAM journal on computing, vol. 32, no. 1, pp. 48–77, 2002.
  50. K. Asadi and M. L. Littman, “An alternative softmax operator for reinforcement learning,” in International Conference on Machine Learning, pp. 243–252, PMLR, 2017.
  51. X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249–256, JMLR Workshop and Conference Proceedings, 2010.
  52. O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küttler, J. Agapiou, J. Schrittwieser, et al., “Starcraft ii: A new challenge for reinforcement learning,” arXiv preprint arXiv:1708.04782, 2017.
  53. C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al., “Dota 2 with large scale deep reinforcement learning,” arXiv preprint arXiv:1912.06680, 2019.
  54. D. Ye, Z. Liu, M. Sun, B. Shi, P. Zhao, H. Wu, H. Yu, S. Yang, X. Wu, Q. Guo, et al., “Mastering complex control in moba games with deep reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 6672–6679, 2020.
  55. Y. Hou, X. Liang, J. Zhang, Q. Yang, A. Yang, and N. Wang, “Exploring the use of invalid action masking in reinforcement learning: A comparative study of on-policy and off-policy algorithms in real-time strategy games,” Applied Sciences, vol. 13, no. 14, p. 8283, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up