Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Optimization Theory Based Deep Reinforcement Learning for Resource Allocation in Ultra-Reliable Wireless Networked Control Systems (2311.16895v2)

Published 28 Nov 2023 in eess.SY, cs.AI, cs.IT, cs.SY, and math.IT

Abstract: The design of Wireless Networked Control System (WNCS) requires addressing critical interactions between control and communication systems with minimal complexity and communication overhead while providing ultra-high reliability. This paper introduces a novel optimization theory based deep reinforcement learning (DRL) framework for the joint design of controller and communication systems. The objective of minimum power consumption is targeted while satisfying the schedulability and rate constraints of the communication system in the finite blocklength regime and stability constraint of the control system. Decision variables include the sampling period in the control system, and blocklength and packet error probability in the communication system. The proposed framework contains two stages: optimization theory and DRL. In the optimization theory stage, following the formulation of the joint optimization problem, optimality conditions are derived to find the mathematical relations between the optimal values of the decision variables. These relations allow the decomposition of the problem into multiple building blocks. In the DRL stage, the blocks that are simplified but not tractable are replaced by DRL. Via extensive simulations, the proposed optimization theory based DRL approach is demonstrated to outperform the optimization theory and pure DRL based approaches, with close to optimal performance and much lower complexity.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. H. Castro, N. Pinto, F. Pereira, L. Ferreira, P. Ávila, J. Bastos, G. D. Putnik, and M. Cruz-Cunha, “Cyber-physical systems using open design: an approach towards an open science lab for manufacturing,” Procedia Computer Science, vol. 196, pp. 381–388, 2022.
  2. O. Bello and S. Zeadally, “Intelligent device-to-device communication in the internet of things,” IEEE Systems Journal, vol. 10, no. 3, pp. 1172–1182, 2014.
  3. N. Promwongsa, A. Ebrahimzadeh, D. Naboulsi, S. Kianpisheh, F. Belqasmi, R. Glitho, N. Crespi, and O. Alfandi, “A comprehensive survey of the tactile internet: State-of-the-art and research directions,” IEEE Communications Surveys & Tutorials, vol. 23, no. 1, pp. 472–523, 2020.
  4. J. P. Hespanha, P. Naghshtabrizi, and Y. Xu, “A survey of recent results in networked control systems,” Proceedings of the IEEE, vol. 95, no. 1, pp. 138–162, 2007.
  5. P. Park, S. Coleri Ergen, C. Fischione, C. Lu, and K. H. Johansson, “Wireless network design for control systems: A survey,” IEEE Communications Surveys & Tutorials, vol. 20, no. 2, pp. 978–1013, 2018.
  6. A. W. Al-Dabbagh and T. Chen, “Design considerations for wireless networked control systems,” IEEE Transactions on Industrial Electronics, vol. 63, no. 9, pp. 5547–5557, 2016.
  7. H. Kagermann, W. Wahlster, , and J. Helbig, “Recommendations for implementing the strategic initiative industrie 4.0,” 2013.
  8. “Technical characteristics and spectrum requirements of wireless avionics intra-communications systems to support their safe operation.” https://www.itu.int/pub/R-REP-M.2283. Accessed: 2023-01-02.
  9. V. S. Hingmire, “Zigbee crosses the chasm: A market dynamics report on ieee 802.15.4 and zigbee, on world, 2010.” PhD Dissertation, 18 December 2015.
  10. I. S. of Automation, Wireless Systems for Industrial Automation: Process Control and Related Applications : ISA-100.11a-2009. ISA, 2009.
  11. “Driving digital transformation in process automation.” https://www.fieldcommgroup.org/. Accessed: 2023-01-01.
  12. “Industry iot consortium.” https://www.iiconsortium.org/about-us/.
  13. J. Montalban, E. Iradier, P. Angueira, O. Seijo, and I. Val, “Noma-based 802.11 n for industrial automation,” IEEE Access, vol. 8, pp. 168546–168557, 2020.
  14. F. Dobslaw, T. Zhang, and M. Gidlund, “End-to-end reliability-aware scheduling for wireless sensor networks,” IEEE Transactions on Industrial Informatics, vol. 12, no. 2, pp. 758–767, 2014.
  15. Y. Wu, G. Buttazzo, E. Bini, and A. Cervin, “Parameter selection for real-time controllers in resource-constrained systems,” IEEE Transactions on Industrial Informatics, vol. 6, no. 4, pp. 610–620, 2010.
  16. N. Pereira, B. Andersson, and E. Tovar, “Widom: A dominance protocol for wireless medium access,” IEEE Transactions on Industrial Informatics, vol. 3, no. 2, pp. 120–130, 2007.
  17. Y. Sadi, S. C. Ergen, and P. Park, “Minimum energy data transmission for wireless networked control systems,” IEEE Transactions on Wireless Communications, vol. 13, no. 4, pp. 2163–2175, 2014.
  18. Y. Sadi and S. C. Ergen, “Joint optimization of wireless network energy consumption and control system performance in wireless networked control systems,” IEEE Transactions on Wireless Communications, vol. 16, no. 4, pp. 2235–2248, 2017.
  19. Z. Zhao, W. Liu, D. E. Quevedo, Y. Li, and B. Vucetic, “Deep learning for wireless networked systems: A joint estimation-control-scheduling approach,” IEEE Internet of Things Journal, pp. 1–1, 2023.
  20. P. M. de Sant Ana, N. Marchenko, P. Popovski, and B. Soret, “Age of loop for wireless networked control systems optimization,” in 2021 IEEE 32nd Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), pp. 1–7, IEEE, 2021.
  21. T. Farjam and T. Charalambous, “Effect of computational power of sensors on event-triggered control mechanisms over a shared contention-based network,” arXiv preprint arXiv:2104.03076, 2021.
  22. H. Ren, C. Pan, Y. Deng, M. Elkashlan, and A. Nallanathan, “Joint power and blocklength optimization for urllc in a factory automation scenario,” IEEE Transactions on Wireless Communications, vol. 19, no. 3, pp. 1786–1801, 2019.
  23. S. Subchan, Z. Zuhair, T. Asfihani, D. Adzkiya, and S. Kim, “Energy optimization on wireless-networked control systems (w-ncss) using linear quadratic gaussian (lqg),” International Journal of Control, Automation and Systems, vol. 19, no. 12, pp. 3853–3861, 2021.
  24. V. Lima, M. Eisen, K. Gatsis, and A. Ribeiro, “Model-free design of control systems over wireless fading channels,” Signal Processing, vol. 197, p. 108540, 2022.
  25. D. Baumann, J.-J. Zhu, G. Martius, and S. Trimpe, “Deep reinforcement learning for event-triggered control,” in 2018 IEEE Conference on Decision and Control (CDC), pp. 943–950, IEEE, 2018.
  26. O. N. Yilmaz, Y.-P. E. Wang, N. A. Johansson, N. Brahmi, S. A. Ashraf, and J. Sachs, “Analysis of ultra-reliable and low-latency 5g communication for a factory automation use case,” in 2015 IEEE international conference on communication workshop (ICCW), pp. 1190–1195, IEEE, 2015.
  27. T. Arampatzis, J. Lygeros, and S. Manesis, “A survey of applications of wireless sensors and wireless sensor networks,” in Proceedings of the 2005 IEEE International Symposium on, Mediterrean Conference on Control and Automation Intelligent Control, 2005., pp. 719–724, IEEE, 2005.
  28. “Ansi/isa-100.11a-2011 wireless systems for industrial automation: Process control and related applications.” https://www.isa.org/products/ansi-isa-100-11a-2011-wireless-systems-for-industr. Accessed: 2023-01-21.
  29. S. Cui, A. J. Goldsmith, and A. Bahai, “Energy-constrained modulation optimization,” IEEE transactions on wireless communications, vol. 4, no. 5, pp. 2349–2360, 2005.
  30. P. Park, H. Khadilkar, H. Balakrishnan, and C. J. Tomlin, “High confidence networked control for next generation air transportation systems,” IEEE Transactions on Automatic Control, vol. 59, no. 12, pp. 3357–3372, 2014.
  31. G. Karagiannis, O. Altintas, E. Ekici, G. Heijenk, B. Jarupan, K. Lin, and T. Weil, “Vehicular networking: A survey and tutorial on requirements, architectures, challenges, standards and solutions,” IEEE communications surveys & tutorials, vol. 13, no. 4, pp. 584–616, 2011.
  32. C. She, C. Yang, and T. Q. S. Quek, “Joint uplink and downlink resource configuration for ultra-reliable and low-latency communications,” IEEE Transactions on Communications, vol. 66, no. 5, pp. 2266–2280, 2018.
  33. Y. Polyanskiy, H. V. Poor, and S. Verdú, “Channel coding rate in the finite blocklength regime,” IEEE Transactions on Information Theory, vol. 56, no. 5, pp. 2307–2359, 2010.
  34. G. Durisi, T. Koch, and P. Popovski, “Toward massive, ultrareliable, and low-latency wireless communication with short packets,” Proceedings of the IEEE, vol. 104, no. 9, pp. 1711–1726, 2016.
  35. M. Dertouzos, “Control robotics: The procedural control of physical processes,” IFIP Congress 1974, 1974.
  36. F. Zhang and A. Burns, “Schedulability analysis for real-time systems with edf scheduling,” IEEE Transactions Comput, vol. 58, no. 9, p. 250–1258, 2009.
  37. F. Eisenbrand and T. Rothvoß, “Edf-schedulability of synchronous periodic task systems is conp-hard,” in Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms, pp. 1029–1034, SIAM, 2010.
  38. Cambridge University Press, 2004.
  39. R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
  40. H. Hasselt, “Double q-learning,” Advances in neural information processing systems, vol. 23, 2010.
  41. M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, 2018.
  42. A. Tavakoli, F. Pardo, and P. Kormushev, “Action branching architectures for deep reinforcement learning,” in Proceedings of the aaai conference on artificial intelligence, vol. 32, 2018.
  43. E. U. T. R. Access, “Further advancements for e-utra physical layer aspects (release 9),” European Telecommunications Standards Institute, 2010.
  44. L. Liang, J. Kim, S. C. Jha, K. Sivanesan, and G. Y. Li, “Spectrum and power allocation for vehicular communications with delayed csi feedback,” IEEE Wireless Communications Letters, vol. 6, no. 4, pp. 458–461, 2017.
  45. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now