Papers
Topics
Authors
Recent
2000 character limit reached

Imitation Game: A Model-based and Imitation Learning Deep Reinforcement Learning Hybrid (2404.01794v1)

Published 2 Apr 2024 in cs.AI

Abstract: Autonomous and learning systems based on Deep Reinforcement Learning have firmly established themselves as a foundation for approaches to creating resilient and efficient Cyber-Physical Energy Systems. However, most current approaches suffer from two distinct problems: Modern model-free algorithms such as Soft Actor Critic need a high number of samples to learn a meaningful policy, as well as a fallback to ward against concept drifts (e. g., catastrophic forgetting). In this paper, we present the work in progress towards a hybrid agent architecture that combines model-based Deep Reinforcement Learning with imitation learning to overcome both problems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. “Playing Atari with Deep Reinforcement Learning” In arXiv preprint arXiv:1312.5602, 2013
  2. “Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model”, 2019, pp. 1–21 arXiv: http://arxiv.org/abs/1911.08265
  3. “Grandmaster level in StarCraft II using multi-agent reinforcement learning” Number: 7782 Publisher: Nature Publishing Group In Nature 575.7782, 2019, pp. 350–354 DOI: 10.1038/s41586-019-1724-z
  4. Eric MSP Veith “Universal Smart Grid Agent for Distributed Power Generation Management” Logos Verlag Berlin GmbH, 2017
  5. “Autonomous Voltage Control for Grid Operation Using Deep Reinforcement Learning” In IEEE Power and Energy Society General Meeting 2019-Augus, 2019 DOI: 10.1109/PESGM40551.2019.8973924
  6. Brandon L Thayer and Thomas J Overbye “Deep reinforcement learning for electric transmission voltage control” In 2020 IEEE Electric Power and Energy Conference (EPEC), 2020, pp. 1–8 IEEE
  7. Zirui Wu, Changcheng Li and Ling He “A novel reinforcement learning method for the plan of generator start-up after blackout” In Electric Power Systems Research 228, 2024, pp. 110068 DOI: 10.1016/j.epsr.2023.110068
  8. “Deep Reinforcement Learning for Anomaly Detection: A Systematic Review” In IEEE Access 10, 2022, pp. 124017–124035 DOI: 10.1109/ACCESS.2022.3224023
  9. Eric Veith, Arlena Wellßow and Mathias Uslar “Learning new attack vectors from misuse cases with deep reinforcement learning” In Frontiers in Energy Research, 2023
  10. “ANALYSE–Learning to Attack Cyber-Physical Energy Systems With Intelligent Agents” In SoftwareX, 2023 DOI: 10.1016/j.softx.2023.101484
  11. “Reinforcement learning for selective key applications in power systems: Recent advances and future challenges” In IEEE Transactions on Smart Grid 13.4 IEEE, 2022, pp. 2935–2958
  12. Michael McCloskey and Neal J. Cohen “Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem” In Psychology of Learning and Motivation 24 Academic Press, 1989, pp. 109–165 DOI: 10.1016/S0079-7421(08)60536-8
  13. “Towards Continual Reinforcement Learning: A Review and Perspectives” arXiv, 2020 DOI: 10.48550/arXiv.2012.13490
  14. “Benchmarking Model-Based Reinforcement Learning” arXiv, 2019 DOI: 10.48550/arXiv.1907.02057
  15. “Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning” In Machine Learning, 2023 DOI: 10.1007/s10994-023-06422-w
  16. “Integrating Model-Driven and Data-Driven Methods for Power System Frequency Stability Assessment and Control” Conference Name: IEEE Transactions on Power Systems In IEEE Transactions on Power Systems 34.6, 2019, pp. 4557–4568 DOI: 10.1109/TPWRS.2019.2919522
  17. “Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach” arXiv, 2020 DOI: 10.48550/arXiv.2006.07476
  18. “Robust agents learn causal world models”, 2024 arXiv:2402.10877 [cs.AI]
  19. Richard S Sutton and Andrew G Barto “Reinforcement learning: An introduction” MIT press, 2018
  20. “Rainbow: Combining Improvements in DQN” In The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 2018, pp. 3215–3222 URL: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/17204/16680
  21. “Supervised assisted deep reinforcement learning for emergency voltage control of power systems” In Neurocomputing 475 Elsevier, 2022, pp. 69–79
  22. “Continuous control with deep reinforcement learning” In 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings, 2016 arXiv:1509.02971
  23. Scott Fujimoto, Herke Hoof and David Meger “Addressing Function Approximation Error in Actor-Critic Methods” ISSN: 2640-3498 In Proceedings of the 35th International Conference on Machine Learning PMLR, 2018, pp. 1587–1596 URL: https://proceedings.mlr.press/v80/fujimoto18a.html
  24. “Proximal Policy Optimization Algorithms”, 2017 arXiv: http://arxiv.org/abs/1707.06347
  25. “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor” In arXiv:1801.01290 [cs, stat], 2018 arXiv:1801.01290
  26. “Distributional deep reinforcement learning-based emergency frequency control” In IEEE Transactions on Power Systems 37.4 IEEE, 2021, pp. 2720–2730
  27. Rafael Figueiredo Prudencio, Marcos R.O.A. Maximo and Esther Luna Colombini “A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems” arXiv, 2022 DOI: 10.48550/arXiv.2203.01387
  28. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems” arXiv, 2020 DOI: 10.48550/arXiv.2005.01643
  29. “A survey on model-based reinforcement learning” In Science China Information Sciences 67.2 Springer, 2024, pp. 121101
  30. Xian Yeow Lee, Soumik Sarkar and Yubo Wang “A Graph Policy Network Approach for Volt-Var Control in Power Distribution Systems” arXiv, 2022 DOI: 10.48550/arXiv.2109.12073
  31. “A Multi-Agent Deep Reinforcement Learning Based Voltage Regulation Using Coordinated PV Inverters” Conference Name: IEEE Transactions on Power Systems In IEEE Transactions on Power Systems 35.5, 2020, pp. 4120–4123 DOI: 10.1109/TPWRS.2020.3000652
  32. “Stability Constrained Reinforcement Learning for Real-Time Voltage Control” ISSN: 2378-5861 In 2022 American Control Conference (ACC), 2022, pp. 2715–2721 DOI: 10.23919/ACC53348.2022.9867476
  33. “Model-augmented safe reinforcement learning for Volt-VAR control in power distribution networks” In Applied Energy 313 Elsevier, 2022, pp. 118762
  34. “Real-Time Optimal Power Flow Using Twin Delayed Deep Deterministic Policy Gradient Algorithm” In IEEE Access 8, 2020, pp. 213611–213618 DOI: 10.1109/ACCESS.2020.3041007
  35. “A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning” In Journal of Modern Power Systems and Clean Energy 8.6, 2020, pp. 1128–1139 DOI: 10.35833/MPCE.2020.000522
  36. “Deep Reinforcement Learning for Economic Dispatch of Virtual Power Plant in Internet of Energy” In IEEE Internet of Things Journal 7.7, 2020, pp. 6288–6301 DOI: 10.1109/JIOT.2020.2966232
  37. “A Learning-Based Power Management Method for Networked Microgrids Under Incomplete Information” In IEEE Transactions on Smart Grid 11.2, 2020, pp. 1193–1204 DOI: 10.1109/TSG.2019.2933502
  38. “Coordination of Electric Vehicle Charging Through Multiagent Reinforcement Learning” In IEEE Transactions on Smart Grid 11.3, 2020, pp. 2347–2356 DOI: 10.1109/TSG.2019.2952331
  39. “Adversarial Resilience Learning—Towards systemic vulnerability analysis for large and complex systems” In ENERGY 2019, The Ninth International Conference on Smart Grids, Green Communications and IT Energy-aware Technologies Athens, Greece: IARIA XPS Press, 2019, pp. 24–32
  40. Eric MSP Veith “An Architecture for Reliable Learning Agents in Power Grids” In ENERGY 2023, The Thirteenth International Conference on Smart Grids, Green Communications and IT Energy-aware Technologies Barcelona, Spain: IARIA XPS Press, 2023, pp. 13–16
  41. Torben Logemann and Eric MSP Veith “NN2EQCDT: Equivalent Transformation of Feed-Forward Neural Networks as DRL Policies into Compressed Decision Trees” In COGNITIVE 2023 : The Fifteenth International Conference on Advanced Cognitive Technologies and Applications 15 ThinkMind, online, 2023, pp. 94–100 IARIA
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up