Time-Varying Constraint-Aware Reinforcement Learning for Energy Storage Control (2405.10536v1)
Abstract: Energy storage devices, such as batteries, thermal energy storages, and hydrogen systems, can help mitigate climate change by ensuring a more stable and sustainable power supply. To maximize the effectiveness of such energy storage, determining the appropriate charging and discharging amounts for each time period is crucial. Reinforcement learning is preferred over traditional optimization for the control of energy storage due to its ability to adapt to dynamic and complex environments. However, the continuous nature of charging and discharging levels in energy storage poses limitations for discrete reinforcement learning, and time-varying feasible charge-discharge range based on state of charge (SoC) variability also limits the conventional continuous reinforcement learning. In this paper, we propose a continuous reinforcement learning approach that takes into account the time-varying feasible charge-discharge range. An additional objective function was introduced for learning the feasible action range for each time period, supplementing the objectives of training the actor for policy learning and the critic for value learning. This actively promotes the utilization of energy storage by preventing them from getting stuck in suboptimal states, such as continuous full charging or discharging. This is achieved through the enforcement of the charging and discharging levels into the feasible action range. The experimental results demonstrated that the proposed method further maximized the effectiveness of energy storage by actively enhancing its utilization.
- The changing price of wholesale UK electricity over more than a decade. https://www.ice.org.uk/knowledge-and-resources/briefing-sheet/the-changing-price-of-wholesale-uk-electricity, 2017. [Online; accessed 30-Jan-2024].
- Energy storage technologies and real life applications–a state of the art review. Applied Energy, 179:350–377, 2016.
- Economic viability of energy storage systems based on price arbitrage potential in real-time us electricity markets. Applied Energy, 114:512–519, 2014.
- Deep reinforcement learning-based energy storage arbitrage with accurate lithium-ion battery degradation model. IEEE Transactions on Smart Grid, 11(5):4513–4521, 2020.
- Shin-ichi Inage. The role of large-scale energy storage under high shares of renewable energy. Advances in Energy Systems: The Large-scale Renewable Energy Integration Challenge, pp. 221–243, 2019.
- The future role of thermal energy storage in 100% renewable electricity systems. Renewable and Sustainable Energy Transition, 4:100059, 2023.
- DeepComp: Deep reinforcement learning based renewable energy error compensable forecasting. Applied Energy, 294:116970, 2021.
- Deep reinforcement learning based real-time renewable energy bidding with battery control. IEEE Transactions on Energy Markets, Policy and Regulation, 2023.
- Reinforcement learning-based optimal scheduling model of battery energy storage system at the building level. Renewable and Sustainable Energy Reviews, 190:114054, 2024.
- Energy maestro–transactive energy mechanism. In 2022 IEEE International Conference on Big Data (Big Data), pp. 6727–6729. IEEE, 2022.
- Federated reinforcement learning for energy management of multiple smart homes with distributed energy resources. IEEE Transactions on Industrial Informatics, 18(1):488–497, 2020.
- Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
- A hybrid heuristic-reinforcement learning-based real-time control model for residential behind-the-meter PV-battery systems. Applied Energy, 355:122244, 2024.
- Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
- A safe reinforcement learning-based charging strategy for electric vehicles in residential microgrid. Applied Energy, 348:121490, 2023.
- Exploiting the sign of the advantage function to learn deterministic policies in continuous domains. arXiv preprint arXiv:1906.04556, 2019.