Reinforcement learning to maximise wind turbine energy generation (2402.11384v1)
Abstract: We propose a reinforcement learning strategy to control wind turbine energy generation by actively changing the rotor speed, the rotor yaw angle and the blade pitch angle. A double deep Q-learning with a prioritized experience replay agent is coupled with a blade element momentum model and is trained to allow control for changing winds. The agent is trained to decide the best control (speed, yaw, pitch) for simple steady winds and is subsequently challenged with real dynamic turbulent winds, showing good performance. The double deep Q- learning is compared with a classic value iteration reinforcement learning control and both strategies outperform a classic PID control in all environments. Furthermore, the reinforcement learning approach is well suited to changing environments including turbulent/gusty winds, showing great adaptability. Finally, we compare all control strategies with real winds and compute the annual energy production. In this case, the double deep Q-learning algorithm also outperforms classic methodologies.
- Optuna: A next-generation hyperparameter optimization framework, 2019.
- Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
- Wind farm control - part i: A review on control system concepts and structures. IET Renewable Power Generation, 15(10):2085–2108, 2021.
- Wind Energy Handbook. Wiley, 2011.
- Reinforcement-based robust variable pitch control of wind turbines. IEEE Access, 8:20493–20502, 2020.
- F. Chollet et al. Keras. https://keras.io, 2015.
- zephyr - large on shore wind turbine benchmark, Mar. 2022.
- M. Churchfield. A method for designing generic wind turbine models representative of real turbines and generic siemens swt-2.3-93 and vestas v80 specifications. National Renewable Energy Laboratory, National Wind Technology Center, 2013.
- Intelligent wind farm control via deep reinforcement learning and high-fidelity simulations. Applied Energy, 292:116928, 2021.
- M. Drela. Xfoil: An analysis and design system for low reynolds number airfoils. In Low Reynolds Number Aerodynamics: Proceedings of the Conference Notre Dame, Indiana, USA, 5–7 June 1989, pages 1–12. Springer, 1989.
- A review on deep reinforcement learning for fluid mechanics. Computers & Fluids, 225:104973, 2021.
- H. Glauert. Airplane Propellers, pages 169–360. Springer Berlin Heidelberg, Berlin, Heidelberg, 1935.
- D. Jager and A. Andreas. Nrel national wind technology center (nwtc): M2 tower; boulder, colorado (data). http://dx.doi.org/10.5439/1052222, 1996. NREL Report No. DA-5500-56489.
- jcwleo. Create sumtree.py. https://github.com/rlcode/per/blob/master/SumTree.py, 2018. Latest commit 2685120 on Jan 29, 2018.
- Marlyc: Multi-agent reinforcement learning yaw control. Renewable Energy, 217:119129, 2023.
- D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Q-learning based maximum power extraction for wind energy conversion system with variable wind speed. IEEE Transactions on Energy Conversion, 35(3):1160–1170, 2020.
- Improving aircraft performance using machine learning: A review. Aerospace Science and Technology, 138:108354, 2023.
- m lundberg. simple-pid. https://github.com/m-lundberg/simple-pid, 2023.
- Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
- S. A. Ning. A simple solution method for the blade element momentum equations with guaranteed convergence. Wind Energy, 17(9):1327–1345, 2014.
- J. G. Njiri and D. Söffker. State-of-the-art in wind turbine control: Trends and challenges. Renewable and Sustainable Energy Reviews, 60:377–393, 2016.
- A review on wind turbine control and its associated methods. Journal of Cleaner Production, 174:945–953, 2018.
- NREL WISDEM Development Team. CCBlade - Blade Element Momentum Theory (BEM) Wind Turbine Python Module. 2019.
- Optimization of wind farm operation with a noise constraint. Wind Energy Science, 8(2):255–276, 2023.
- A. Puech and J. Read. An improved yaw control algorithm for wind turbines via reinforcement learning. pages 614–630, Cham, 2023. Springer Nature Switzerland.
- T. Revaz and F. Porté-Agel. Large-eddy simulation of wind turbine flows: A new evaluation of actuator disk models. Energies, 14(13), 2021.
- Artificial neural network based reinforcement learning for wind turbine yaw control. Energies, 12(3), 2019.
- Performance enhancement of the artificial neural network–based reinforcement learning for wind turbine yaw control. Wind Energy, 23(3):676–690, 2020.
- Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
- Wind turbine pitch reinforcement learning control improved by pid regulator and learning observer. 2020.
- J. Sierra-García and M. Santos Peñas. Performance analysis of a wind turbine pitch neurocontroller with unsupervised learning. 2020:1–15, 09 2020.
- J. E. Sierra-García and M. Santos. Exploring reward strategies for wind turbine pitch control by reinforcement learning. Applied Sciences, 10(21):7462, 2020.
- Reinforcement Learning: An Introduction. MIT Press, 1998.
- Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
- C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8:279–292, 1992.
- An adaptive network-based reinforcement learning method for mppt control of pmsg wind energy conversion systems. IEEE Transactions on Power Electronics, 31(11):7837–7848, 2016.
- Data-driven torque and pitch control of wind turbines via reinforcement learning. Renewable Energy, 215:118893, 2023.