A Lightweight Calibrated Simulation Enabling Efficient Offline Learning for Optimal Control of Real Buildings (2310.08569v1)
Abstract: Modern commercial Heating, Ventilation, and Air Conditioning (HVAC) devices form a complex and interconnected thermodynamic system with the building and outside weather conditions, and current setpoint control policies are not fully optimized for minimizing energy use and carbon emission. Given a suitable training environment, a Reinforcement Learning (RL) model is able to improve upon these policies, but training such a model, especially in a way that scales to thousands of buildings, presents many real world challenges. We propose a novel simulation-based approach, where a customized simulator is used to train the agent for each building. Our open-source simulator (available online: https://github.com/google/sbsim) is lightweight and calibrated via telemetry from the building to reach a higher level of fidelity. On a two-story, 68,000 square foot building, with 127 devices, we were able to calibrate our simulator to have just over half a degree of drift from the real world over a six-hour interval. This approach is an important step toward having a real-world RL control system that can be scaled to many buildings, allowing for greater efficiency and resulting in reduced energy consumption and carbon emissions.
- Reinforced model predictive control (RL-MPC) for building energy management. Applied Energy 309 (2022), 118346.
- Reinforcement learning for whole-building HVAC control and demand response. Energy and AI 2 (2020), 100020.
- Energy modelling and control of building heating and cooling systems with data-driven and hybrid models—A review. Renewable and Sustainable Energy Reviews 183 (2023), 113496.
- Data-driven offline reinforcement learning for HVAC-systems. Energy 261 (2022), 125290.
- Gnu-rl: A practical and scalable reinforcement learning solution for building hvac control using a differentiable mpc policy. Frontiers in Built Environment 6 (2020), 562239.
- Fast Human-in-the-loop Control for HVAC Systems via Meta-learning and Model-based Offline Reinforcement Learning. IEEE Transactions on Sustainable Computing (2023).
- EnergyPlus: creating a new-generation building energy simulation program. Energy and buildings 33, 4 (2001), 319–331.
- Cheng Gao and Dan Wang. 2023. Comparative study of model-based and model-free reinforcement learning control performance in HVAC systems. Journal of Building Engineering 74 (2023), 106852.
- Basil Kouvaritakis and Mark Cannon. 2016. Model predictive control. Switzerland: Springer International Publishing 38 (2016).
- Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020).
- Simeng Liu and Gregor P Henze. 2006. Experimental analysis of simulated reinforcement learning control for active and passive building thermal storage inventory: Part 2: Results and analysis. Energy and buildings 38, 2 (2006), 148–161.
- Fundamentals of computational fluid dynamics. Appl. Mech. Rev. 55, 4 (2002), B61–B61.
- Mengxue Lu and Joseph Lai. 2020. Review on carbon emissions of commercial buildings. Renewable and Sustainable Energy Reviews 119 (2020), 109545.
- Karl Mason and Santiago Grijalva. 2019. A review of reinforcement learning for autonomous building energy management. Computers & Electrical Engineering 78 (2019), 300–312.
- Heating, ventilating, and air conditioning: analysis and design. John Wiley & Sons.
- R Sendra-Arranz and A Gutiérrez. 2020. A long short-term memory artificial neural network to predict daily HVAC consumption in buildings. Energy and Buildings 216 (2020), 109952.
- E M Sparrow. 1993. Heat Transfer: Conduction [Lecture Notes].
- Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
- A Long-Short Term Memory Recurrent Neural Network Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems. (2017).
- A Comparison of Classical and Deep Reinforcement Learning Methods for HVAC Control. arXiv preprint arXiv:2308.05711 (2023).
- Control strategies for energy optimization of HVAC systems in small office buildings using energyplus tm. In 2019 IEEE Innovative Smart Grid Technologies-Asia (ISGT Asia). IEEE, 2698–2703.
- Deep reinforcement learning for building HVAC control. In Proceedings of the 54th annual design automation conference 2017. 1–6.
- A review of deep reinforcement learning for smart building energy management. IEEE Internet of Things Journal 8, 15 (2021), 12046–12063.
- Multi-agent deep reinforcement learning for HVAC control in commercial buildings. IEEE Transactions on Smart Grid 12, 1 (2020), 407–419.
- Building HVAC scheduling using reinforcement learning via neural network based model approximation. In Proceedings of the 6th ACM international conference on systems for energy-efficient buildings, cities, and transportation. 287–296.
- Hybrid-model-based deep reinforcement learning for heating, ventilation, and air-conditioning control. Frontiers in Energy Research 8 (2021), 610518.
- EnergyPlus model-based predictive control within design–build–operate energy information modelling infrastructure. Journal of Building Performance Simulation 8, 3 (2015), 121–134.
- Data-driven predictive control for smart HVAC system in IoT-integrated buildings with time-series forecasting and reinforcement learning. Applied Energy 338 (2023), 120936.
- Towards optimal control of air handling units using deep reinforcement learning and recurrent neural network. Building and Environment 168 (2020), 106535.