Controlling Commercial Cooling Systems Using Reinforcement Learning (2211.07357v2)

Abstract: This paper is a technical overview of DeepMind and Google's recent work on reinforcement learning for controlling commercial cooling systems. Building on expertise that began with cooling Google's data centers more efficiently, we recently conducted live experiments on two real-world facilities in partnership with Trane Technologies, a building management system provider. These live experiments had a variety of challenges in areas such as evaluation, learning from offline data, and constraint satisfaction. Our paper describes these challenges in the hope that awareness of them will benefit future applied RL work. We also describe the way we adapted our RL system to deal with these challenges, resulting in energy savings of approximately 9% and 13% respectively at the two live experiment sites.

• The paper demonstrates that the RL-based BCOOLER algorithm reduces energy use by up to 13% while maintaining safety and operational constraints.
• It frames HVAC control as a Markov Decision Process and uses an ensemble of neural networks to balance energy efficiency with rule compliance.
• Real-world experiments over three months validate its effectiveness, suggesting broad potential for industrial HVAC optimization.

Overview of "Controlling Commercial Cooling Systems Using Reinforcement Learning"

This paper presents a comprehensive analysis of using reinforcement learning (RL) to control commercial cooling systems, building upon previous efforts in optimizing the cooling systems at Google's data centers. The research was conducted through live experiments in real-world settings in collaboration with Trane Technologies. These experiments encountered various challenges including evaluation constraints, learning from offline data, and maintaining compliance with operational constraints.

Research Methodology

The paper utilizes reinforcement learning (RL) because it transforms HVAC control into a sequential decision-making problem, suitable for RL paradigms. The problem is framed as a Markov Decision Process (MDP), characterized by states expressed through sensor measurements, actions corresponding to equipment setpoints, and rewards linked to energy consumption. The RL agent aims to minimize energy use while adhering to operational constraints in the control of a chiller plant, which is typically a significant component of HVAC systems.

System Implementation

The core algorithm, referred to as BCOOLER (BVE-based Constrained Optimization Learner with Ensemble Regularization), is built to address the specific challenges encountered in controlling cooling systems. It employs an ensemble of neural networks to predict action value functions and constraint violations. Unlike existing controllers, which typically rely on fixed heuristic rules, BCOOLER demands adaptive, real-time decision-making capabilities.

Key Challenges

The paper details several key challenges:

• Data Limitations: Learning and evaluation were conducted with limited data in the absence of high-fidelity simulators.
• Constraint Satisfaction: Ensuring compliance with complex operational constraints was crucial, particularly to maintain user safety and equipment integrity.
• Non-stationary Environments: Changes in environmental and operational factors required adaptability.
• Real-time Inference: Decisions were made under time constraints, necessitating efficient action space pruning and evaluation.
• Multi-timescale Dynamics: Actions had varying temporal impacts, necessitating a hierarchical control approach.

Experimental Evaluation

BCOOLER was tested over three months across two facilities with varying operational contexts, achieving energy savings of 9% and 13% relative to the traditional heuristic controllers. Importantly, the agent was able to maintain occupant comfort and system safety by satisfying pre-defined constraints.

The results suggest a greater optimization capacity in cooler conditions and lower loads, aligning with conditions where operational settings are often suboptimized by simpler heuristic methods. Moreover, BCOOLER effectively managed equipment trade-offs, such as balancing chiller and cooling tower power use, demonstrating its nuanced understanding of HVAC operations.

Implications and Future Work

This research highlights the practical potential of RL in increasing energy efficiency within HVAC systems, offering a path toward broader application in varied industrial contexts. The paper suggests future work focusing on improving data efficiency through enhanced domain-specific modeling, extended simulation capabilities for knowledge transfer, and generalization across different facility types. Incorporating human feedback into AI models can further tailor agent behavior to align with diverse operational environments.

In conclusion, this paper underscores the viability of RL for optimizing commercial cooling systems, pointing toward its broader applicability in real-time industrial control tasks. The insights derived from handling unique challenges in this field contribute significantly to the RL and smart building control literature.