Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning (1904.09374v2)

Abstract: Modern distribution grids are currently being challenged by frequent and sizable voltage fluctuations, due mainly to the increasing deployment of electric vehicles and renewable generators. Existing approaches to maintaining bus voltage magnitudes within the desired region can cope with either traditional utility-owned devices (e.g., shunt capacitors), or contemporary smart inverters that come with distributed generation units (e.g., photovoltaic plants). The discrete on-off commitment of capacitor units is often configured on an hourly or daily basis, yet smart inverters can be controlled within milliseconds, thus challenging joint control of these two types of assets. In this context, a novel two-timescale voltage regulation scheme is developed for distribution grids by judiciously coupling data-driven with physicsbased optimization. On a faster timescale, say every second, the optimal setpoints of smart inverters are obtained by minimizing instantaneous bus voltage deviations from their nominal values, based on either the exact alternating current power flow model or a linear approximant of it; whereas, on the slower timescale (e.g., every hour), shunt capacitors are configured to minimize the longterm discounted voltage deviations using a deep reinforcement learning algorithm. Extensive numerical tests on a real-world 47- bus distribution network as well as the IEEE 123-bus test feeder using real data corroborate the effectiveness of the novel scheme.

• The paper introduces a novel two-timescale voltage control framework combining deep reinforcement learning for slow capacitor adjustments with fast optimization for smart inverter control.
• It employs DRL techniques like experience replay and hyper DRL to integrate diverse assets and ensure scalability across large grids.
• Numerical tests on real-world systems validated the method's superior performance in minimizing voltage deviations and showed linearized models are effective for fast decisions.

Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning

The paper under discussion presents a novel framework for voltage regulation in modern distribution grids, leveraging deep reinforcement learning (DRL) within a two-timescale control scheme. The framework effectively integrates the control of traditional utility-owned devices such as shunt capacitors and contemporary smart inverters, thus addressing the increasing complexity due to the penetration of electric vehicles and renewable energy sources.

Overview of the Framework

The proposed methodology operates on two distinct timescales to manage voltage magnitudes within desired limits. On a slower timescale, usually hourly, the shunt capacitors are configured utilizing a DRL approach that models the system as a Markov Decision Process (MDP). This helps in optimizing the long-term voltage profile by recognizing patterns in consumption and generation dynamics.

In contrast, the faster timescale operates on a per-second basis, adjusting the smart inverters' reactive power setpoints. This rapid adjustment aims to minimize instantaneous bus voltage deviations using either the exact alternating current (AC) power flow model or its linear approximation. Crucially, the former is solved using a second-order cone program (SOCP) which, although computationally intensive, provides robustness for smaller networks. Meanwhile, for larger networks, the linear approximation offers a computationally efficient alternative, framed as a quadratic program.

Technical Contributions

The research presents several key contributions:

• Integration of Diverse Assets: Demonstrated within the context of a hybrid optimization-learning framework is the successful integration of two distinct control assets: utility-owned shunt capacitors and smart inverters.
• DRL for Slow Timescale Decisions: This is a significant advance as it demonstrates the potential of DRL in addressing the curse of dimensionality typically faced in MDPs with extensive state and action spaces.
• Advanced Computational Techniques: Use of target networks and experience replay within the DRL setup ensures stability and convergence in learning the optimal control policies, which is a critical requirement for real-world applications.
• Scalability: Incorporated hyper deep Q-networks that enable scalability of the DRL approach, ensuring that it can effectively manage distribution grids with a large number of controllable devices.

Numerical Validation

The efficacy of the proposed system was validated through comprehensive numerical tests on a real-world 47-bus distribution feeder and the IEEE 123-bus test system. Results showcased the DRL-based method's superior performance in minimizing voltage deviations, confirming that it learns optimal control strategies over time. The paper underscores the capacity of the simple linearized models to closely approximate the performance of more accurate AC models, thereby justifying their use for fast-timescale decisions.

Implications and Future Work

The dual-time-scale strategy for voltage regulation presents a promising approach that blends physical and data-driven models to manage the challenges posed by increased penetrations of distributed and intermittent resources. The results indicate potential improvements in operational efficiencies and responsiveness in adaptive grid environments.

Future research might explore further refinements to the DRL framework, particularly in better capturing the dynamics in states and actions reflective of more complex network behaviors. Additionally, extending this approach to encompass other forms of grid operations such as optimal power flow in real time and holistic unit commitment strategies could significantly enhance grid reliability and resilience.

In conclusion, the paper presents a sophisticated, yet practical, strategy for modern grid voltage control. The integration of fast-scaling computational techniques with data-driven insights is poised to make significant impacts on the future of smart grid management.