- The paper demonstrates that an RL framework can co-optimize design and control, outperforming traditional two-step approaches.
- It employs a parametric design distribution with entropy regularisation to prevent convergence to local optima.
- Experiments on a building-scale PV-battery system reveal superior performance and cost reduction compared to MILP and rule-based methods.
Reinforcement Learning for Efficient Design and Control Co-optimisation of Energy Systems
In the context of the ongoing energy transition towards decentralised renewable energy sources, this paper addresses the complexity of integrating heterogeneous and weather-dependent renewables into energy systems. The authors propose a novel reinforcement learning (RL) framework tailored for the co-optimisation of design and control in energy systems. This approach aims to enhance system efficiency and improve the integration of renewable sources without the need for explicit system modelling.
Background and Methodology
The traditional development process for renewable-integrated energy systems typically separates the design phase from the operational control phase, resulting in suboptimal performance. Previous research has utilised mathematical modelling techniques such as linear programming, stochastic models, robust optimisation, and evolutionary algorithms to address these challenges. However, these methods often require detailed system modelling and may not effectively handle the complex interactions inherent in modern energy systems.
Advancements in RL, particularly in gradient-based techniques like Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), have shown promise in computing control policies for complex problems. Building on these advancements, the proposed RL framework integrates RL's model-free capabilities to jointly optimise both design and control. This integrated approach eliminates the need for system modelling and leverages a parametric design distribution to explore continuous design spaces efficiently.
Reinforcement Learning Framework
The RL framework introduced in this paper employs a parametric design distribution, distinguishing it from previous deterministic representations of design variables. It incorporates entropy regularisation into the optimisation process to prevent convergence to local optima. The framework utilises Deep Deterministic Policy Gradient (DDPG), an off-policy actor-critic algorithm, allowing for efficient learning from past experiences stored in a replay buffer and accommodating infinite time horizons.
During the co-optimisation process, the framework updates both the control policy and the design distribution parameters iteratively. The control policy is refined to enhance expected returns for sampled designs, while the design distribution is adjusted to increase the likelihood of high-performance designs. This simultaneous updating process guides the system towards an optimal design and associated control policy, maximising overall system performance.
Experimental Setup
The proposed framework is evaluated on a building-scale PV-battery system with the aim of minimising total electricity costs. The system consists of a PV installation and a stationary lithium-ion battery, with an additional bidirectional EV charging point. The evaluation benchmarks the RL co-optimisation against traditional approaches: a MILP-based design optimisation followed by RL-based control, and a rule-based control policy with design optimisation.
The experiments use a historical dataset of PV production and electricity consumption, divided into training and validation sets. The RL-based design optimisations show that the co-optimisation converges to superior operational performance compared to both the design-only and best two-step approaches. The results demonstrate that the integrated framework effectively computes high-performing design parameters and control policies.
Results and Discussion
The experimental results reveal that the co-optimisation framework achieves better performance than traditional methods. The co-optimised system efficiently integrates renewable sources and reduces electricity costs. The framework's capability to provide an interval of optimal design values offers flexibility and insights into design sensitivities, unlike the deterministic solutions often produced by MILP.
However, the framework's sensitivity to hyperparameters and the challenge of scaling to additional design parameters highlight areas for future research. Extending the dataset to include multiple years and integrating more complex system dynamics could enhance the framework's robustness and applicability.
Conclusion
This paper successfully demonstrates the feasibility and effectiveness of an RL-based framework for the co-optimisation of design and control in energy systems. By leveraging recent advances in policy gradient techniques, the framework bridges the gap between theoretical RL research and practical energy system optimisation. The findings underscore the importance of integrated co-optimisation strategies for improving system performance in the dynamic and complex landscape of renewable energy integration.
The proposed RL framework provides a promising direction for future research, with potential applications extending beyond energy systems. Its model-free nature and adaptability to various control algorithms make it a versatile tool for addressing co-optimisation challenges in diverse domains.