ReACT: Reinforcement Learning for Controller Parametrization using B-Spline Geometries (2401.05251v1)

Abstract: Robust and performant controllers are essential for industrial applications. However, deriving controller parameters for complex and nonlinear systems is challenging and time-consuming. To facilitate automatic controller parametrization, this work presents a novel approach using deep reinforcement learning (DRL) with N-dimensional B-spline geometries (BSGs). We focus on the control of parameter-variant systems, a class of systems with complex behavior which depends on the operating conditions. For this system class, gain-scheduling control structures are widely used in applications across industries due to well-known design principles. Facilitating the expensive controller parametrization task regarding these control structures, we deploy an DRL agent. Based on control system observations, the agent autonomously decides how to adapt the controller parameters. We make the adaptation process more efficient by introducing BSGs to map the controller parameters which may depend on numerous operating conditions. To preprocess time-series data and extract a fixed-length feature vector, we use a long short-term memory (LSTM) neural networks. Furthermore, this work contributes actor regularizations that are relevant to real-world environments which differ from training. Accordingly, we apply dropout layer normalization to the actor and critic networks of the truncated quantile critic (TQC) algorithm. To show our approach's working principle and effectiveness, we train and evaluate the DRL agent on the parametrization task of an industrial control structure with parameter lookup tables.

• The paper introduces ReACT, which autonomously tunes controllers using deep reinforcement learning and B-spline geometries to simplify the action-space complexity.
• It employs LSTM networks along with actor regularization, dropout, and layer normalization to enhance training stability and generalization.
• Experiments demonstrate that ReACT converges faster and outperforms traditional DRL methods in noisy, dynamic industrial environments.

Introduction

In industrial settings, the creation of robust and high-performing control systems is necessary for the smooth operation of complex machinery. Typically, these systems rely on controllers that need to be meticulously tuned, which can be both challenging and time-consuming, especially when dealing with nonlinear dynamics that vary with operating conditions.

Reinforcement Learning for Efficient Parametrization

Researchers have taken a step forward by using deep reinforcement learning (DRL) along with B-spline geometries to address this problem. They have developed a framework that navigates the complexity of controller parametrization by allowing an agent to autonomously determine the best parameters for the controller. This new process, encapsulated in an approach named ReACT (Regularized actor and critic TQC), uses B-spline geometries as a novel interface to reduce the action-space complexity that the DRL agent must navigate.

B-Spline Geometries in Action

B-spline geometries provide a versatile tool for representing multi-dimensional spaces smoothly and compactly. By utilizing these geometries, the researchers can map complex controller parameters across multiple conditions using control points to shape the control law adaptively. The ReACT approach hinges on the premise that control points can be incrementally adjusted in response to the feedback from the system, effectively tuning a controller's performance dynamically.

The model learns to map numerous operational conditions to controller parameters more efficiently by training on time-series data using long short-term memory (LSTM) neural networks. It incorporates techniques such as actor regularization with dropout and layer normalization into the DRL training routine to improve the stability and generalization of the learning process.

Experimentation and Results

Experiments were conducted on simulated control systems, challenging the ReACT method against traditional DRL algorithms. The results showed ReACT's superiority in terms of quicker training convergence and superior task performance. Key contributions of the research include the effective parametrization of high-dimensional controller spaces using B-splines and the proposed self-competition reward mechanism that encourages ongoing improvement during the training phase.

Conclusion

The ReACT framework offers a promising path towards automating the controller tuning process, driving efficiencies, and potentially leading to more consistent operational outcomes in industrial applications. The main advantage is the combination of B-spline geometries with a reinforcement learning agent offering a systemic and dynamic way to optimize controller parameters catering to the complex interplay of changing system operating conditions.

By performing robustly in the presence of noise and disturbances, this approach opens doors to future advancements, where more sophisticated aspects such as the stability and robustness of the controlled systems could be well within the learning domain of such an intelligent agent.