Introduction to Online Control

Abstract: This text presents an introduction to an emerging paradigm in control of dynamical systems and differentiable reinforcement learning called online nonstochastic control. The new approach applies techniques from online convex optimization and convex relaxations to obtain new methods with provable guarantees for classical settings in optimal and robust control. The primary distinction between online nonstochastic control and other frameworks is the objective. In optimal control, robust control, and other control methodologies that assume stochastic noise, the goal is to perform comparably to an offline optimal strategy. In online nonstochastic control, both the cost functions as well as the perturbations from the assumed dynamical model are chosen by an adversary. Thus the optimal policy is not defined a priori. Rather, the target is to attain low regret against the best policy in hindsight from a benchmark class of policies. This objective suggests the use of the decision making framework of online convex optimization as an algorithmic methodology. The resulting methods are based on iterative mathematical optimization algorithms, and are accompanied by finite-time regret and computational complexity guarantees.

• The paper presents a framework that minimizes regret against benchmark policies in adversarial dynamic environments.
• It integrates online convex optimization with traditional control methods like LQR to achieve finite-time performance guarantees.
• It explores algorithms such as the Gradient Perturbation Controller to adapt control strategies in partially observable and unknown systems.

An Overview of Online Nonstochastic Control in Dynamical Systems

This document serves as a comprehensive introduction to online nonstochastic control, a modern paradigm in control theory that addresses the challenges associated with decision-making in dynamic environments subject to adversarial disturbances. The text systematically builds a framework that incorporates concepts from online convex optimization to achieve finite-time guarantees in optimal and robust control scenarios. Crucially, online nonstochastic control focuses on minimizing regret against a benchmark class of policies, adapting to unpredictable or adversarial perturbations in the process.

Background and Motivation

Traditional control frameworks, such as solution concepts in dynamical systems and reinforcement learning through Markov Decision Processes (MDP), form the backbone of control theory. Control theory typically involves manipulating physical systems to achieve desired functionalities, which has evolved to incorporate algorithmic approaches as computational capabilities advanced.

Online nonstochastic control differentiates itself from standard optimal and robust control frameworks by considering adversarially chosen perturbations, thus not relying on stochastic assumptions. This approach is particularly motivated by practical considerations wherein environmental disturbances do not follow well-defined probabilistic models, a stark contrast to standard assumptions in stochastic control models.

Linear Dynamical Systems and Control

The text articulates the nuances of linear dynamical systems (LDS) that operate according to linear time-variant or time-invariant transitions. It highlights efficient verification for stabilizability and controllability in LDS, which are intrinsic properties relevant to achieving desired control outcomes. Specifically, stabilizability ensures that there exists a feedback control mechanism to drive the system states towards zero, while controllability refers to the ability to steer system states to any target state over time via appropriate control inputs.

The document further elaborates on optimal control strategies through the Linear Quadratic Regulator (LQR) framework, which seeks to minimize a quadratic cost function in systems subject to Gaussian noise. Solutions to LQR, characterized by the Riccati equation, allow deriving state-feedback policies that are linear with respect to the system states.

Policy Classes and Nonstochastic Control

The discussion transitions to an exploration of various policy classes applicable in control tasks, outlining their representational power and implications for online decision-making. These include State-Action Control (SAC), Disturbance-Action Control (DAC), Linear Dynamical Controllers, and Disturbance Response Controllers for partially observed systems.

Online nonstochastic control maritally integrates these policy classes using a benchmark of policies against which regret is minimized. The Gradient Perturbation Controller (GPC) and Gradient Response Controller (GRC) are notable algorithms that leverage online gradient descent techniques to adjust policy parameters iteratively. These algorithms achieve sublinear regret against comparable policy classes, demonstrating adaptability and robustness in adversarial settings.

Challenges of Partial Observability and Unknown Systems

The complexity of real-world systems often extends beyond fully observable dynamics to partially observed environments, necessitating sophisticated control approaches like Dynamic State Estimation and learning-based strategies for system identification. In online nonstochastic control, disturbance response controllers become essential to handling partial observability, providing a robust framework to approximate the best policy despite incomplete information.

Furthermore, the text tackles the issue of unknown system parameters, detailing methodologies for system identification with adversarial noise. Monte Carlo sampling techniques and spectral filtering are presented as advanced strategies to efficiently infer system matrices from observed data, paving the way for improved control policy adaptation in previously uncharacterized systems.

Conclusion and Implications

The document illustrates how online nonstochastic control offers practical and theoretical advancements over traditional methods, providing computational efficiency and adaptability in complex, uncertain environments. By harnessing online optimization and convex analytical approaches, this framework effectively bridges gaps between theory and real-world applications, marking a significant shift towards robustness and adaptability in control systems. Future research directions may continue to explore scalable techniques for partially observed and nonlinear systems, further broadening the applicability of online nonstochastic control principles.