Closed-Loop Control Policy Learning

• Closed-loop control policy learning is the synthesis of feedback controllers that map sensor data to control actions for stabilizing, regulating, or optimizing dynamic systems.
• Recent methods integrate classical techniques (MPC, LQR) with reinforcement and imitation learning, leveraging neural networks for enhanced adaptability and robustness.
• Key advancements include formal guarantees on stability and safety, data-efficient sim-to-real transfers, and scalable distributed optimization across diverse application domains.

Closed-loop control policy learning refers to the synthesis, identification, or adaptation of parameterized feedback controllers that map online sensor data to control actions, aiming to stabilize, regulate, or optimize the trajectory of a dynamic system under real-time conditions. It is central in both classical control theory (e.g., MPC, LQR) and contemporary machine learning and reinforcement learning frameworks, with guarantees ranging from empirical robustness and sample efficiency to formal stability and constraint satisfaction. Recent developments leverage advances in neural network architectures, imitation learning, distributional robustness, formal methods, and scalable distributed optimization. The following sections delineate the principal technical approaches, theoretical results, and representative applications of closed-loop control policy learning as documented in recent literature.

1. Formal Definition and Problem Settings

Closed-loop control policy learning is formulated as the search for a (potentially parameterized) mapping $\pi_\theta : \mathcal{O} \rightarrow \mathcal{A}$ from observed signals $\mathcal{O}$ (states, outputs, or histories) to control actions $\mathcal{A}$, optimizing a system-level objective under feedback. The core settings include:

• Model-based optimal control: Direct feedback law synthesis using system models, e.g., MPC, LQR, or Hamilton-Jacobi-Bellman approaches (Ahn et al., 2022, Hu et al., 2023, Molu, 2023).
• Imitation learning/counterexample-guided synthesis: Learning closed-loop policies from expert trajectories, with formal constraints (e.g., reachability, safety) (Ahn et al., 2022, Ravanbakhsh et al., 2019, Makdah et al., 2021).
• Reinforcement learning: Online or offline RL to optimize closed-loop returns under uncertain or partially observed dynamics, including actor-critic, policy gradient, and distributed RL (Amici et al., 20 May 2026, Nasir et al., 2022, Rosolia et al., 2018, Zhang et al., 2024).
• Safe and constraint-aware learning: Policy learning under explicit state, input, or stability constraints, often using barrier functions, Lyapunov theory, or probabilistic safety envelopes (Hirt et al., 2024, Hao et al., 3 Oct 2025, Zhang et al., 2024).
• Multiagent/general-sum games: Feedback Nash equilibria learning in multiagent settings, notably Markov potential games with coupled constraints (Macua et al., 2018, Zhang et al., 2024).

These settings differ in policy parameterization (linear/affine, deep neural, diffusion-based, graph-based, etc.), supervision (model-based or data-driven), task specification (regulation, tracking, reachability), and guarantees (sample efficiency, stability, safety, regret bounds).

2. Algorithmic Architectures and Learning Frameworks

A wide range of architectures and learning procedures are deployed for closed-loop policy learning:

• Operator-Encoded and RF-Augmented Policies: The Model-Based Closed-Loop Control Algorithm (MB-CC) for stochastic PDEs incorporates a regularity feature (RF) block to extract noise-robust features and an operator-encoded network, mapping SPDE states to optimal control with priors (Hu et al., 8 May 2025).
• Diffusion-Model Policies: In visuomotor control, coupled diffusion-policy architectures generate temporally consistent anchor sequences (via a global diffusion head) and smooth, fine-grained subtrajectories (via a local diffusion head). Constraint injection is realized through masked denoising strategies, allowing instantaneous referencing to human-provided trajectory waypoints (Ma et al., 7 Apr 2026).
• Neural and Data-Efficient Model-Based Policies: Neural operators, RNNs, or GPR-compensated networks approximate system or residual dynamics for continuum manipulation, supporting both model-based and model-free control loops (Amici et al., 20 May 2026, Wang et al., 2022).
• Parameter-Efficient Feedback Representation: Piecewise affine interpolators (Rosolia et al., 2018), Lipschitz-constrained regressors (Makdah et al., 2021), and graph-based solvers realize efficient feedback policies with explicit closed-loop guarantees.
• Structured Policy Learning in Games and Distributed Systems: In Markov potential games, the existence of a scalar potential enables Nash equilibrium computation by solving a single-agent OCP, making closed-loop policy learning tractable using deep reinforcement learning with potential-aligned objectives (Macua et al., 2018).
• Barrier Function and Safety-Handling Methods: Control barrier function-based policy adaptations maintain invariance of performance bounds under policy updates, often by embedding quadratic programming (QP) safety corrections into gradient-based RL or optimization (Hao et al., 3 Oct 2025, Hirt et al., 2024, Zhang et al., 2024).

3. Theoretical Guarantees and Performance Bounds

Significant attention is paid to sample complexity, stability, robustness, and regret:

• Sample Complexity and Distributional Robustness: In on-policy imitation learning for MPC, forward training (rollout-based collection and stage-wise policy fitting) achieves exponentially greater efficiency—requiring only $O(\tau^* / (\epsilon \delta^2))$ expert rollouts versus $O(\exp(T))$ for batch behavior cloning—while ensuring stability and constraint satisfaction (Ahn et al., 2022).
• Stability, Feasibility, and Safety: Multiple frameworks instantiate Lyapunov-theoretic or ISS guarantees for the learned policy, either probabilistically (via GP barriers in Bayesian optimization (Hirt et al., 2024)) or deterministically (via explicit barrier functions in CBF-based policy updates (Hao et al., 3 Oct 2025), or robust constraint satisfaction in interpolated LP feedback (Rosolia et al., 2018)).
• Robust Generalization and Adversarial Performance: Imposing Lipschitz constraints on the policy admits finite-sample sup-norm error bounds and a fundamental tradeoff: tighter Lipschitz constraints improve adversarial measurement-noise robustness at the cost of nominal closed-loop performance (Makdah et al., 2021).
• Regret and Approximate Optimality: Empirical and analytical regret (suboptimality vs. the expert or open-loop optimum) can be explicitly bounded in terms of training error, expressiveness of the function class, and coverage of the demonstration or exploration distribution (Ravanbakhsh et al., 2019, Hu et al., 2023, Makdah et al., 2021).
• Multiagent and Distributed Stability: In distributed MPC, receding-horizon actor–critic algorithms with local or barrier-augmented cost structure are shown to guarantee closed-loop stability under mild local conditions, with empirical scalability up to $10^4$ agents and explicit region-of-attraction certificates (Zhang et al., 2024).

4. Data Efficiency, Sim-to-Real Transfer, and Practical Deployment

Multiple studies prioritize sample efficiency and sim-to-real robustness:

• Data-Efficient Policy Transfer: Sim-to-real transfer for continuum and microfiber manipulation exploits compact surrogate models (RNNs trained in simulation) augmented with lightweight real-data corrections (e.g. GPR residuals), enabling high-precision closed-loop control with as few as 100 real samples—a two-order-of-magnitude improvement over purely real-data training (Wang et al., 2022, Amici et al., 20 May 2026).
• Domain-Invariant and Feedback-Robust Strategies: Feedback policies exploiting real-time geometric or visual state estimation enable iterative correction for unmodeled effects (e.g., microscale friction), achieving sub-millimeter accuracy in unstructured deformable manipulation without domain randomization (Amici et al., 20 May 2026).
• End-to-End Sim-to-Real in Manufacturing: Closed-loop RL with physics-inspired fast simulators, combined with privilege-augmented rewards and noise injection, bridges the sim-to-real gap in industrial deposition applications (Piovarci et al., 2022).
• Multi-asset and Task-agnostic Policies: In CLRM for reservoir management, a single deep RL policy with well-indexed embeddings achieves near-optimal closed-loop performance across highly variable assets, reducing simulation cost by ~3× vs. assetwise training (Nasir et al., 2022).

5. Formal Verification, Constraints, and Structured Synthesis

Guaranteeing correctness, safety, and constraint enforcement is central:

• Counterexample-Guided Synthesis and Formal Reachability: Iterative CEGIS loops combine learning (from MPC demonstrator), verification (by falsification against specification), and set-valued policy representation, producing feedback policies formally certified to satisfy temporal logic properties and reachability requirements (Ravanbakhsh et al., 2019).
• Mixed $\mathcal{H}_2/\mathcal{H}_\infty$ Policy Synthesis: Differential-game-based, iterative algorithms for closed-loop Riccati equations produce robust controllers balancing disturbance attenuation and nominal performance, in a model-free setting (Molu, 2023).
• Safe Learning in MPC with Neural Parameterization: Constrained Bayesian optimization over neural MPC cost functionals, with black-box GP barriers for stability and state constraints, enables high-dimensional policy adaptation with probabilistically enforced closed-loop feasibility at each step (Hirt et al., 2024).
• Barrier-augmented Distributed RL: Recentered and relaxed barrier terms plus "force field"-inspired policy augmentation enable safe constraint satisfaction in distributed multirobot RL, while maintaining convergence and recursive feasibility (Zhang et al., 2024).

6. Application Domains and Empirical Results

Diverse real and synthetic domains are used to validate closed-loop policy learning:

• Robotic Manipulation and Visuomotor Control: Coupled-diffusion closed-loop policies handle OOD errors and trajectory rerouting with region-penetration rates near 100% on synthetic and real "via-point" tasks, outperforming DDP, Octo, and MPD by 20–50 pp (Ma et al., 7 Apr 2026).
• Power Systems and Adaptive Oscillation Suppression: Policy-gradient-based closed-loop gain tuning in EMT-in-the-loop simulation achieves ≈80% damping of subsynchronous oscillations in real-world power grid events, surpassing hand-tuned PI retuning by >2× (Mukherjee et al., 8 Nov 2025).
• Multiagent and Swarm Control: Distributed actor–critic receding-horizon training enables stable deployment to swarms of up to 10,000 robots or drones, handling nonlinearities and constraints without post-training adaptation (Zhang et al., 2024).
• Iterative/Autonomous Racing: Data-driven closed-loop interpolated policies for autonomous racing achieve performance within 1% of MPC with 30× reduction in online evaluation cost (Rosolia et al., 2018).
• Continuum and Soft Robot Control: Sim- and data-efficient closed-loop policies achieve <1.5 cm mean tracking error in soft continuum arms with only 100 real samples (Wang et al., 2022).
• Manipulation with Free Terminal Time: Closed-loop policies trained with marching solvers, QRnet extension, and adaptive IVP-ART sampling attain 98% success rate and 17% mean suboptimality in 14D/7D robot manipulation with free terminal time (Hu et al., 2023).

7. Tradeoffs, Open Challenges, and Future Directions

Research in closed-loop control policy learning brings to the fore several intrinsic tradeoffs and active frontiers:

• Tradeoff between nominal performance and robustness: There exists a provable tradeoff between minimizing closed-loop imitation error (nominal regret) and achieving adversarial robustness, mediated by the expressivity and regularity (e.g., Lipschitz constant) of the feedback policy (Makdah et al., 2021).
• Scalability and distributed computation: Explicit distributed learning architectures (e.g., DLPC) circumvent the computational bottlenecks of conventional DMPC in large-scale multiagent systems (Zhang et al., 2024).
• Formal correctness vs. sample-efficiency: CEGIS and similar counterexample-driven techniques guarantee safety and reachability but may require more computation or demonstration effort compared to purely data-driven approximations (Ravanbakhsh et al., 2019).
• Safe exploration and online adaptation: Integrating Lyapunov/barrier theory, probabilistic safety envelopes (via GP surrogates), and control-theoretic invariants into RL remains an active area to ensure safety during learning and deployment (Hirt et al., 2024, Hao et al., 3 Oct 2025).
• Generalization across domains and tasks: Multi-asset and domain-agnostic policy architectures have been shown to match or exceed the performance of specialist models at much lower sample cost (Nasir et al., 2022).
• Open challenges: Handling partial observability, nonlinear constraints, high-dimensional state/action spaces, and joint optimization of perception and control modules remain substantive challenges.

---

References:

• (Ahn et al., 2022): Model Predictive Control via On-Policy Imitation Learning
• (Ma et al., 7 Apr 2026): Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation
• (Amici et al., 20 May 2026): Closed-Loop Sim-to-Real Reinforcement Learning for Deformable Microfiber Shape Control
• (Wang et al., 2022): A Data-Efficient Model-Based Learning Framework for the Closed-Loop Control of Continuum Robots
• (Zhang et al., 2024): Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC
• (Macua et al., 2018): Learning Parametric Closed-Loop Policies for Markov Potential Games
• (Hao et al., 3 Oct 2025): A Control-Barrier-Function-Based Algorithm for Policy Adaptation in Reinforcement Learning
• (Ravanbakhsh et al., 2019): Formal Policy Learning from Demonstrations for Reachability Properties
• (Hu et al., 2023): Learning Free Terminal Time Optimal Closed-loop Control of Manipulators
• (Rosolia et al., 2018): Simple Policy Evaluation for Data-Rich Iterative Tasks
• (Mukherjee et al., 8 Nov 2025): Policy Gradient-Based EMT-in-the-Loop Learning to Mitigate Sub-Synchronous Control Interactions
• (Piovarci et al., 2022): Closed-Loop Control of Direct Ink Writing via Reinforcement Learning
• (Molu, 2023): Mixed $\mathcal{H}_2/\mathcal{H}_\infty$-Policy Learning Synthesis
• (Hirt et al., 2024): Safe and Stable Closed-Loop Learning for Neural-Network-Supported Model Predictive Control
• (Makdah et al., 2021): Learning Lipschitz Feedback Policies from Expert Demonstrations: Closed-Loop Guarantees, Generalization and Robustness
• (Nasir et al., 2022): Multi-Asset Closed-Loop Reservoir Management Using Deep Reinforcement Learning
• (Jodogne et al., 2011): Closed-Loop Learning of Visual Control Policies