Multi-Agent AC-MPC

• MA-AC-MPC is a framework that integrates model predictive control within multi-agent actor-critic reinforcement learning to achieve safe, dynamically feasible, and robust cooperation.
• It employs a two-layer hierarchy, combining a differentiable MPC in the actor network with a centralized critic to align global objectives and manage constraints efficiently.
• Empirical evaluations show enhanced sample efficiency, significant cost reductions, and successful transfer from simulation to hardware in applications like drone collaboration and battery management.

Multi-Agent Actor-Critic Model Predictive Control (MA-AC-MPC) combines model predictive control with multi-agent actor-critic reinforcement learning to address the optimal control problem in cooperative multi-agent systems. This methodology embeds a differentiable, short-horizon MPC layer within each agent's policy network and leverages a centralized critic for global objective alignment. The approach is motivated by the need for safe, dynamically feasible, and robust policy generation under uncertainty, sparse rewards, and stringent constraints in settings where agents must cooperate to maximize collective outcomes (Llanes et al., 4 Jun 2026, Kordabad et al., 2021).

1. Multi-Agent Problem Formulation

MA-AC-MPC is defined for a team of $K$ agents sharing an environment with stochastic dynamics. The global Markovian state at each time $t$ is $s_t\in\mathcal{S}$, with agent-specific observations $o_t^{(i)}$, and each agent’s policy $\pi^{(i)}(a^{(i)}|o^{(i)})$ prescribes actions $a_t^{(i)}\in\mathcal{A}^{(i)}$. State transitions follow: $$s_{t+1} \sim P(s_{t+1}|s_t, a_t^{(1)},\dots,a_t^{(K)})$$ Agents optimize a shared discounted reward: $$J(\Pi) = \mathbb{E}_{\Pi}\left[ \sum_{t=0}^\infty \gamma^t R(s_t, a_t^1, \dots, a_t^K, s_{t+1}) \right]$$ with $\gamma\in[0,1)$. Each agent may obey individual dynamics: $x_{t+1}^{(i)} = f_i(x_t^{(i)}, u_t^{(i)})$ with constraints $t$0, $t$1. Coupling between agents can be realized through global constraints or shared reward structure (Llanes et al., 4 Jun 2026, Kordabad et al., 2021).

2. Actor–Critic and Embedded Model Predictive Control

The core architecture is a two-layer hierarchy:

• Actor (High-Level Planner): The actor network $t$2 maps each observation $t$3 to MPC parameters $t$4, encoding references and penalty weights for cooperative objectives.
• Differentiable MPC (Low-Level Controller): For each agent and timestep, an MPC problem is solved:

$t$5

subject to system dynamics and constraints, typically quadratic stage/terminal cost forms and linear/nonlinear dynamics. The first control $t$6 is executed. Gradients $t$7 are propagated back to update the actor via policy optimization.

• Centralized Critic: The critic $t$8 accesses the full global state during centralized training to estimate expected returns and shape policy gradients (e.g., via MAPPO-style objectives) (Llanes et al., 4 Jun 2026, Kordabad et al., 2021).

3. Learning and Optimization Procedure

MA-AC-MPC training follows centralized training/decentralized execution (CTDE). The core procedure is:

1. Rollout Generation: For each timestep, each agent $t$9 receives $s_t\in\mathcal{S}$0, computes $s_t\in\mathcal{S}$1, solves the differentiable MPC to produce $s_t\in\mathcal{S}$2, executes the joint action, and records transitions $s_t\in\mathcal{S}$3.
2. Critic Update: The critic $s_t\in\mathcal{S}$4 is trained to estimate the return using collected rollouts and a clipped value loss.
3. Policy Update: Actors are updated via policy gradients computed with a surrogate PPO objective, leveraging the differentiable MPC layer for end-to-end policy learning.
4. Curriculum Learning: Progressive environmental complexity is introduced to facilitate robust policy emergence (Llanes et al., 4 Jun 2026).

During deployment, only the actor and internal MPC are used per agent, with no centralized access.

4. Constraint Handling and Robustness Mechanisms

MA-AC-MPC enforces constraints at multiple levels:

• State and Input Constraints: Directly imposed within the MPC optimization, e.g., via box constraints on controls and soft or hard constraints on states.
• Coupling/Global Constraints: In purely physical settings (e.g., battery storage, grid peak constraints), hard coupling is imposed within the MPC problem (Kordabad et al., 2021); in pursuit-evasion and teaming, agent coupling is managed via centralized critics and shared observation vectors (Llanes et al., 4 Jun 2026).
• Slack Variables and Penalties: Soft state constraints are managed using slack variables with large penalties to preserve feasibility and stability (Kordabad et al., 2021).
• Robustness to Parametric Variation: Curriculum learning, parameter randomization (e.g., mass/inertial disturbances), and actor regularization contribute to robust policy transfer from simulation to hardware (Llanes et al., 4 Jun 2026).

5. Empirical Evaluation and Performance

Extensive benchmarking of MA-AC-MPC includes tasks in both energy systems and robotic collaboration:

Task	MA-AC-MPC Performance	MLP-based Baseline
Pursuit-Evasion (2v2, sim/hardware)	>90% win-rate in ~1.2M steps; 100% hardware success	Requires ~4M steps; lower hardware success
Drone Landing on Rover (heterogeneous)	5/5 hardware trials, mean error 0.055 m	3/5 hardware, mean error 0.240 m

In battery storage management, MA-AC-MPC reduces closed-loop cost by over 30% from its initial value, enforces strict peak-power constraints, and converges key parameters steadily. Policies evolve from boundary-violating to consistently feasible behaviors, and show convergence to bang–bang optimal structures in linear stochastic settings (Kordabad et al., 2021). In robotics, it achieves higher sample efficiency, better transfer to hardware, and increased robustness against unmodeled physical variations compared to neural-network-only actor baselines (Llanes et al., 4 Jun 2026).

6. Practical Implementation Considerations

Practical deployments utilize:

• Simulation Environments: JAX-based Crazyflie sim, MuJoCo for hardware-in-the-loop, and "first_principles" nonlinear modeling to match real-world device behavior.
• MPC Solvers: acados (real-time NMPC via leap–c), typically running at ≥50 Hz, enforcing both box and soft constraints with low inference latency ($s_t\in\mathcal{S}$51 ms per agent with MPC) (Llanes et al., 4 Jun 2026).
• Actor–Critic Networks: Two-layer MLPs, with distributed MAPPO for MARL and well-characterized PPO hyperparameters.
• Batch Learning: Episode buffers of length 256, mini-batch optimization, multiple training epochs per batch.
• Resource Usage: Substantially longer wall clock time per step for MPC variants versus MLP (e.g., $s_t\in\mathcal{S}$6162 h/2M steps on CPU vs 18–30 h/4M for MLP) (Llanes et al., 4 Jun 2026).

7. Significance, Limitations, and Research Directions

MA-AC-MPC presently demonstrates:

• High sample efficiency and increased robustness, e.g., consistent outperformance of deep neural actor baselines across environmental variations and real-system deployment.
• Effective handling of constraints, dynamic feasibility, and safety-critical control via embedded optimization layers in RL frameworks.
• Success in complex heterogenous teams (drone-rover collaboration), with transferability from simulation to hardware documented for both homogeneous and heterogeneous settings (Llanes et al., 4 Jun 2026, Kordabad et al., 2021).

This suggests that embedding short-horizon MPC inside policy networks can significantly improve generalization and safety in multi-agent RL control, albeit at higher compute expense due to embedded optimization. A plausible implication is that further advances in differentiable optimization and distributed RL frameworks may extend MA-AC-MPC’s applicability to even larger-scale and more complex cooperative settings involving both hard coupling and decentralized information flow.