Merging model-based control with multi-agent reinforcement learning for multi-agent cooperative teaming strategies

Abstract: In this work, we propose a framework that combines multi-agent reinforcement learning (MARL) with model-based control to achieve safe, dynamically feasible actions in cooperative multi-agent tasks. Multi-agent reinforcement learning provides the advantage of learning cooperative policies for multi-agent teams from discrete non-differentiable rewards in a long planning horizon. Model-predictive control is robust and offers safe, dynamically feasible actions in a fast replanning framework for short horizons. We propose an algorithm that extends actor-critic model predictive control for MARL which we refer to as multi-agent actor-critic model predictive control (MA-AC-MPC). We demonstrate the capabilities of this algorithm by applying it to a multi-agent pursuit-evasion scenario. Specifically, we compare the evader team's strategy using the MA-AC-MPC model and a multi-layer perceptron model (MA-AC-MLP). The pursuer team uses augmented proportional navigation as it is accepted as an advanced adversarial control law. We also provide an example with a heterogeneous environment where a drone and omni-wheeled rover cooperate to achieve repeatable and successful landing with 100% success rate in hardware for MA-AC-MPC compared to 60% for MA-AC-MLP. We demonstrate the robustness of the proposed MA-AC-MPC algorithm in hardware for both environments.

• The paper presents MA-AC-MPC, a novel framework that integrates model predictive control with multi-agent reinforcement learning for enhanced cooperative control in robotics.
• It decouples high-level neural planning from low-level trajectory optimization using differentiable MPC solvers to ensure constraint satisfaction and robust performance.
• Experimental results show superior sample efficiency, a 100% hardware landing success rate, and strong sim-to-real consistency compared to traditional MLP-based policies.

A Framework for Multi-Agent Reinforcement Learning with Embedded Model Predictive Control

Introduction

This paper presents a structured fusion of Model Predictive Control (MPC) and Multi-Agent Reinforcement Learning (MARL) tailored for real-world cooperative robotic tasks. The proposed algorithm, Multi-Agent Actor-Critic Model Predictive Control (MA-AC-MPC), extends the AC-MPC framework to the multi-agent domain, combining the policy optimization flexibility of RL with the robustness and constraint satisfaction of MPC. The approach is validated in two representative settings: a 3D pursuit-evasion task and a heterogeneous drone-rover landing scenario, revealing significant empirical advantages over pure neural MLP-based policies (MA-AC-MLP), especially in terms of transferability, robustness to dynamics mismatch, and hardware performance.

Figure 1: Homogeneous (2v2 pursuit-evasion) and heterogeneous (drone-rover) environments with MA-AC-MPC in MuJoCo.

Background and Related Work

MARL provides a framework for learning decentralized or centralized policies for cooperative or competitive teams under complex rewards and partial observability, typically modeled as Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs). While MLP-based policies can be effective in simulation, their deployment in hardware is fraught with issues such as dynamics mismatch and lack of constraint satisfaction. Recent literature on integrating MPC in RL strategies (e.g., differentiable MPC layers, policy cost tuning, or critic guidance) is predominantly single-agent oriented or non-end-to-end in the multi-agent setting, resulting in suboptimal cooperation and sample inefficiency. The proposed MA-AC-MPC closes this gap by enabling true end-to-end gradient-based training with differentiable, constraint-aware MPC actors for each agent.

Figure 2: MA-AC-MPC architecture—neural network outputs MPC cost parameters, MPC layer computes real-time feasible actions per agent.

MA-AC-MPC Methodology

In MA-AC-MPC, each agent's policy comprises a neural network that generates MPC cost parameters (stage/terminal reference and weighting), which are then fed to a differentiable, real-time-capable nonlinear MPC backend (via leap-c with acados). During centralized training, a global critic function conditions the learning signal across all agents to maximize collective rewards. Observations passed to the actor contain partial inter-agent state to foster decentralized cooperation, using mechanisms such as parameter sharing, agent ID encoding, and death-masking for non-active agents.

Crucially, this hierarchy decouples high-level planning (neural policy) from low-level trajectory optimization (MPC), enabling policies to generalize across dynamic regimes and unmodeled perturbations.

Experimental Evaluation: Pursuit-Evasion

The first benchmark is a 2 vs. 2 pursuit-evasion scenario with delayed rewards and hard collision constraints. The evaders are homogeneous (shared policy), and the pursuers use a fixed, high-performance proportional navigation controller. MA-AC-MPC is compared against multiple fully-connected MA-AC-MLP baselines under identical curriculum regimes and reward structures.

Figure 3: Training dynamics for reward, win rate, and curriculum level for MA-AC-MPC and MLP baselines.

MA-AC-MPC achieves higher mean total reward and attains required curriculum advancement thresholds with fewer environment steps. Even with a minimal MPC prediction horizon ($N=2$), policy sample efficiency is markedly superior due to explicit handling of agent dynamics in the MPC actor.

Robustness to significant dynamics perturbations is quantified via a mass-sweep experiment where evaluation mass is swept over $\pm$10 g from the nominal value (inducing up to 25% mismatch), breaking the simulation-to-execution match for MLPs but not for MA-AC-MPC.

Figure 4: Evader win rates vs. mass perturbation for both MA-AC-MPC and MLP; MA-AC-MPC consistently outperforms.

No scenarios were detected where MLPs outperformed MA-AC-MPC, and the latter's win rate difference is strictly positive across the grid. Hardware transfer is demonstrated on real Crazyflie drones through ROS2/Crazyswarm2 integration. The observed trajectories across simulation (Crazyflow), high-fidelity sim (CrazySim/MuJoCo), and physical hardware are highly consistent for MA-AC-MPC, affirming effective sim2real transfer and dynamics robustness.

Figure 5: Hardware, Crazyflow, and CrazySim trajectory overlays highlight sim-to-real consistency.

Experimental Evaluation: Heterogeneous Drone-Rover Landing

The second benchmark is a cooperative landing task involving a drone and an omni-directional rover (Yahboom X3) executing independent policies. Each actor network feeds tailored references to its corresponding agent's MPC solver (drone: multirotor, rover: mecanum drive). The environment employs dense team rewards, strict safety constraints, and needs fine-grained trajectory precision.

Curriculum learning is used for stabilization, incrementally increasing initial state perturbations, maximum rover velocity, and dynamic uncertainty.

Figure 6: Close-up of Crazyflie drone on ROSMASTER X3 rover.

During training, MLP policies initially achieve marginally higher sample efficiency, as judged by simulated landing rates; however, MA-AC-MPC rapidly overtakes as scenarios become less ideal (i.e., under domain randomization and out-of-distribution deployment).

Figure 7: Landing success rate during training for both methods across curriculum; MA-AC-MPC overtakes with increased complexity.

The critical distinction emerges in hardware: MA-AC-MPC policies exhibit 100% landing success rate in all hardware trials versus 60% for MA-AC-MLP. The successful landings of MA-AC-MPC are characterized by consistently low position error (mean $<$ 6 cm), tight endpoint variance, and no catastrophic failures; MLP policies frequently miss or overshoot the moving pad.

Figure 8: 3D hardware trajectories—MA-AC-MPC always converges precisely, MLP exhibits greater variability and two major failures.

Implications and Future Directions

MA-AC-MPC demonstrates clear advantages in real-world multi-agent collaborative tasks where physics constraints, safety, and robust generalization to domain/system mismatch are paramount. Its decomposition—neural planner for high-level task inference and differentiable MPC for physical feasibility—offers an interpretable, sample-efficient, and hardware-compliant alternative to end-to-end black-box MARL architectures.

Implications for the design of multi-agent robotic systems include:

• Strong theoretical and practical incentive for integrating explicit short-horizon model-based control within RL agents, particularly via hierarchical/cost-parameterized policies.
• Systematic robustness to large unmodeled disturbances and parameter mismatches (i.e., dynamics transfer invariance).
• Scalable potential for deployment in more complex, larger-scale teams, given the per-agent decentralized MPC structure.
• Provide a foundation for more formal convergence, safety, and stability results in decentralized cooperative MARL with real-time constraints.

Future extensions should consider longer horizons and improved parallelization (e.g., GPU/MPC solver acceleration), adaptive online model identification for unmodeled/heterogeneous agents, and applications to adversarial or partially observable scenarios.

Conclusion

MA-AC-MPC substantively outperforms MLP-based policies for multi-agent cooperative control in both homogeneous and heterogeneous tasks. It provides strong robustness to agent-specific perturbations, excellent sample and generalization efficiency, and substantial improvements in sim-to-real transfer, especially in hardware. The architecture and empirical performance motivate broader adoption of hybrid actor structures—neural/planner plus model-based optimizer—in multi-agent and safety-critical robotics.