CrazyMARL: Multi-Agent RL for Aerial Robotics

• CrazyMARL is a family of decentralized multi-agent reinforcement learning systems designed for cooperative aerial robotics using shared policy structures and CTDE strategies.
• One variant employs a PPO-based framework for collision-free, centimeter-accurate landing of drone swarms on static and moving platforms.
• Another variant uses decentralized direct motor-control policies to enable cooperative transport of cable-suspended payloads with improved recovery over traditional baselines.

Searching arXiv for CrazyMARL-related papers to ground the article in current literature. CrazyMARL is the name used for two closely related multi-agent reinforcement learning systems for cooperative aerial robotics, both implemented on Crazyflie platforms but targeting different control problems. In one usage, CrazyMARL denotes a Proximal Policy Optimization-based framework for collision-free, centimeter-accurate landing of drone swarms on static and moving platforms (Aschu et al., 2024). In another, it denotes a decentralized direct motor-control policy for cooperative transport of cable-suspended payloads, explicitly modeling slack–taut cable transitions and emphasizing zero-shot sim-to-real transfer under disturbances (Lorentz et al., 17 Sep 2025). Across both usages, the common thread is decentralized execution with shared policy structure, deployment on resource-constrained Crazyflie hardware, and an attempt to replace or reduce reliance on analytical centralized control.

1. Terminological scope and research context

The term CrazyMARL does not designate a single invariant algorithmic artifact; rather, it identifies two research systems built around Crazyflie quadrotors and multi-agent RL.

Paper	Task	Core setup
"MARLander: A Local Path Planning for Drone Swarms using Multiagent Deep Reinforcement Learning" (Aschu et al., 2024)	Precise landing of a drone swarm at relocated target locations	PPO, CTDE, parameter sharing, continuous velocity commands
"CrazyMARL: Decentralized Direct Motor Control Policies for Cooperative Aerial Transport of Cable-Suspended Payloads" (Lorentz et al., 17 Sep 2025)	Cooperative transport of cable-suspended payloads	decentralized parameter-shared IPPO under CTDE, direct normalized motor commands

The first system is presented as a local path-planning and landing method for drone swarms trained in a realistic simulated environment and deployed with Crazyflie drones and a Vicon indoor localization system (Aschu et al., 2024). The second system is formulated as a Dec-POMDP for teams of quadrotors transporting a point-mass payload via massless cables of fixed rest-length $L$, with the dynamics simulated in MuJoCo using tendons (Lorentz et al., 17 Sep 2025).

This suggests that CrazyMARL is best understood as a research lineage or naming convention rather than a single standardized framework. A plausible implication is that the name identifies a family of decentralized or CTDE MARL controllers for aerial multi-robot coordination on Crazyflie-class platforms.

2. Swarm landing formulation on Crazyflie hardware

In the landing setting, CrazyMARL is built atop Proximal Policy Optimization in a centralized-training, decentralized-execution paradigm with parameter sharing (Aschu et al., 2024). A single neural network hosts both an actor head, representing the policy $\pi_\theta$, and a critic head, representing the value function $V_\theta$, trained jointly. During training, the centralized critic receives the concatenated observations of all $N$ agents, while at execution each agent queries only its own policy. No explicit message passing is required at runtime; coordination is described as emerging through shared rewards and the centralized critic.

The shared actor-critic architecture takes as input a flattened vector of length $4N$ for the experimental case $N=2$, comprising each agent’s local state. The hidden layers are fully connected with ReLU activations and sizes $512$, $256$, and $128$. The actor head predicts $N\times 3$ continuous velocity commands, $\pi_\theta$0, while the critic head emits a scalar value estimate $\pi_\theta$1 (Aschu et al., 2024).

Each Crazyflie agent $\pi_\theta$2 at time $\pi_\theta$3 observes

$\pi_\theta$4

where $\pi_\theta$5 is the 3D position of drone $\pi_\theta$6 relative to its assigned landing target, $\pi_\theta$7 is the unit quaternion orientation, $\pi_\theta$8 is the linear velocity in the inertial frame, and $\pi_\theta$9 is the angular velocity. Neighbor information is implicitly encoded through the centralized critic, which sees all $V_\theta$0 for $V_\theta$1; no additional message passing occurs during execution. The action for each agent is a continuous velocity set-point,

$V_\theta$2

The simulation environment is a custom multi-agent Gym-style environment in PyBullet. The workspace is a cube $V_\theta$3 m, with agents and platform spawned uniformly at random at each episode start. Maximum commanded velocity is clipped at $V_\theta$4 m/s. No explicit sensor or dynamics noise is injected, but the PyBullet solver is described as replicating aerodynamic drag and ground effects. In moving-platform episodes, the UR10-equipped platform follows a linear trajectory with speed sampled in $V_\theta$5 m/s (Aschu et al., 2024).

3. Reward design, optimization, and coordination in the landing system

The landing system uses a composite per-step reward designed to encourage precise, smooth, and collision-free landings (Aschu et al., 2024):

$V_\theta$6

with

$V_\theta$7

for each agent, and

$V_\theta$8

The terminal bonus $V_\theta$9 is awarded only when all $N$0 agents are simultaneously within a small radius $N$1 of their targets. The distance-difference term shapes approach versus retreat, the velocity penalty favors smooth descents, and the hard negative term discourages inter-drone collisions.

Training is reported for $N$2 M time-steps using PPO with discount factor $N$3, GAE $N$4, learning rate $N$5, clip range $N$6, value loss coefficient $N$7, entropy coefficient $N$8, batch size $N$9, and $4N$0 epochs per update. The combined PPO loss at update $4N$1 is given as

$4N$2

where

$4N$3

Coordination is attributed to shared network parameters and the centralized critic during training rather than to explicit inter-agent communication. Because agents do not exchange real-time messages, inference is stated to scale in $4N$4. The only quadratic term in computation is within the reward penalty for collision checks, which can be pruned via locality by considering only near neighbors (Aschu et al., 2024). This suggests a particular conception of decentralization: coordination is learned implicitly through training-time access to joint state and a shared reward, while deployment remains communication-free.

4. Cooperative cable-suspended payload transport

In the payload-transport setting, CrazyMARL is formulated for $4N$5 quadrotors jointly transporting a point-mass payload $4N$6 of mass $4N$7 via massless cables of fixed rest-length $4N$8 (Lorentz et al., 17 Sep 2025). The state space at time $4N$9 is

$N=2$0

Each agent $N=2$1 takes action $N=2$2 based on its local observation $N=2$3, and the agents share a common reward $N=2$4 and discount $N=2$5.

The cable dynamics are expressed through a hybrid slack–taut law. Cable length is

$N=2$6

and tension $N=2$7 along

$N=2$8

obeys

$N=2$9

Thus $512$0 implies a taut cable, while $512$1 implies slack and $512$2. The payload equations of motion are

$512$3

and each UAV satisfies

$512$4

The paper notes that exact kinematics are handled by MuJoCo’s physics engine, but that these expressions capture the slack/taut switch.

The RL framework is described as decentralized, parameter-shared IPPO under a CTDE paradigm, with no runtime communication required. The local observation for agent $512$5 is

$512$6

Actions are mapped from $512$7 to normalized thrusts $512$8, then to commanded thrust $512$9. A first-order motor lag is included per rotor:

$256$0

$256$1

$256$2

The actor is an MLP with layers $256$3 and tanh activations, with Gaussian mean head and learned log-standard deviation. The critic is an MLP $256$4 with tanh and scalar output $256$5. Total trainable parameters are reported as on the order of $256$6–$256$7 k for the actor and about $256$8 k for the critic, with exact counts depending on input dimensions (Lorentz et al., 17 Sep 2025).

5. Learning objective, randomization strategy, and robustness in transport

The transport system uses Independent Proximal Policy Optimization with shared parameters (Lorentz et al., 17 Sep 2025). For each agent,

$256$9

where

$128$0

with $128$1. The value loss is

$128$2

and an entropy bonus $128$3 is included. Updates use $128$4 PPO epochs per batch, $128$5 minibatches, and GAE with $128$6.

The reward is factorized as

$128$7

The tracking term is

$128$8

with

$128$9

and

$N\times 3$0

where

$N\times 3$1

The stability term is

$N\times 3$2

with

$N\times 3$3

$N\times 3$4

$N\times 3$5

$N\times 3$6

and

$N\times 3$7

The safety term is

$N\times 3$8

where $N\times 3$9 with $\pi_\theta$00, $\pi_\theta$01 with $\pi_\theta$02, and

$\pi_\theta$03

with

$\pi_\theta$04

The energy term is

$\pi_\theta$05

and for $\pi_\theta$06, $\pi_\theta$07 is the average over $\pi_\theta$08 of

$\pi_\theta$09

with $\pi_\theta$10 m and $\pi_\theta$11 m.

Training uses MuJoCo at $\pi_\theta$12 Hz with $\pi_\theta$13 s and MuJoCo tendons modeling hybrid cable slack–taut transitions. Domain randomization includes randomized initial states; actuator model randomization with per-quad base thrust sampled from $\pi_\theta$14 N plus $\pi_\theta$15 N and clipped to $\pi_\theta$16 N; $\pi_\theta$17 s; observation noise with $\pi_\theta$18, $\pi_\theta$19; and disturbances comprising random forces and torques on UAVs, random forces on the payload, and occasional RPM jumps. RL hyperparameters include $\pi_\theta$20 environments, rollout length $\pi_\theta$21, total environment steps $\pi_\theta$22, learning rate $\pi_\theta$23, gradient-norm clip $\pi_\theta$24, entropy coefficient $\pi_\theta$25, value-loss coefficient $\pi_\theta$26, discount $\pi_\theta$27, and episode length $\pi_\theta$28 steps, approximately $\pi_\theta$29 s (Lorentz et al., 17 Sep 2025).

6. Empirical results, transfer, and limitations

For the landing application, CrazyMARL was deployed on two Bitcraze Crazyflie 2.1 drones in a Vicon motion-capture arena (Aschu et al., 2024). Vicon updates at $\pi_\theta$30 Hz provide position and orientation, while velocities are estimated via a first-order filter. Control commands are throttled at $\pi_\theta$31 Hz and clipped to $\pi_\theta$32 m/s. Safety thresholds trigger an emergency hover when horizontal speed exceeds $\pi_\theta$33 m/s or altitude falls below $\pi_\theta$34. Landing pads are described as $\pi_\theta$35 m cylinders on the UR10 TCP and are instrumented with AprilTags for fail-safe vision fallback.

Quantitatively, the landing system is evaluated across static-platform trials $\pi_\theta$36 and moving-platform trials $\pi_\theta$37. For static platforms, CrazyMARL reports mean Euclidean landing error $\pi_\theta$38 cm, standard deviation $\pi_\theta$39 cm, and success rate $\pi_\theta$40, whereas the PID+APF baseline reports $\pi_\theta$41 cm, $\pi_\theta$42 cm, and success rate $\pi_\theta$43. For moving platforms, CrazyMARL reports $\pi_\theta$44 cm, $\pi_\theta$45 cm, and success rate $\pi_\theta$46, while PID+APF reports $\pi_\theta$47 cm, $\pi_\theta$48 cm, and success rate $\pi_\theta$49. A two-sample $\pi_\theta$50-test on static errors yields $\pi_\theta$51, $\pi_\theta$52 (Aschu et al., 2024). The source text summarizes these results by stating that CrazyMARL halves landing error and raises success rates by approximately $\pi_\theta$53 over PID+APF while maintaining sub-$\pi_\theta$54 cm accuracy on moving targets.

For payload transport, the main baseline comparison is a two-UAV setting with $\pi_\theta$55 trials and harsh initializations (Lorentz et al., 17 Sep 2025). CrazyMARL reports success $\pi_\theta$56 and mean payload speed $\pi_\theta$57 m/s, while a rigid-rod centralized baseline reports success $\pi_\theta$58 and speed $\pi_\theta$59 m/s. The source states that the RL policy recovers more reliably and quickly, whereas the baseline often spirals under swing. In generalization sweeps with two UAVs, the paper reports a sharp drop in success for cable lengths $\pi_\theta$60 m, improved success for moderate increases in the range $\pi_\theta$61–$\pi_\theta$62 m, degradation for $\pi_\theta$63 m, a robust plateau over payload mass variation with moderate degradation for extreme light or heavy payloads, tolerance to observation noise up to approximately $\pi_\theta$64, and insensitivity to random seed with all runs within $\pi_\theta$65.

Scalability results in the transport setting report $\pi_\theta$66 success and approximately $\pi_\theta$67 s settling for $\pi_\theta$68, $\pi_\theta$69 success for $\pi_\theta$70, $\pi_\theta$71 success for $\pi_\theta$72 with an added $\pi_\theta$73 g payload and coordinated load-sharing, and near $\pi_\theta$74 for $\pi_\theta$75 with an added $\pi_\theta$76 g payload, attributed to breakdown due to peer-ordering sensitivity (Lorentz et al., 17 Sep 2025). Zero-shot sim-to-real transfer is demonstrated on Crazyflie 2.1 hardware with STM32F405, onboard EKF and motion capture, a TFLite policy running at $\pi_\theta$77 Hz, and direct PWM output. Reported trials include single-UAV flight with and without payload, two-UAV cooperative transport, and operation in $\pi_\theta$78 m/s average wind, with outcomes including stable auto-takeoff, rapid disturbance rejection under pushes and wind, and payload figure-8 tracking under gusts.

The two CrazyMARL systems converge on several common implications. Both emphasize decentralized execution without runtime communication, both rely on shared policy structure, and both use simulation environments intended to preserve transferability to real Crazyflie hardware. At the same time, they expose distinct limitations. In the landing case, the source argues for extensibility from two agents to tens of Crazyflies through parameter sharing and linear inference cost (Aschu et al., 2024); this is a stated implication rather than an experimentally demonstrated scaling result. In the transport case, the source explicitly identifies permutation-sensitive peer-state encoding as a current limitation and points to order-invariant or attention-based peer encoders, centralized critics such as MAPPO for larger teams, and integrated obstacle avoidance for cluttered environments as promising extensions (Lorentz et al., 17 Sep 2025).