MuJoCo-Drones-Gym: A GPU-Accelerated Multi-Drone Simulator for Control and Reinforcement Learning

Abstract: Robotic simulators are a cornerstone of modern research in aerial robotics, serving both as a vehicle for the development of new control algorithms and as the data source for training reinforcement learning (RL) policies. Yet, existing quadcopter learning environments often face a trade-off between physical fidelity, multi-agent support, and the throughput required by modern deep RL pipelines. In this paper, we present MuJoCo-Drones-Gym, an open-source Gymnasium-compatible multi-drone environment built on top of the MuJoCo physics engine. MuJoCo-Drones-Gym supports an arbitrary number of Bitcraze Crazyflie 2.x nano-quadcopters and exposes a modular API for selecting (i)~the physics model (rigid-body MuJoCo, explicit Python dynamics, or any subset of ground effect, blade drag, and inter-drone downwash), (ii)~the action interface (per-motor RPMs, collective normalized thrust, velocity setpoints, or PID waypoint commands), and (iii)~the observation space (kinematic state vectors, RGB / depth / segmentation cameras, or neighbourhood adjacency information). A PettingZoo ParallelEnv wrapper enables drop-in multi-agent reinforcement learning, while a suite of seven task environments, hover, velocity tracking, multi-drone hover, waypoint navigation, formation flight, gate racing, and a generic multi-agent template, demonstrates the breadth of the interface. We describe the environment design, the underlying physics and quadcopter dynamics, and illustrate its use through control and learning examples that mirror those of the closely related gym-pybullet-drones project, while taking advantage of MuJoCo's improved contact handling, rendering, and parallelizability.

• The paper introduces a high-fidelity, GPU-accelerated multi-drone simulator using MuJoCo for enhanced control and reinforcement learning.
• It features a modular design with configurable physics, aerodynamic effects, and versatile controllers that facilitate robust sim-to-real policy transfer.
• Empirical results using PID and SE(3) controllers demonstrate sub-centimeter tracking precision, validating its effectiveness for advanced aerial robotics tasks.

MuJoCo-Drones-Gym: Architecture, Capabilities, and Implications for Multi-Drone RL

Motivation and Context

MuJoCo-Drones-Gym addresses a critical gap in the aerial robotics ecosystem, providing a modern, GPU-accelerated simulation platform for reinforcement learning (RL) and classical control in multi-drone contexts. It evolves from the widely adopted gym-pybullet-drones environment by leveraging the MuJoCo physics engine's superior integrator quality, contact handling, and GPU vectorization. Key priorities include physical fidelity, batch throughput for RL, and modularity in control interfaces—all requirements for advanced RL research and sim-to-real transfer.

Modular Design and Physics Models

The simulator is built around a highly modular architecture. It natively supports arbitrary numbers of Bitcraze Crazyflie 2.x quadcopters and provides a selectable physics stack, featuring six distinct modes:

• Pure MuJoCo rigid-body dynamics.
• Explicit integration via Python (for analytical validation).
• Toggleable aerodynamic effects: ground effect, blade drag, and inter-drone downwash.

Motor models and aerodynamic interactions are parameterized, supporting X- and $+$-configurations, and a 250g racing drone. Rotor forces are calculated using quadratic motor-speed models, with aerodynamic effects applied selectively, facilitating accurate emulation of real quadcopter behaviors.

Figure 1: PID-controlled Bitcraze Crazyflie 2.x hover demonstration in MJ-drones-gym.

Figure 2: Downwash interaction between two stacked Crazyflies, where the lower drone compensates for the aerodynamic wake.

Environments, Tasks, and Wrappers

MuJoCo-Drones-Gym ships with seven task environments, spanning hover, velocity tracking, multi-drone hover, waypoint navigation, formation flight, gate racing, and generic multi-agent scenarios. Each task exposes configurable observation and action spaces. Notable features include:

• Five distinct action interfaces (RPM, normalized thrust, velocity setpoints, PID commands, and attitude control).
• Composable wrappers for wind (CONSTANT, GUST, DRYDEN, SINUSOIDAL, and COMBINED), procedural obstacles (e.g., FOREST, URBAN, RANDOM, GATES), metric-driven curriculum, and domain randomization.

Figure 3: Single-drone hover at $z = 1$~m under combined wind and turbulence, demonstrating steady-state drift.

Figure 4: Procedurally generated obstacle scenes: FOREST (left) and RANDOM (right).

Figure 5: Multiple instantiations of domain randomization demonstrating variability across physical and sensor parameters.

The domain randomization wrapper exposes per-episode variations in mass, inertia, aero coefficients, motor lag, sensor biases, and environmental conditions. This is essential for robust sim-to-real policy transfer, preventing overfitting to a narrow simulation domain.

Multi-Agent Support and GPU Vectorization

A critical advancement is full multi-agent RL support via PettingZoo's ParallelEnv API, enabling scalable MARL experiments. Additionally, the MJXVectorAviary backend supports batched simulation of thousands of environments on a single GPU using JAX and MuJoCo MJX, vastly increasing RL throughput—especially for modern population-based and JAX-native pipelines.

The vectorized interface exposes the Gymnasium contract for drop-in compatibility with Stable-Baselines3 and other numpy-based RL frameworks. Its flexibility allows for pure JAX workflows (eliminating host-device traffic) or conventional numpy-based approaches.

Figure 6: MultiHoverAviary / FormationAviary-style multi-drone scene.

Figure 7: MultiAgentAviary benchmark with 12 drones executing antipodal goal navigation, avoiding collisions through learned or filtered control.

Controllers and Numerical Results

MuJoCo-Drones-Gym integrates cascaded PID and DSL-style PID controllers as both baselines and inner loops for higher-level action interfaces. Empirical results demonstrate robust hover, waypoint tracking, and disturbance rejection:

• PID gains tuned for MuJoCo dynamics achieve stable hover with sub-centimeter drift (see Figure 1).
• The SE(3) geometric controller achieves aggressive trajectory tracking (vertical circles) with mean error $0.7$~cm and peak $3.5$~cm, qualitatively outperforming pure PID approaches.

Figure 8: Vertical circle trajectory executed by an SE(3) geometric controller with high tracking precision.

Practical and Theoretical Implications

Practically, MuJoCo-Drones-Gym offers an advanced platform for RL researchers targeting aerial robotics, MARL, and sim-to-real transfer. The modularity via wrappers and domain randomization promotes robust policy learning, supporting curriculum-based and disturbance-aware training. The GPU-accelerated backend unlocks RL throughput scaling for large populations and expensive tasks.

Theoretically, the simulator enables investigation into multi-agent coordination, aerodynamics-aware control, and robustness to real-world phenomena not captured by earlier simulators. The mirror API relative to gym-pybullet-drones ensures minimum friction for legacy controllers and policies, simplifying longitudinal studies across simulator generations.

The bold claim that controllers and policies port across both simulation platforms with minimal adaptation is supported by identical aerodynamic models and parameter conventions. Furthermore, the omission of aerodynamic effects in MJXVectorAviary backend—motivated by XLA compilability—highlights ongoing challenges in unifying physical fidelity and computational tractability within GPU-native RL pipelines.

Future Directions

Future development priorities include:

• Extending MJXVectorAviary to support full aerodynamic models and vision-based observations.
• Enabling end-to-end JAX-native training and sim-to-real transfer benchmarks with empirical flight log calibration.
• Adding richer perception modalities (e.g., stereo, event cameras) for vision-based quadrotor RL.
• Open-sourcing reference policies and pretrained checkpoints for reproducibility.

Conclusion

MuJoCo-Drones-Gym delivers a high-fidelity, GPU-accelerated, and modular simulation platform for multi-drone RL and control research. Its architecture, task suite, and compatibility with contemporary RL tooling position it as a practical and extensible benchmark for aerial robotics. The simulator's strong claims regarding policy portability, throughput, and fidelity are substantiated by empirical examples, making it a valuable foundation for future studies in scalable multi-drone learning and robust sim-to-real transfer.