Neural MMO: Multiagent RL Platform

• Neural MMO is a research platform that simulates persistent, procedurally generated worlds for multiagent reinforcement learning.
• It integrates MMO-style dynamics like exploration, combat, and resource management to benchmark robustness and generalization.
• Recent upgrades enhance performance and task flexibility with CleanRL, PufferLib, and goal-conditional benchmarks, advancing multiagent training.

Neural MMO is a free and open-source research platform for massively multiagent reinforcement learning, inspired by massively multiplayer online role-playing games and designed to combine large agent populations, long time horizons, procedurally generated worlds, and open-ended or task-conditioned objectives in a computationally accessible environment (Suarez et al., 2021, Suárez et al., 2023). Across its major releases, it has functioned both as a persistent-world sandbox for studying exploration, specialization, and competition, and as a benchmark for robustness, generalization, and multi-task learning under standardized APIs, baselines, and competitions (Suarez et al., 2019, Suarez, 2024).

1. Historical development and scope

The earliest Neural MMO papers framed the environment as a persistent, procedurally generated, massively multiagent game world in which every player is a learning agent. The 2019 release emphasized persistence, large and variable populations, finite resources, and combat as mechanisms for inducing ecological pressure, with survival time as the default objective (Suarez et al., 2019). The 2020 v1.3 paper extended that framing with a progression system, distributed infrastructure, and generalized input-output abstractions for complex observation and action spaces, while retaining the core design goal of MMO-like large-scale interaction in a tractable RL environment (Suarez et al., 2020).

The 2021 platform paper consolidated Neural MMO as a research platform for massively multiagent reinforcement learning. It described procedurally generated, persistent virtual worlds; configurable systems for Resource, Combat, Progression, and NPC/Equipment; and canonical configurations ranging from SmallMaps at $128 \times 128$ with a population cap of 256 agents and a training horizon of 1024 timesteps to LargeMaps at $1024 \times 1024$ with a population cap of 1024 agents and a horizon of 8192 timesteps (Suarez et al., 2021). In that formulation, Neural MMO was positioned as a step toward “foundation policies”: generalist policies trained in multimodal, cognitively realistic tasks.

Neural MMO 2.0, published in 2023, was a complete rewrite of the earlier codebase. Its central novelty was a flexible task system built around a vectorized GameState, predicates, and tasks, together with approximately three-fold higher performance and explicit compatibility with CleanRL via PufferLib (Suárez et al., 2023). The canonical 2.0 competition setting used 128 agents per world, and the environment was explicitly oriented toward generalization to unseen tasks, maps, and opponents.

2. World model, mechanics, and agent interface

In the 2021 platform description, the world is a grid map composed of tile types including grass, forest, stone, water, and lava. Forest tiles contain food resources, water pools provide renewable water, stone is impassable, and lava is lethal on contact. Agents maintain food, water, and health; food and water are consumed each tick; exhaustion of food or water causes health loss; and sufficiently high food and water permit health regeneration (Suarez et al., 2021). This induces a multiobjective survival problem in which navigation, foraging, and spatial competition are inseparable.

Combat introduces direct strategic interaction. The 2021 platform paper defines three combat styles—Melee, Range, and Mage—whose outcomes depend on offensive and defensive combat stats such as accuracy, damage, evasion, and max health (Suarez et al., 2021). Earlier versions also describe melee, ranged, and mage attacks with style-specific range and effects, including a freezing effect for mage in the 2019 and 2020 formulations (Suarez et al., 2019, Suarez et al., 2020). Progression systems link repeated interaction to long-term capability growth: foraging yields experience in resource-related skills, while combat yields experience in offense and defense skills, altering damage, accuracy, survivability, and carrying capacity over time (Suarez et al., 2021).

Later releases expanded the economic and item systems. The NeurIPS 2022 challenge on v1.6 added equipment, professions, trading, and a shrinking “blue circle” fog mechanic, with a global market in which agents could buy and sell items using gold (Liu et al., 2023). Neural MMO 2.0 describes worlds with survival, combat, professions, resources, items, and an environment-wide global market. It specifies five gathering professions—Herbalism, Fishing, Prospecting, Carving, and Alchemy—together with 7 resources and 16 item types across 10 quality levels, including weapons, armor, consumables, tools, and ammunition (Suárez et al., 2023).

Observations are locally partial rather than globally omniscient. In the 2021 platform paper, each agent receives a fixed crop of nearby tiles together with the set of other agents occupying those tiles, represented as objects with discrete and continuous attributes (Suarez et al., 2021). Formally, for agent $i$, the local observation is $o_t^i = \Omega^i(s_t)$, where $\Omega^i$ extracts a local neighborhood from the global state. Actions are structured rather than monolithic. In the 2021 formulation,

$$a_t = \big(a_t^{\text{move}}, a_t^{\text{attack}}\big),$$

with movement chosen from directions and attack decomposed into style and target selection (Suarez et al., 2021). Neural MMO 2.0 extends this with profession actions, item usage, and market actions in addition to movement and combat (Suárez et al., 2023).

3. Task systems, rewards, and formal problem formulations

The default reward structure in the 2021 platform paper is intentionally minimal: survival. For agent $i$,

$$r_t^i = \begin{cases} -1 & \text{if agent } i \text{ dies at time } t, \ 0 & \text{otherwise,} \end{cases}$$

with return

$R^i = \sum_{t=0}^{T} \gamma^t r_t^i.$

The same paper also introduced an achievement-based reward system in which agents receive points for completing gameplay milestones such as acquiring armor, defeating other players, or traversing the map, with each achievement only achievable once per episode (Suarez et al., 2021). This separated open-ended environment dynamics from specific training objectives.

Neural MMO 2.0 generalized this objective layer into three modules: GameState, Predicates, and Tasks. GameState stores environment state in vectorized or flattened tensors and includes event datastores for events such as successful hits, resource gathering, and player kills. Predicates are functions of signature predicate(gs: GameState, subject: Group, **kwargs) -> float that return values in $[0,1]$ rather than booleans, thereby encoding partial completion and dense reward. The paper reports 25 built-in predicates and supports custom predicates over individuals or groups, with tasks created from predicates and assignable per-agent, per-team, or even cross-assigned to reward one agent for another’s performance (Suárez et al., 2023). Examples such as DistanceTraveled, FullyArmed, and KillPredicate illustrate that tasks can encode navigation, equipment acquisition, and combat milestones without changing simulator mechanics.

The 2023 competition reframed Neural MMO as a goal-conditional benchmark. Each agent was assigned a task, and the policy was conditioned on a task descriptor $1024 \times 1024$0, yielding a goal-conditional policy $1024 \times 1024$1 over a multi-agent Markov game (Suárez et al., 17 Aug 2025). The competition used 1297 training tasks and 63 evaluation tasks, with low overlap between training and evaluation if predicates and arguments were considered jointly. Task definitions, expressed as code predicates, were embedded into fixed-length vectors using a LLM and then projected into the policy network, so the conditioning variable was part of the learned control stack rather than an external planner (Suárez et al., 17 Aug 2025).

4. Learning architectures and training practice

Baseline architectures have generally mirrored Neural MMO’s structured observations and actions. The 2021 platform paper used RLlib’s PPO with a single-layer LSTM of 64 hidden units, an encoder that embedded tiles and agents separately, a self-attention module over agents, a conv-pool-dense module over tiles, and a decoder that produced action-argument scores via dot-product similarity between latent state and candidate argument embeddings (Suarez et al., 2021). All agents shared policy parameters during training, so the many-agent dimension became part of the rollout batch, while behavior emerged from interaction rather than agent-specific weights.

With Neural MMO 2.0, the default training stack shifted toward CleanRL and PufferLib. The environment implemented the PettingZoo ParallelEnv API, while PufferLib supplied a vectorized multi-agent backend and wrappers that allowed standard PPO pipelines to operate on the environment with minimal environment-specific code (Suárez et al., 2023). The release paper states that individual baseline models were trainable in 8 A100 hours and that desktop-grade GPUs also sufficed, emphasizing engineering accessibility as part of the platform’s design.

The NeurIPS 2023 competition baseline used a 3.9M-parameter PPO policy with a task-embedding projection layer, structured encoders for tiles, nearby agents, items, and market offers, and pointer networks for variable-length decisions such as target and item selection. CleanRL supplied the PPO implementation, while PufferLib’s “Clean PuffeRL” handled batching and communication across subprocesses and, crucially, eliminated the heavy zero-padding that otherwise arises when many agents die early in an episode (Suárez et al., 17 Aug 2025). The winning submissions improved mainly through feature scaling, action-space pruning, larger numbers of training maps, better alignment between training and evaluation configurations, and recurrent memory such as LSTMs, rather than through radically larger models (Suárez et al., 17 Aug 2025).

Competition results reinforced that modest architectural or systems decisions can dominate nominal algorithm choice. A later summary of the 2023 competition reports that the top solution achieved a score 4x higher than the baseline within 8 hours of training on a single 4090 GPU (Suárez et al., 17 Aug 2025). This suggests that Neural MMO’s difficulty lies at least as much in environment-specific representation, reward, and systems engineering as in the choice of generic RL optimizer.

5. Neural MMO as a benchmark for robustness and generalization

Neural MMO has been used repeatedly as a competitive benchmark. The IJCAI 2022 challenge, later summarized in 2023, used 16 teams of 8 agents in procedurally generated $1024 \times 1024$2 worlds over 1024-step episodes and scored teams with an Achievement metric built from four subtasks: exploration, foraging, equipment progression, and combat (Chen et al., 2023). The benchmark was explicitly designed to test robustness to map variation and opponent variation, with PvE stages against scripted and learned baselines and a PvP track ranked by TrueSkill. The competition received 1600+ submissions, and the paper treated this scale as evidence that competitions can serve as practical benchmark-forming mechanisms for hard MARL problems (Chen et al., 2023).

The NeurIPS 2022 challenge ran on Neural MMO v1.6 and added equipment, professions, trading, and a fog-of-war “blue circle” mechanic. Its evaluation combined a defeat score, based on player eliminations, with a survival score derived from team ranking by last death time (Liu et al., 2023). The competition again used 16 teams of 8 agents on procedurally generated $1024 \times 1024$3 maps. The paper emphasized that robust policies had to generalize not only across map layouts but also across opponent styles, and it documented clear differences between policies that performed well against scripted baselines and those that remained competitive against stronger learned opponents (Liu et al., 2023).

The NeurIPS 2023 competition shifted the emphasis from survival or mixed outcome scoring to goal-conditional task completion on held-out tasks, maps, and opponents. In its reported evaluation, the organizer baseline achieved PvE and PvP task completion rates of 5.98% and 6.39%, while the co-winning submissions “Takeru” and “Yao Feng” reached 17.09% and 25.21%, and 16.56% and 24.88%, respectively (Suárez et al., 17 Aug 2025). Neural MMO therefore functions not only as a many-agent simulator but also as a controlled generalization benchmark in which the train-test gap can be induced along multiple axes simultaneously.

A recurrent methodological theme is that simple scalar reward is not always an adequate evaluation statistic in open-ended multiagent worlds. The 2021 platform paper explicitly argues that average reward is not a reliable metric because improved combat capability can make the world more dangerous, reducing lifetime even as policy competence increases (Suarez et al., 2021). It recommends multi-metric dashboards and tournament-based rating, including measures such as lifetime, exploration, equipment, kills, achievements, and TrueSkill-style skill ratings.

6. Tooling, extensions, and open problems

A major part of Neural MMO’s recent adoption has come from tooling rather than from simulator mechanics alone. PufferLib was designed to make environments, models, and learning libraries “play nice,” and Neural MMO is one of its flagship complex simulators. PufferLib wraps the environment so mainstream RL libraries see flattened Box observations and flattened MultiDiscrete actions, while an unflattening helper restores structure inside the model. It also pads and sorts multi-agent data into canonical order, supports serial, multiprocessing, and Ray vectorization backends, and implements EnvPool-style asynchronous batching (Suarez, 2024). For Neural MMO specifically, the paper reports a single-core step rate of $1024 \times 1024$4 steps per second with emulation overhead of $1024 \times 1024$5, desktop throughput of $1024 \times 1024$6 SPS under standard PufferLib vectorization and $1024 \times 1024$7 SPS under Puffer Pool, and notes that “no learning library could handle Neural MMO 2.0 natively,” while PufferLib enabled competent policies in as little as 8 hours on a single desktop (Suarez, 2024).

Meta MMO extends Neural MMO into a collection of many-agent minigames, including Survival, Team Battle, Multi-task Training/Evaluation, Protect the King, Race to the Center, King of the Hill, and Sandwich. It treats Neural MMO as a “meta-environment” with configurable subsystems, adaptive difficulty, and task embeddings, and studies whether one shared policy can act as a generalist across multiple games (Choe et al., 2024). In the reported experiments, a generalist policy matched or outperformed specialist policies on several minigames when trained with the same number of target-task samples, providing a concrete many-agent generalization benchmark layered directly on top of Neural MMO’s mechanics (Choe et al., 2024).

Several limitations recur across releases. The 2021 platform paper identifies many-team cooperation, evaluation in open-ended settings, and the fact that learned policies can lag well-designed scripted bots on some tasks as open problems (Suarez et al., 2021). The Neural MMO 2.0 paper notes that prior competitions did not fully exploit all game systems and that team-based tasks were postponed in the NeurIPS 2023 competition because of limitations in existing learning libraries (Suárez et al., 2023). PufferLib, while solving much of the interfacing problem, does not yet support continuous actions and is targeted at CPU environments rather than GPU-native simulators (Suarez, 2024). Meta MMO adds that no interactive client is provided, constraining research on human-agent interaction and mixed populations (Choe et al., 2024).

Taken together, these releases position Neural MMO as both an environment family and a research program. It is a platform for studying large-population RL under partial observability, long horizons, and mixed cooperative-competitive pressure; a benchmark for robustness to tasks, maps, and opponents; and an ecosystem in which simulator design, API design, vectorization, and reward specification are all first-order research concerns (Suarez et al., 2021, Suárez et al., 2023).