CoGrid Multi-Agent Simulation

• CoGrid is a multi-agent, grid-based simulation library that supports both AI training and human-AI interaction studies.
• Its modular architecture uses composable features, modular rewards, and dual backends (NumPy/JAX) to create scalable, customizable environments.
• Integration with Multi-User Gymnasium transforms simulations into browser-based experiments with interactive, low-latency deployment.

Searching arXiv for the specified paper and related records. arxiv_search query: (McDonald et al., 16 Apr 2026) arxiv_search results for "(McDonald et al., 16 Apr 2026)":

• "CoGrid & the Multi-User Gymnasium: A Framework for Multi-Agent Experimentation" (McDonald et al., 16 Apr 2026) CoGrid is a multi-agent, grid-based simulation library introduced together with Multi-User Gymnasium (MUG) as part of a framework for multi-agent experimentation, especially human-AI interaction studies. It is a Python framework for building custom multi-agent environments that are easy to modify, compatible with standard reinforcement-learning tooling, and usable both for AI training and for human-AI experiments. In the paper’s framing, CoGrid addresses the lack of accessible infrastructure for bespoke multi-agent tasks by combining a modular environment library with a deployment path to browser-based studies through MUG (McDonald et al., 16 Apr 2026).

1. Design scope and research role

CoGrid is designed for settings in which researchers need to create custom tasks rather than rely on fixed benchmarks. The motivating problem is not only multi-agent reinforcement learning in the narrow benchmark sense, but also the study of social decision making, coordination, adaptation, trust, and human-AI complementarity. The paper therefore positions CoGrid as an experimentation substrate rather than a standardized benchmark suite, with explicit design goals of accessibility, modularity, customizability, support for arbitrary multi-agent experiments, and a path from simulation to human studies (McDonald et al., 16 Apr 2026).

The library’s representational core is a 2D grid world inspired by Minigrid but extended to support multiple agents natively. Each cell in an $n\times m$ grid is either empty or occupied by a GridObj, and agents are represented as Agent objects inheriting from the same GridObj base class. This unifies agents with other environment entities and allows the grid to hold an arbitrary number of agents.

A central methodological claim is that researchers often face a mismatch between “training environments” and “human experiment environments.” CoGrid is meant to reduce that mismatch by making one environment implementation usable both for high-throughput AI training and for browser-compatible human experiments. This suggests that CoGrid’s primary contribution is not a single task domain, but a reusable task-construction layer spanning MARL and human-subject experimentation.

2. Formal model and environment abstractions

CoGrid models environments as partially observable Markov decision processes with tuple

$$(\mathcal{X}, \mathcal{A}, \mathcal{O}, \mathcal{T}, \mathcal{R}, \Omega),$$

where $\mathcal{X}$ is the state space, $\mathcal{A}$ the action space, $\mathcal{O}$ the observation space, $$\mathcal{T}:\mathcal{X}\times\mathcal{A}\to\mathcal{X}$$ the transition function, $$\mathcal{R}:\mathcal{X}\times\mathcal{A}\to\mathbb{R}$$ the reward function, and $\Omega:\mathcal{X}\to\mathcal{O}$ the observation function. Operationally, a CoGrid environment maintains an underlying world state, receives actions from one or more agents, updates the state through the transition function, emits rewards, and exposes only partial observations to each agent (McDonald et al., 16 Apr 2026).

Environments are implemented as subclasses of CoGridEnv. The architecture separates task-specific elements into composable parts: layouts can be specified with ASCII grid encodings; objects are Python classes with properties and rendering logic; rewards are modular classes; and observations are assembled from named Feature components. This decomposition is one of the paper’s central design ideas, because it avoids fixing one observation structure, one reward scheme, or one action set across all tasks.

Observations are deliberately not standardized into a single default interface. Each environment lists desired features in configuration, each Feature subclass defines a pure function from current state to a vector, and these feature functions are composed at initialization into one observation function whose outputs are concatenated for each agent. The paper emphasizes both the conceptual and systems value of this design: different multi-agent tasks require different informational interfaces, and pure compositional feature functions are easier to keep compatible with JAX compilation.

Rewards follow the same modular pattern. Each Reward subclass implements a compute method receiving the previous state, current state, and actions, and returning a per-agent reward array. The paper highlights helper abstractions for JAX-compatible reward specification, including an InteractionReward base class that lets users declaratively specify a triggering action, what object the agent is holding, and what object it is facing.

3. Backend architecture, API compatibility, and interaction design

A major systems contribution is CoGrid’s dual-backend architecture. Simulation code is written against a backend-agnostic array namespace, cogrid.backend.xp, which dispatches at runtime to either NumPy or JAX. With the NumPy backend, environments run as ordinary Python/Numpy code, which the paper identifies as important for ease of development and for web compatibility via Pyodide/WebAssembly. With the JAX backend, the same code can be JIT-compiled and vectorized with jax.vmap, allowing batched parallel simulation on accelerators such as GPUs (McDonald et al., 16 Apr 2026).

This dual-backend design is paired with explicit API positioning. CoGrid follows the PettingZoo API for multi-agent reinforcement learning, in contrast to Gymnasium’s single-agent standardization. The paper presents this as a way to remain compatible with standard MARL tooling while keeping the library closer in spirit to Minigrid—simple, hackable, Pythonic—than to heavier systems such as Griddly or Melting Pot.

On actions, CoGrid provides a default discrete action set similar to Minigrid: turn left, turn right, move forward, pickup/drop, toggle, and no-op. It also provides a direct-movement scheme—move left, move right, move up, move down—in place of rotation-plus-forward movement. The paper explicitly motivates this second option by human playability, since keyboard mappings for human participants should feel intuitive if the same environment will later be used in browser-based studies.

Rendering follows Minigrid’s tile-based approach. Every object has a render function that paints an RGB tile at its location using graphical primitives such as lines, circles, or polygons, and agent avatars are triangles rotated by orientation. The native renderer supports simulation and debugging, while MUG can replace these visuals with richer web assets when the environment is deployed as a human-facing experiment.

4. Integration with Multi-User Gymnasium

CoGrid is paired with Multi-User Gymnasium, a deployment framework that turns compatible Python environments into browser-based experiments. MUG is not limited to CoGrid, but the paper presents the two systems as intentionally complementary: CoGrid creates the environment in a form suitable for MARL and browser compatibility, while MUG adds instructions, matchmaking, waiting rooms, surveys, data logging, AI policy integration, and multiplayer synchronization (McDonald et al., 16 Apr 2026).

MUG’s architecture is scene-based. Experiments are composed from Scene objects, with GymScene as the key scene type for task interaction. In a GymScene, the researcher specifies the environment constructor, control mappings, rendering hooks, AI policies, and experiment logic. Scenes are sequenced by a Stager, which defines experiment flow and can randomize or counterbalance conditions. This supports consent pages, instruction pages, waiting rooms, gameplay scenes, questionnaires, and ending screens within one configurable pipeline.

Execution can occur in two modes. In server-authoritative mode, the Python environment runs on the server; clients send actions $a_t$, the server advances the state $s_t$, and updated state is returned to the browser. In browser-side mode, MUG executes Python environments directly in the browser using Pyodide, which compiles Python to WebAssembly. This execution path is one reason the NumPy backend matters: browser-executable environments cannot depend on a JAX runtime.

For low-latency multiplayer, MUG also supports peer-to-peer networking with rollback netcode modeled after GGPO. The paper writes the two-player transition as

$$(\mathcal{X}, \mathcal{A}, \mathcal{O}, \mathcal{T}, \mathcal{R}, \Omega),$$0

Each client runs the environment locally with a shared random seed, applies its own action immediately, predicts missing remote actions—typically by repeating the last known action—steps forward speculatively, and rolls back to the last confirmed state if actual delayed input differs from the prediction. This requires deterministic environments and explicit state serialization through get_state() and set_state().

5. Case studies and empirical profile

The paper’s principal CoGrid case study is an Overcooked environment replicating the “Cramped Room” layout from Overcooked-AI. This example showcases the environment-building workflow: subclass CoGridEnv, define task objects such as onions and plates as registered GridObjs, specify modular reward classes such as soup-delivery rewards, and compose observation features. In the Overcooked observation design, each chef receives one-hot encodings of direction and inventory, proximity and distance features for task-relevant objects, pot status features, distance to the partner, and the agent’s own grid position (McDonald et al., 16 Apr 2026).

The Overcooked study also serves as the dual-backend performance benchmark. The paper reports that the NumPy backend runs at about 450 steps per second, while the JAX backend scales from roughly 4,500 steps/s with one instance to 5.6 million steps/s at 1,024 parallel instances, outperforming the cited JaxMARL comparison by about 1.9x at that scale. The training study used PPO via RLlib with self-play, and the trained policy reached about 7.5 dishes delivered per episode.

The end-to-end pipeline was then deployed through MUG in browser-based human studies. Human-human and human-AI participants each played 20 episodes, with waiting rooms, task instructions, and post-task questionnaires managed through MUG. The paper reports not only overall task performance but also division of labor, observing that human-AI teams maintained stable aggregate performance while humans gradually took on a larger share of dish deliveries. This suggests that the framework can expose adaptation effects that would be difficult to study if training and experimentation were separated into distinct software stacks.

A second case study, Slime Volleyball, serves a different purpose. It demonstrates that MUG is not restricted to CoGrid or to grid worlds: the environment is a fast-paced, non-grid, physics-based Gymnasium environment adapted for deployment through MUG. Its inclusion clarifies the division of labor between the two tools: CoGrid is the grid-based simulation layer, whereas MUG is the generalized experiment-deployment layer.

6. Limitations, comparative context, and place in the tooling landscape

CoGrid is intentionally restricted to grid-based environments. The paper presents that restriction as a simplicity choice, but it excludes many continuous-control and physics-rich domains. MUG is broader, yet it currently lacks built-in explicit communication channels such as text chat or structured messaging between players; interaction is mediated only through actions and observations in the shared environment. For rollback-based multiplayer, environments must be deterministic and must implement state save/restore. The proof-of-concept studies reported in the paper were run with server-side execution, even though browser-side execution is supported and documented (McDonald et al., 16 Apr 2026).

The paper positions CoGrid against Minigrid, Griddly, Melting Pot, and JaxMARL. Compared with Minigrid, it adds native multi-agent support and a JAX backend. Compared with Griddly and Melting Pot, it sacrifices some engine complexity and expressive power in favor of readability, accessibility, and lower setup burden. Compared with JaxMARL or NAVIX, it is not JAX-only, which is presented as crucial for web compatibility.

MUG is contrasted with nodeGame, Overcooked-AI’s custom interactive demo, SHARPIE, CrowdPlay, and HIPPO-Gym. Its distinctive combination, as described in the paper, is support for Gymnasium/PettingZoo APIs, multi-human interaction, client-side execution through Pyodide, low-latency peer-to-peer multiplayer with GGPO-style rollback, and experiment-lifecycle features such as instructions, matchmaking, surveys, and data collection.

In that context, CoGrid is best understood as the simulation half of a broader stack. It provides a modular POMDP-based grid-world library with a single environment implementation that can support both browser-compatible NumPy execution and accelerator-oriented JAX training. MUG then operationalizes CoGrid environments—or any compatible Gymnasium/PettingZoo environment—as browser-based experiments involving arbitrary mixtures of humans and AI. The resulting workflow closes a methodological gap identified by the paper: a researcher can design a custom multi-agent task, train AI agents in that same task, and then study human-AI interaction online without rewriting the environment in a separate engine or web framework.