CoGrid & the Multi-User Gymnasium: A Framework for Multi-Agent Experimentation

Abstract: The increasing integration of AI in everyday life brings with it new challenges and questions for regarding how humans interact with autonomous agents. Multi-agent experiments, where humans and AI act together, can offer important opportunities to study social decision making, but there is a lack of accessible tooling available to researchers to run such experiments. We introduce two tools designed to reduce these barriers. The first, CoGrid, is a multi-agent grid-based simulation library with dual NumPy and JAX backends. The second, Multi-User Gymnasium (MUG), translates such simulation environments directly into interactive web-based experiments. MUG supports interactions with arbitrary numbers of humans and AI, utilizing either server-authoritative or peer-to-peer networking with rollback netcode to account for latency. Together, these tools can enable researchers to deploy studies of human-AI interaction, facilitating inquiry into core questions of psychology, cognition, and decision making and their relationship to human-AI interaction. Both tools are open source and available to the broader research community. Documentation and source code is available at {cogrid, multi-user-gymnasium}.readthedocs.io. This paper details the functionality of these tools and presents several case studies to illustrate their utility in human-AI multi-agent experimentation.

• The paper introduces a modular, grid-based multi-agent environment toolkit with dual NumPy/JAX support that boosts simulation throughput.
• It integrates human-AI interaction with a browser-based platform that leverages client-side execution and rollback netcode for real-time experiments.
• Benchmarking on Overcooked and Slime Volleyball demonstrates significant improvements in performance and scalability for multi-agent research.

CoGrid & the Multi-User Gymnasium: An Integrated Framework for Multi-Agent Experimentation

Motivation and Context

The increasing complexity and ubiquity of AI reinforce the urgent need for robust, reproducible frameworks for multi-agent experimentation, particularly those encompassing both human and AI agents. Despite advances in RL and multi-agent research, extant platforms emphasize agent-centric simulation, often neglecting critical infrastructure for human-AI and human-human interaction. Current solutions limit extensibility, accessibility, or experimental throughput—bottlenecking empirical progress in both cognition and AI complementarity research. The "CoGrid & the Multi-User Gymnasium" framework directly targets these limitations by releasing (1) a modular, grid-based, multi-agent environment toolkit with dual NumPy/JAX backend support, and (2) a generalized platform for deploying such environments as browser-based interactive experiments supporting arbitrary numbers of human and/or AI agents (2604.15044).

Technical Architecture of CoGrid

At its core, CoGrid generalizes the Minigrid architecture for multi-agent environments, supporting POMDP formalism $$(\mathcal{X}, \mathcal{A}, \mathcal{O}, \mathcal{T}, \mathcal{R}, \Omega)$$. Its core innovations are in modularization, backend abstraction, and PettingZoo-compliant APIs. All environment logic is backend-agnostic, with array operations dispatched at runtime to either NumPy or JAX, facilitating both rapid prototyping (NumPy) and hardware-accelerated batched simulation on the GPU/TPU (JAX) without altering environment source code. The agent abstraction is extended such that agents are grid objects, directly tracked and manipulated within the primary state representation.

Observations, actions, and rewards are fully modular—environment authors compose observations from feature maps, construct reward functions via declarative conditions, and specify action semantics at the environment configuration level. This enables highly tailored environments for RL, human-AI teaming, or behavioral studies. Visualization follows the Minigrid tile system but permits arbitrary complexity via user-coded rendering.

Figure 1: A visualization of the full map from the python{search_rescue} environment in CoGrid, demonstrating compositional object and agent representations for multi-agent collaboration and partial observability.

Multi-User Gymnasium: Real-Time, Integrated Web Deployment

Multi-User Gymnasium (MUG) bridges the simulation/experimentation gap: it wraps any Gymnasium/PettingZoo-compliant environment and automates the creation of interactive, browser-based experimental pipelines. Environments can execute fully client-side via Pyodide (Python in WebAssembly), eliminating network-induced latency for tight control loops—crucial for real-time, responsive human-agent dynamics. For multiplayer settings, MUG implements GGPO-style rollback netcode: clients proceed speculatively with predicted remote actions, and perform local rollback/replay when authoritative actions arrive, ensuring deterministic state flow with minimal perceptual disruption.

Figure 3: Experiment flow using a python{Stager} in the Multi-User Gymnasium, supporting modular sequencing of arbitrary "scenes" (information, interactive, survey, etc.).

MUG includes experiment lifecycle primitives: customizable onboarding/consent, randomized or counterbalanced condition assignment, built-in surveys, data collection hooks, flexible matchmaking, device/browser/latency filtering, and integration with online participant platforms. All experimental components are developer-accessible and modifiable through a unified configuration.

Benchmarking: Throughput and Environment Flexibility

Experimental results in Overcooked (Cramped Room) demonstrate the practical impact of CoGrid's architectural choices. On GPU hardware, the JAX backend in CoGrid achieves up to 5.6 million steps per second with 1024 parallel instances, a 1.9x improvement over JaxMARL, and orders-of-magnitude greater than the Python/NumPy mode (Figure 4). This performance enables efficient iterative RL experimentation, large-scale ablation studies, and accelerated environment randomization.

Figure 5: Environment throughput in the CoGrid Overcooked environment versus Overcooked-AI and JaxMARL baselines, demonstrating superior parallel performance with JAX-accelerated backend.

Human-AI and Human-Human Experimentation Pipeline

In proof-of-concept studies, Overcooked and Slime Volleyball were instantiated as both human-AI and human-human experiments. Overcooked agents were trained via PPO in pure simulation, then paired with humans in MUG-deployed tasks. Human-human pairs improved dish delivery performance across episodes, whereas human-AI pairs remained static—mirroring prior findings and emphasizing agents' inflexible policy. Per-actor contribution analysis indicates humans adapt roles to complement the AI’s policy, revealing richer team strategy dynamics than are visible from aggregate reward alone.

Figure 2: The Overcooked Cramped Room implementation as rendered with native CoGrid visualization—minimal, modular graphics corresponding to agent, item, and location classes.

Figure 4: Performance across episodes in both Overcooked studies; Human-human pairs steadily improve while human-AI pairs plateau due to the fixed RL agent strategy.

Figure 6: The relative contributions in the Human-AI study, showing that humans deliver more dishes over time in response to the agent’s fixed behavior.

A similar pattern arises in Slime Volleyball: humans paired with RL agents achieve longer episode durations (proxy for skillful play) than human-human pairs, and agent possession statistics mirror their robust cue-response policy. However, the framework’s flexibility enables such general tasks to be studied with little overhead in environment or experiment configuration.

Figure 7: The original Slime Volleyball visualization, leveraged directly via MUG.

Figure 8: Average episode length (skill proxy) in Slime Volleyball, highlighting longer rallies for human-AI pairs.

Figure 9: Agent and human ball-possession statistics per episode in Slime Volleyball; AI dominates time in possession, reflecting policy consistency.

Implications, Limitations, and Future Directions

This framework provides a strong foundation for controlled human-AI/multi-agent experimentation at scale. Practically, it eliminates major sources of engineering overhead: cross-browser simulation, low-latency multiplayer, synchronized state management, and rapid iteration on environment logic and experiment flow. Theoretically, it expands the empirical scope: RL agent learning, human-agent teaming, rapid environment prototyping, and parameteric social dilemmas are all within easy reach.

Current limitations stem from a lack of support for explicit inter-agent communication (e.g., text/chat primitives). The modular grid paradigm, while rich, is less suited for continuous-control or high-fidelity physics tasks; extending the abstraction beyond gridworlds could further amplify research relevance. Incorporating structured and unstructured communication, direct demonstration paradigms, and richer observation/action representations are logical next steps. The libraries are open-source, facilitating community-driven extension.

Conclusion

CoGrid and the Multi-User Gymnasium represent a technical advance in accessible, extensible, high-throughput infrastructure for multi-agent RL and human-AI experimentation. By unifying backend-agnostic simulation with direct, low-latency, web-native deployment, they dissolve persistent barriers in reproducibility, parallelization, and scalability for social decision making and agent complementarity research. Their demonstrated application to Overcooked and Slime Volleyball underscores their generality and utility in both RL and human-in-the-loop contexts. Broader adoption is likely to accelerate methodological advances and more nuanced empirical studies of human-agent systems, directly impacting data-driven cognition, RL, and social AI research communities.