Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

Abstract: As LLMs are increasingly deployed as autonomous agents, they must coordinate with others over long horizons in open-ended interactive tasks. Yet existing evaluations rarely test these demands together, instead emphasising single-agent tasks, short interactions, or highly structured multi-agent settings. We introduce $alem$, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics. Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world with exploration, crafting, trading, and combat. We evaluate $13$ modern LLMs zero-shot within homogeneous teams, with trained MARL agents as reference points. Current LLM agents remain far from solving alem, averaging only ~6% normalised return, but their failures are not uniform. On the hardest coordination setting, zero-shot Gemini-3.1-Pro-High approaches MARL agents trained for one billion steps, while GPT-5.4-High achieves strong base-task reward but much lower coordination reward. This contrast shows that individual task competence does not imply coordination competence. Ablations show that communication is the largest contributor to coordination, while memory and reasoning help when used to maintain multi-step plans. Overall, our results identify coordination as a distinct bottleneck for frontier LLM agents, separate from single-agent capabilities. Alem makes this bottleneck measurable and provides a controlled testbed for developing agents that communicate, allocate roles, and execute shared plans. Code is available at https://github.com/alem-world/alem-env.

• The paper introduces ALEM with innovations in procedural task generation, adaptive role allocation, and explicit communication channels.
• Evaluation of LLMs and MARL agents reveals modest coordination rewards, underscoring the gap between base-task proficiency and true collaborative performance.
• Ablation studies demonstrate that removing communication or reasoning traces sharply degrades coordination, emphasizing their critical role in multi-agent systems.

Benchmarking Long-Horizon Multi-Agent Coordination in Language Agents: The ALEM Benchmark

Motivation and Context

Recent progress in embedding LLMs as autonomous agents leads to increasing requirements for adaptive, robust, and communicative collaboration in multi-agent, open-ended, long-horizon environments. While prior benchmarks such as MultiAgentBench, Collab-Overcooked, and Melting Pot offer frameworks for probing aspects of agent collaboration, they are fundamentally constrained either by short time horizons, static coordination motifs, lack of explicit communication channels, or inability to parametrize and scale coordination difficulty systematically. This paper introduces ALEM, a highly extensible JAX-based open-world benchmark for evaluating LLM and MARL agents under challenging multi-agent coordination pressure, filling critical gaps left by previous work.

The ALEM Environment and Task Structure

Built atop Craftax-Coop’s framework, ALEM extends its capacity to probe coordination via four central innovations: (i) procedurally generated and parameterized coordination tasks spanning the full handover-to-synchronous spectrum; (ii) soft specialization which mitigates rigid role locking while incentivizing adaptive role allocation; (iii) an explicit, high-bandwidth communication channel; and (iv) a single parameter α controlling task coupling, temporal separation, solo failure rates, and non-specialist inefficiency, thus enabling finely tuned scaling of coordination difficulty.

Within each episode, mining, construction, combat, and crafting tasks are dynamically instantiated to require either synchronous (joint N-agent) or timed handover-dependent actions, with game-theoretic reward structures (pure coordination and assurance games). The resulting POSG is non-trivial and induces strong, non-reducible inter-agent dependencies, as quantified by information-theoretic dependence metrics; baseline removal ablations show multi-agent failures cannot be attributed to mere base-environment challenge.

Agents can interact with the environment via pixel, symbolic, or a text-based interface explicitly mapping environment state, constraints, and action space, making ALEM uniquely suitable for both RL and LLM agents under controlled observation and action semantics.

Systematic Evaluation: LLM and MARL Agents

ALEM provides three classes of metrics: base-task progress, explicit coordination achievement, and total normalized return. Thirteen state-of-the-art LLMs (including Gemini-3.1-Pro-High, GPT-5.4-High, Gemma-4-31B-it, multiple Qwen and Llama variants) are evaluated zero-shot; four strong MARL algorithms (IPPO, MAPPO, HyperMARL-IPPO, PQN-VDN) serve as RL baselines, with up to one billion steps of environment interaction. Mixed and homogeneous teams are systematically compared.

Key Results:

• All LLMs and MARL baselines are far from saturating ALEM; total normalized return for frontier LLMs is ~6% on Hard, even after prompt tuning and with explicit action affordance parsing.
• Gemini-3.1-Pro-High achieves up to 17.5% coordination reward on Hard, matching MARL performance after one billion steps.
• Base-task proficiency is not predictive of coordination skill: GPT-5.4-High leads in general reward but lags dramatically on coordination metrics, while models with modest base progress (e.g., Gemma-4-26B-A4B-it, Qwen3.5-122B-A10B) sometimes outperform in coordination reward.
• There is no monotonic scaling with model size or architecture: Qwen and Gemma mixtures-of-experts do not dominate, and within-family comparisons show highly nontrivial scaling, underlining the role of post-training phenomena and possibly alignment or reasoning trace style.

Notably, coordination reward is highly sensitive to task category: handover tasks are comparatively accessible, while synchronous-joint requirements and multi-stage (e.g., construction/chained crafting) coordination are effectively unsolved by all agents except Gemini-3.1-Pro-High, which achieves only 7% construction task coverage compared to 0% for the strongest MARL.

Harness Ablations: Communication, Memory, Reasoning

Ablation studies make several findings explicit:

• Communication emerges as the unequivocal bottleneck for coordination: Removing the communication channel drops Gemini-3.1-Pro-High from 17.5% to 5.3% coordination success (Gemma-4-31B-it: 8.8%→3.8%).
• Scratchpad (episodic memory) is only beneficial when utilized for explicit plan-tracking: Gemini-3.1-Pro-High leverages it for forward-looking plans, but Gemma-4-31B-it merely records state summaries with minimal future intent, reducing impact.
• Reasoning traces are critical: Both general progress and explicit coordination degrade sharply when reduced (Gemini: Coord.% 17.5→9.7, Base% 13.8→10.3; Gemma: Coord.% 8.8→5.5, Base% 7.3→4.6). Quality of reasoning modulates the use of both communication and memory fields, as shown by lexical and structural analysis.

Qualitative analysis of memory and communication fields reveals that only certain LLMs spontaneously adopt planner-like behavioral traces, linking interfaces and agent design to emergent coordination competencies.

Heterogeneous Team Experiments

Mixed LLM teams (same-family and cross-family) perform near the mean of their constituent homogeneous teams, not matching the coordination skill of their strongest member nor collapsing to their weakest. This suggests nontrivial emergent misalignment and incompatible deliberation/communication conventions across architectures, highlighting open challenges in heterogeneous multi-agent LLM interaction.

Implications and Directions for AI Research

The ALEM benchmark exposes that single-agent base-task proficiency, even in long-horizon, procedurally generated environments, fails as a proxy for emergent multi-agent coordination capacity. Among current LLMs, only some proprietary models reach MARL parity, and only on easier coordination tasks. Strong numerical claims are supported by extensive stratified bootstrap CIs and ablation replicates; no LLM nor MARL agent comes close to saturating the benchmark, even after extensive training.

From the environment design perspective, ALEM demonstrates that coupling explicit, parametric coordination tasks with scalable JAX infrastructure enables robust evaluation and rapid iteration for both RL and foundation model agents. The procedural generation and difficulty scaling mechanisms preclude memorization, while explicit communication interfaces force direct probe of higher-level agentic capabilities.

Practically, ALEM provides the first platform to systematically tune for and diagnose inter-agent coordination failure modes, which will be essential for real-world deployments involving pooled LLMs, agent-assisted software engineering, or embodied collaborative robotics. On the theoretical side, the fact that coordination is structurally separated from base competence—despite both emerging from similar pretraining—imposes tight constraints on the prospects of unsupervised, scale-driven emergence of “social intelligence” in current LLMs.

Future research avenues opened by ALEM include:

• studying VLMs and multi-modal coordination,
• harness/agent wiring strategies for robust cross-model interoperability,
• memory and communication protocols for hierarchical and team-centric planning,
• and establishing the minimal set of architectural or training innovations required to achieve nontrivial coverage on challenging, long-horizon multi-agent tasks.

Conclusion

ALEM sets a new standard for evaluating open-ended multi-agent coordination in both RL and LLM agents. The results unequivocally demonstrate that base-agent competence and even long-horizon proficiency do not transfer to coordinated, collaborative behaviour. Communication is currently the most salient limiting factor, and the environment provides a sound foundation for ongoing progress in both the practical engineering of agent teams and the scientific understanding of multi-agent intelligence in neural architectures.