Collaborative Maze-Solving Benchmark

• The paper establishes a controlled grid-based framework that assesses multi-agent coordination via defined action spaces and communication protocols.
• It modulates complexity through parameters like maze size, wall density, and path length while enforcing collision and legality rules.
• Evaluation protocols use metrics such as success rate, exploration cost, and communication efficiency to benchmark diverse algorithmic strategies.

A collaborative maze-solving benchmark is a controlled empirical framework for evaluating the ability of multiple agents—human, AI, or robotic—to coordinate their actions, share information, and achieve goals in structured maze environments under varying constraints. Such benchmarks underpin comparative research in agent collaboration, multi-robot systems, distributed planning, and collective intelligence by exposing the unique challenges of partial observability, heterogeneity, and communication.

1. Formal Definitions and Problem Structures

Collaborative maze-solving benchmarks operationalize agent cooperation through tightly-specified grid-based environments, well-defined agent action spaces, and explicit communication or observation rules.

A prototypical instance draws a maze $m$ from a parameterized family $\mathcal{M}(\theta)$, where $\theta=(N,\rho,\ell)$ governs grid size ($N\times N$), wall density $\rho\in[0,1]$, and average path length $\ell$ between designated start and goal cells. For example, the benchmark in "The Collaboration Gap" (Davidson et al., 4 Nov 2025) uses $N=6$, $\rho=0.30$, and $\ell\approx 7$–$9$ in its main experiments.

Distributed observability is fundamental: for agent-based collaboration, the benchmark typically decomposes $m$ into $k$ non-overlapping, partially obfuscated local maps $m^1, m^2, \ldots$ so that $m^1 \cup m^2 = m$ and $m^1\cap m^2 = \emptyset$ up to hidden cells. Each agent observes only its $m^i$ and perhaps local state (e.g., position $p_i$). In multi-robot benchmarks, maps are unknown to all agents and are incrementally constructed through exploration, observation sharing, or frontiers detected via sensor range (Linardakis et al., 2024).

Agent actions are defined via a set $\mathcal{A}$ (e.g., $$\{\text{up},\text{down},\text{left},\text{right},\text{stay}\}$$), with transitions executed jointly or synchronously depending on protocol. Benchmarks generally enforce collision and legality constraints.

2. Benchmark Design Principles and Complexity Modulation

Collaborative benchmarks must isolate joint capabilities, support fine-grained complexity control, and enable scalability.

Key design levers:

• Maze Size and Structure: Parameter $N$ can range from small ($4\times4$) to very large ($256\times256$ in POGEMA (Skrynnik et al., 2024)), influencing the memory and planning demands.
• Obstacle/Wall Density: Varying $\rho$ modifies pathfinding difficulty and interaction frequency.
• Path Length: Average/optimal distance $\ell$ controls the sequentiality and potential for deadlock or communication breakdown.
• Partial Observability: Imposed through map obfuscation or egocentric sensing (sensing radius $R$).
• Agent Heterogeneity: Explicitly tested by benchmarking both homogeneous (identical models) and heterogeneous agent teams with varying privileges or types (Davidson et al., 4 Nov 2025).

Benchmarks like POGEMA (Skrynnik et al., 2024) and Linardakis et al. (Linardakis et al., 2024) supplement grid randomization with reproducible map seeds, different maze-generation algorithms (random, wall-based, warehouse, or public benchmarks), and variations in initial and goal placement. Agent counts $K$ or $N$ are varied to probe scalability and congestion.

3. Evaluation Protocols and Automated Grading

Collaborative maze-solving evaluation protocols establish strict, automated statistical procedures for reproducibility and comparability.

Primary Protocol Features:

• Solo vs. Collaborative Baselines: Models are run in solo (full or distributed view), homogeneous, and heterogeneous (cross-model) pairings. For example, "The Collaboration Gap" (Davidson et al., 4 Nov 2025) evaluates 32 models in all configurations.
• Rollout Limits and Dialogue Rules: Turn/step counts are capped (e.g., 50-move limit). Communication is typically unconstrained in format but must satisfy minimal protocol rules (agreement required for move execution, single move at a time).
• Transcript Parsing: Resulting agent dialogues/transcripts are parsed by a grader agent, which extracts move sequences for outcome evaluation (Davidson et al., 4 Nov 2025).
• Performance Metrics:
  • Binary route validity $y\in\{0,1\}$.
  • Weighted outcome $$w = \frac{d_{\mathrm{last}} - d_{\mathrm{opt}}}{d_{\mathrm{last}}}$$, with $d_{\mathrm{last}}$ the Manhattan distance to goal after rollout, and $d_{\mathrm{opt}}$ the shortest possible path.
  • Collaboration efficiency, measured as product of median message length and communication rounds.

POGEMA (Skrynnik et al., 2024) and Linardakis et al. (Linardakis et al., 2024) define additional metrics relevant for cooperative pathfinding and exploration:

Metric	Definition (abridged)	Context
Success Rate (SR)	Fraction of runs where all agents reach goals	(Skrynnik et al., 2024)
Sum-of-Costs (SoC)	$\sum_{i=1}^K L_i$, $L_i$ = steps to goal	(Skrynnik et al., 2024)
Makespan (MS)	$\max_{i=1,\ldots,K} L_i$	(Skrynnik et al., 2024)
Exploration Rounds (R)	Rounds until $100\%$ coverage	(Linardakis et al., 2024)
Exploration Cost	$\sum_{i=1}^N d_i$	(Linardakis et al., 2024)
Efficiency (Eff)	$M/\text{Cost}$ ($M$ = free cells explored)	(Linardakis et al., 2024)
Congestion	Agent-to-map density ratio	(Skrynnik et al., 2024)

Statistical analysis involves mean, standard deviation, bootstrapped CIs, and paired or nonparametric significance tests.

4. Core Empirical Insights and Identified Gaps

Systematic benchmarking exposes distinct collaboration failure modes and sharp gaps between solo and joint performance.

Collaboration Gap: In "The Collaboration Gap" (Davidson et al., 4 Nov 2025), almost all LLMs display a "collaboration gap"—performance drops sharply when moving from solo to joint (homogeneous or heterogeneous) settings. For example, GPT-4.1's weighted outcome drops by $\Delta w \approx 0.32$ in this transition, and distilled/mini models often degrade to near-zero, highlighting lack of robustness in distributed protocol induction.

Ordering and Relay Effects: The order of agent turns is highly consequential. Stronger-first (strong agent primes dialogue before handing over) improves outcomes by up to 20 points in $w$. The "relay inference" protocol—freezing $K$ turns from a strong agent before switching to a weaker agent—substantially closes the gap, with as few as two priming turns yielding major gains.

Grounding and Theory of Mind Failures: Benchmarks reveal persistent errors in coordinate axis alignment, agreement on map schemas, and misinterpretation of partner intentions. Stronger models tend to propose explicit negotiation protocols (e.g., agreeing on coordinate origins) but weaker models omit such steps, leading to deadlocks or misaligned execution (Davidson et al., 4 Nov 2025).

Multi-Agent Pathfinding (MAPF) and Exploration (MAE) Benchmarks: In POGEMA (Skrynnik et al., 2024) and Linardakis et al. (Linardakis et al., 2024), classical search/planning (e.g., LaCAM) and hybrid methods outperform pure MARL by a wide margin on success rate and SoC. MARL methods, while computationally efficient per step, underperform in complex, congested, or bottlenecked scenarios, whereas hybrid planners optimally balance runtime scalability and solution optimality. In exploration settings, cost–utility-based algorithms (CU) achieve lower cost and rounds than potential-field or nearest-frontier baselines.

5. Algorithmic Approaches: MAE, MAPF, and Swarm Paradigms

Benchmarks support comparison of diverse algorithmic paradigms:

• Distributed Cost–Utility (CU): As in Linardakis et al. (Linardakis et al., 2024), agents use shared maps and a utility function combining wavefront cost and expected information gain to assign frontiers.
• Centralized and Hybrid Planners: POGEMA (Skrynnik et al., 2024) evaluates LaCAM, RHCR, and hybrid algorithms (e.g., SCRIMP, Follower), showing trade-offs between solution quality and scalability.
• Swarm-Based Approaches: Density-driven swarms of memoryless, markerless particles (Sánchez et al., 22 Sep 2025) solve mazes via local density and flow rules, with a single kinetic parameter $\eta$ controlling the transition between diffusive and fast-traveling regimes. These systems achieve linear-in-size solving times $O(s)$ in "fast" parameter regions, providing competitive scaling relative to classical graph search but using minimal agent state.
• Model-Free MARL: Parameter-shared agents learn decentralized coordination using QMIX, QPLEX, MAMBA, and similar architectures, evaluated via Gym/PettingZoo APIs (Skrynnik et al., 2024).

Each paradigm is benchmarked on communication, exploration cost, solution optimality, and robustness to scaling.

6. Research Recommendations and Future Directions

Collaborative maze-solving benchmarks highlight the need for evaluation that targets collaboration as a distinct, first-class capability. Recommendations include:

• Collaboration-Aware Evaluation: Benchmarks should systematically expose partial observability and unpredictable protocol negotiation, treating joint performance as a critical metric independent of solo capability (Davidson et al., 4 Nov 2025).
• Training Innovations: Standard pre-training and distillation pipelines fail to impart grounding, role inference, or robust conflict resolution. New training objectives must explicitly reward alignment with diverse partners and mutual model trust in distributed settings.
• Interaction and UI Design: Human–AI and AI–AI systems should structure interactions to allow stronger agents to prime context or schemas early, leveraging relay or "expert-seeding" protocols to mitigate late-stage misalignment (Davidson et al., 4 Nov 2025).
• Benchmark Expansion: Extensions are encouraged towards dynamic scenarios (changning obstacles, dynamic goals), communication-constrained settings, and richer environments involving non-zero-sum objectives or tool use.
• Cross-Benchmark Generalization: A unified platform (e.g., POGEMA) allows fair comparison of learning-based, search-based, and swarm-based strategies, incentivizing solutions that generalize across topologies and scales (Skrynnik et al., 2024).

A plausible implication is that as agents become increasingly heterogeneous and operate under real-world constraints, simple stylized benchmarks may underestimate true collaboration breakdown rates, underscoring the value of isolating and quantifying these effects early in development pipelines.

7. Comparative Tables and Key Results

Empirically grounded comparison enables algorithm selection and research targeting. Example results (abbreviated from (Linardakis et al., 2024)):

Method	Rounds	Cost	Efficiency	Time (s)
New–CU	24.3	198.7	0.0757	0.052
CU–MNM	26.1	213.4	0.0702	0.049
HEDAC	29.9	240.2	0.0625	0.056

On MAPF tasks (Skrynnik et al., 2024), LaCAM achieves top success rates ($\sim$0.95), with MARL methods trailing ($\sim$0.4 best-case), especially under high agent-density congestion.

The reproducible, data-rich nature of these collaborative maze-solving benchmarks provides a robust foundation for advancing the reliable deployment and joint alignment of heterogenous agent-based systems in both simulation and real-world environments.