Multi-Agent Exploration

• Multi-agent exploration is the study of coordinating multiple agents to efficiently map and cover unknown environments under constraints such as sparse rewards and partial observability.
• It employs formulations based on Dec-POMDP and CTDE, using diverse spatial representations like voxel grids, topological maps, and point clouds for tailored exploration strategies.
• Recent research demonstrates significant improvements in safety, coordination, and computational efficiency through intrinsic rewards, hierarchical planning, and optimized communication protocols.

Searching arXiv for the cited multi-agent exploration papers to ground the article and verify bibliographic details. {"query":"multi-agent exploration arXiv MACE collaborative exploration unknown environments PIMAEX MUI-TARE strangeness-driven exploration", "max_results": 10} Searching arXiv for the supplied titles and closely related multi-agent exploration work. Multi-agent exploration studies how multiple agents discover unknown environments, poorly visited regions of a state space, or task-relevant intermediate structures while operating under partial observability, sparse rewards, communication limits, safety constraints, or finite interaction budgets. In the literature represented here, the term covers at least four closely related problem classes: complete exploration of physical environments by robots, coordinated exploration in cooperative MARL, task-driven exploration under communication or abstraction constraints, and more recent extensions to LLM-mediated optimization and scientific search (Toumieh et al., 2022, Wang et al., 2019, Psomiadis et al., 2024, Carbonati et al., 30 Mar 2026). The common difficulty is not merely visiting novel states, but coordinating who explores what, when, and with what information so that redundancy, unsafe interactions, and uninformative detours are reduced.

1. Problem setting and main formulations

A large fraction of the literature formulates multi-agent exploration as a Dec-POMDP or CTDE problem. In the sparse landmark-complex setting, exploration is modeled as a Dec-POMDP with joint state space, joint action space, partial observations, and a team reward that combines simplex discovery, communication cost, collision penalties, completion bonus, and time penalty (Sun et al., 2022). SMMAE likewise starts from a fully cooperative Dec-POMDP under CTDE, while MAexp formalizes multi-agent exploration as a Decentralized Partially Observable Markov Decision Process with shared team reward and continuous-control policies (Zhang et al., 2023, Zhu et al., 2024). LEMAE uses the standard Dec-POMDP for cooperative agents with a sparse extrinsic reward that is nonzero only at the success state (Qu et al., 2024).

Embodied exploration papers often specialize this generic formulation to complete coverage of unknown space. In MACE, exploration ends once every voxel is labeled free or occupied, or no path to any unknown voxels remains (Toumieh et al., 2022). In MUI-TARE, the loop continues until all surfaces are covered, while the additional complication is that agents may have unknown initial positions and their sub-maps must be merged robustly (Yan et al., 2022). Reward-free exploration theory abstracts away rewards entirely: the problem becomes learning an accurate transition model of a finite-horizon tabular MDP so that an $\epsilon$-optimal policy can later be computed for any reward function (Barnea et al., 1 Feb 2026).

The term “MACE” is used for two different contributions in this corpus. “MACE: Multi-Agent Autonomous Collaborative Exploration of Unknown Environments” is a voxel-based robotic exploration framework with centralized map merging and goal assignment (Toumieh et al., 2022). “Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing” is a decentralized MARL method that approximates global novelty by communicating local novelty and adds a hindsight weighted mutual-information reward (Jiang et al., 2024).

Paradigm	Representative papers	Exploration object
Embodied mapping and coverage	MACE, MUI-TARE, MANTM, BAMAX, MAexp	voxels, sub-maps, topological nodes, hex cells, point clouds
Cooperative MARL	EITI/EDTI, CMAE, Mixed Curiosity, SIM, MESE, SMMAE, FoX, PIMAEX, novelty-sharing MACE	joint states, sub-states, formations, peers’ novelty or returns
Task-driven and semantic exploration	map compression, LEMAE, multi-agent LLM BO, MACC	path-relevant uncertainty, key states, weighted search criteria, blackboard hypotheses
Reward-free exploration	phased cooperative exploration	transition kernels

2. Spatial representation and map construction

Representation determines what counts as “known,” “unknown,” and “worth exploring.” MACE adopts a regular 3D voxel grid: the workspace $W \subset \mathbb{R}^3$ is divided into cubic voxels of side $\Delta$, with $\Delta=0.3$ m in the reported system, and LiDAR ray tracing labels voxels as free, occupied, or unknown (Toumieh et al., 2022). Border voxels are then extracted by the condition

$$\text{state}(v)=\text{free} \quad\text{and}\quad \exists\,u\in N(v):\;\text{state}(u)=\text{unknown},$$

clustered by connectivity, and converted into potential goals via centroids and nearest-centroid voxels. This yields a frontier-like mechanism entirely inside a voxel representation.

MUI-TARE uses a different decomposition. Each agent builds sub-maps segmented into smaller “segments,” and candidate overlaps are encoded in a factor graph $G=(V,E)$ whose inner-connection weights $\omega_{i,j}$ quantify overlap quality (Yan et al., 2022). Exploration is therefore coupled to an explicit sub-map merging problem: repeating another agent’s past trajectory is not a fixed conservative rule, but an adaptive verification process driven by overlap quality and the verification gain

$$G(i,j,t)= \frac{\omega^t_{i,j}-\omega^{t_0}_{i,j}+C_\epsilon}{\|t-t_0\|+\epsilon}.$$

This changes exploration from pure coverage into coverage under uncertain multi-map registration.

MANTM replaces metric-heavy exchange by a topological graph containing visited “Main Nodes” and frontier-like “Ghost Nodes” (Yang et al., 2023). Main nodes are created from panoramic RGB-D embeddings using a cosine threshold, ghost nodes are placed uniformly around each main node, and merged graphs are formed by fusing nearby main nodes in a common global frame. The paper explicitly argues that topological maps consist only of nodes and edges with abstract but essential information and are less influenced by scene structures. A plausible implication is that multi-agent exploration can trade metric precision for representation stability and lower communication cost.

Other works push representation further toward task structure. In the sparse landmark-complex setting, observations add 0-, 1-, and 2-dimensional simplices to an abstract simplicial complex, so exploration quality is measured by discovered simplices rather than occupancy coverage (Sun et al., 2022). MAexp models environments as continuous 2D/3D point clouds rather than discrete grids, with locally adaptive density and GPU-parallelized point-cloud filtering (Zhu et al., 2024). BAMAX, by contrast, represents each agent’s state on an $N\times N$ hexagonal grid through six feature channels, including explored cells, unexplored cells, own location, other-agent locations, walls, and a local six-neighbor radar (Kalra et al., 2024). Collectively, these works show that “exploration” is representation-relative: frontier voxels, topological ghost nodes, simplices, point-cloud regions, and unexplored hex cells induce different coordination and planning problems.

3. Planning, coordination, and safety in embodied exploration

Once a representation is fixed, the central question becomes how multiple agents are coordinated without collisions, deadlock, or excessive duplication. MACE organizes its system into a Local Module running onboard each agent and a Global Module acting as a central hub (Toumieh et al., 2022). Local mapping, single-agent planning, and control are decentralized, whereas map merging and goal assignment are centralized. Agents plan synchronously at 10 Hz, goals are assigned at 5 Hz, and time-aware Safe Corridors are embedded in a MIQP so that intra-agent collision safety and safety from static obstacles are guaranteed. The corridors are made time-dependent by adding separating half-spaces derived from other agents’ predicted positions at each horizon step.

MUI-TARE addresses a different coordination bottleneck: deciding when a suspected overlap between two agents’ sub-maps should be accepted, rejected, or actively verified (Yan et al., 2022). If the overlap quality $\omega_{i,j}$ exceeds a threshold, the merge is accepted immediately; otherwise, the selected agent backtracks to the overlap and follows a transformed look-ahead segment of the other agent’s trajectory. After relative poses are known, cooperative planning is posed as the min-max multi-depot vehicle routing problem,

$W \subset \mathbb{R}^3$0

so that agents in the same merged sub-map are planned jointly rather than independently. Here exploration efficiency and merge robustness are treated as a coupled optimization problem.

Hierarchical or two-stage planners recur in several systems. MANTM’s Hierarchical Topological Planner selects a main node coarsely and a ghost node finely through graph attention and MAPPO training (Yang et al., 2023). MAexp explicitly separates “where-to-go” from “how-to-go”: an attention-based Multi-Agent Target Generator produces a coarse region and a fine target offset, while a Single-Agent Motion Planner based on the Dynamic Window Approach executes robot-compatible motion under Ackermann constraints (Zhu et al., 2024). The task-driven map-compression framework also centralizes action selection, but its objective is not full coverage; sensors are routed by solving local discounted MDPs whose rewards are derived from uncertainty along the Seeker’s planned path (Psomiadis et al., 2024).

Backtracking appears as an explicit control primitive in BAMAX (Kalra et al., 2024). When an agent is judged “stuck,” the method uses A* on the explored free-space graph to find the nearest node bordering an unexplored hex and feeds the resulting path back as override actions until frontier contact is re-established. This contrasts with MACE’s predictive safe-corridor coordination and MUI-TARE’s verification-driven repetition: backtracking here is not an incidental recovery behavior but a named component of the exploration algorithm.

4. Intrinsic objectives and coordinated exploration in MARL

In cooperative MARL, exploration is commonly reshaped through intrinsic rewards, auxiliary policies, or explicit coordination objectives. Influence-based exploration introduced two canonical forms: EITI, which uses conditional mutual information to reward states where one agent affects other agents’ transition dynamics, and EDTI, which uses Value of Interaction to measure how one agent’s behavior changes another agent’s expected return (Wang et al., 2019). PIMAEX generalizes this line by defining a social influence reward with three terms weighted by $W \subset \mathbb{R}^3$1, and its specialized reward ties an agent’s return to how much it influences peers’ combined extrinsic and intrinsic rewards; the paper identifies the $W \subset \mathbb{R}^3$2 term as the main novel term (Kölle et al., 2 Jan 2025). The decentralized novelty-sharing MACE introduces weighted mutual information,

$W \subset \mathbb{R}^3$3

and turns it into a hindsight intrinsic reward based on actually observed accumulated novelty $W \subset \mathbb{R}^3$4 (Jiang et al., 2024).

Curiosity-style methods differ mainly in what novelty they measure. The Mixed Curiosity Module uses a two-headed predictor per agent: one head predicts the agent’s next local observation and the other predicts the next joint observation, and the intrinsic reward is the sum of individual and joint prediction errors with $W \subset \mathbb{R}^3$5 in the reported experiments (Reyes et al., 2022). Strangeness-driven exploration defines observation-level strangeness from autoencoder reconstruction error and global state-level strangeness from state-prediction error, then trains a separate exploration Q-function $W \subset \mathbb{R}^3$6 on the mixed reward $W \subset \mathbb{R}^3$7 while preserving a goal Q-function trained only on extrinsic reward (Kim et al., 2022). MESE selects a low-diversity sub-state via conditioned entropy and applies RND only to that sub-state, thereby turning exploration into a team reward over coordinates deemed hard to reach cooperatively (Tao et al., 2023).

A second group of methods alters the search space or the exploration policy itself. CMAE projects the global state into restricted spaces, ranks them by normalized entropy, samples a low-entropy projection, and gives all agents a shared goal chosen as the rarest projected state in a minibatch (Liu et al., 2021). FoX defines a formation-based equivalence relation $W \subset \mathbb{R}^3$8 over the exploration space, uses count-based novelty over formations, and adds a mutual-information-style formation-awareness reward so that agents infer current formations from local observations (Jo et al., 2023). SMMAE trains an independent exploration policy for each agent to maximize local state-space coverage and adaptively adjusts each agent’s exploration probability according to the stability of the joint team policy, approximated through cross-entropy and mutual-information surrogates (Zhang et al., 2023).

A recurring misconception is that coordinated exploration is adequately handled by naive independent novelty bonuses or $W \subset \mathbb{R}^3$9-greedy noise. Several of these papers explicitly argue otherwise. The Mixed Curiosity paper states that straightforward multi-agent extensions provide either individual or collective novelty only and therefore do not provide a distinct but collaborative intrinsic reward signal (Reyes et al., 2022). PIMAEX reports that vanilla PPO fails to learn in the Consume/Explore environment while peer-incentivized variants outperform PPO+RND (Kölle et al., 2 Jan 2025). The novelty-sharing MACE reports that IPPO+r$\Delta$0 fails to solve any sparse task because it does not induce coordination (Jiang et al., 2024). The broad pattern is that effective exploration rewards are rarely scalar novelty terms attached independently to each agent; they are typically interaction-aware, abstraction-aware, or explicitly shared.

5. Communication, abstraction, and semantic guidance

Communication is often the hidden bottleneck in multi-agent exploration. MACE keeps communication minimal by sending local voxel grids and planned trajectories while centralizing only global map consistency and goal assignment; the local voxel grid can be built in $\Delta$1 ms via GPU (Toumieh et al., 2022). MUI-TARE reports ROS multimaster communication at approximately 786 KB/s per agent and emphasizes that adaptive merge overhead occurs mainly at overlap events, keeping average bandwidth and CPU load low (Yan et al., 2022). MANTM argues that topological maps reduce communication traffic because agents exchange node and edge information rather than full metric submaps (Yang et al., 2023). The task-driven compression framework makes communication itself a decision variable through abstraction indices $\Delta$2, compressed occupancies $\Delta$3, variances $\Delta$4, and bit cost $\Delta$5 (Psomiadis et al., 2024). The decentralized novelty-sharing MACE goes further and communicates only a scalar local novelty $\Delta$6 each step, using $\Delta$7 as an approximation to unavailable global novelty (Jiang et al., 2024).

Abstraction is also used to make guidance task-relevant rather than merely novel. LEMAE asks an LLM to generate symbolic key states and discriminator functions $\Delta$8, then uses Subspace-based Hindsight Intrinsic Reward

$\Delta$9

to densify reward in the subspace relevant to each key state (Qu et al., 2024). Its Key State Memory Tree stores observed transitions between key states and modulates exploration randomness according to branch depth. The key claim is not generic “semantic reasoning,” but that the LLM is used only a few times per task to produce reusable symbolic discriminators.

Two recent extensions generalize multi-agent exploration beyond standard RL environments. In multi-agent LLM Bayesian optimization, a Strategy Agent outputs normalized weights over exploitation, informativeness, diversity, and representativeness, and a Generation Agent proposes candidates conditioned on the resulting scalarized acquisition function (Carbonati et al., 30 Mar 2026). In MACC, independently managed LLM-based agents interact through an Incentive-Driven Blackboard that records submissions, reproduction attempts, performance, reproducibility flags, and collaboration impact, while a parameterized incentive mechanism allocates rewards (Oyama et al., 4 Mar 2026). These frameworks preserve the central structure of multi-agent exploration—division of labor, shared memory, incentive shaping, and redundancy control—even though the explored object is now a candidate set or a scientific hypothesis space rather than a physical environment.

6. Empirical patterns, theoretical limits, and open problems

Empirical results in embodied systems consistently report gains from explicit coordination. MACE, evaluated in AirSim on a $\Delta=0.3$0 m world with 90 cylindrical obstacles and up to 4 agents, reports that flight distance and exploration time scale approximately as $\Delta=0.3$1, while the minimum inter-agent distance always exceeds the collision radius, yielding Safety ratio $\Delta=0.3$2 (Toumieh et al., 2022). MUI-TARE reports being up to 50% more efficient than baselines on average while merging sub-maps robustly, with planner runtimes remaining within real-time budget (Yan et al., 2022). MANTM reports at least 26.40% fewer steps than planning-based baselines and at least 7.63% fewer than RL-based competitors in unseen scenarios (Yang et al., 2023). MAexp reports sampling speed approximately 40 times faster than existing platforms through point-cloud representation and GPU parallelization (Zhu et al., 2024). BAMAX reports faster coverage and less backtracking than DFS, BFS, Collaborative DFS, and Collaborative BFS on hex grids from $\Delta=0.3$3 to $\Delta=0.3$4, and on $\Delta=0.3$5 it is described as 38% faster than the next best method (Kalra et al., 2024).

MARL results show a similar pattern: methods that shape exploration around interaction structure or search abstractions outperform undirected baselines. PIMAEX reports that each single-term peer-influence variant outperforms PPO+RND and that the $\Delta=0.3$6-only variant achieves the best joint return with markedly lower standard deviation at inference (Kölle et al., 2 Jan 2025). The novelty-sharing MACE reports reliable convergence to 100% success in Pass, SecretRoom, and MultiRoom within roughly 1k–5k PPO updates, while ablations that remove either the $\Delta=0.3$7-weight or the log-ratio degrade performance (Jiang et al., 2024). FoX reports substantial gains on sparse SMAC and Google Research Football scenarios, including strong improvements on the larger-agent settings (Jo et al., 2023). CMAE reports that only CMAE solves Pass, Secret-Room, and Push-Box within 3 M steps in the sparse setting considered there (Liu et al., 2021). These results do not imply a single dominant exploration principle; rather, they show that several different coordination devices—shared goals, influence bonuses, formation abstractions, sub-state selection, or peer incentives—can each be effective when aligned with the structure of the task.

At the theoretical end, reward-free cooperative exploration exposes a sharp adaptivity limit. For a tabular finite-horizon MDP, MR.MARFE achieves an $\Delta=0.3$8-approximation of the dynamics using $\Delta=0.3$9 phases and $$\text{state}(v)=\text{free} \quad\text{and}\quad \exists\,u\in N(v):\;\text{state}(u)=\text{unknown},$$0 agents per phase (Barnea et al., 1 Feb 2026). The lower bound shows that any algorithm restricted to $$\text{state}(v)=\text{free} \quad\text{and}\quad \exists\,u\in N(v):\;\text{state}(u)=\text{unknown},$$1 phases requires $$\text{state}(v)=\text{free} \quad\text{and}\quad \exists\,u\in N(v):\;\text{state}(u)=\text{unknown},$$2 agents for constant accuracy. The central conclusion is that a polynomial number of agents is possible only when the number of learning phases is on the order of the horizon.

Open problems recur across otherwise disparate papers. Several robotic systems assume static environments, perfect or low-latency communication, or reliable SLAM, and explicitly note that dynamic scenes, asynchronous delays, or SLAM errors are not addressed (Yang et al., 2023, Psomiadis et al., 2024). Some MARL methods rely on true global state during training, which limits direct transfer to fully decentralized settings (Kim et al., 2022). LEMAE currently requires symbolic state descriptions and task information for prompting (Qu et al., 2024). PIMAEX, SMMAE, and related methods note that scalability to larger populations and richer state-action spaces remains to be tested (Kölle et al., 2 Jan 2025, Zhang et al., 2023). Reward-free theory identifies extension to function approximation as open (Barnea et al., 1 Feb 2026). Taken together, these limitations suggest that multi-agent exploration remains less a single solved problem than a family of coordination problems whose tractability depends on representation, communication model, and the degree to which task structure can be exposed or learned.