--- title: Multi-Agent Exploration url: https://www.emergentmind.com/topics/multi-agent-exploration type: topic --- # Multi-Agent Exploration Searching arXiv for the cited multi-agent exploration papers to ground the article and verify bibliographic details. {"query":"multi-agent exploration arXiv MACE collaborative exploration unknown environments PIMAEX MUI-TARE strangeness-driven exploration", "max_results": 10} Searching arXiv for the supplied titles and closely related multi-agent exploration work. Multi-agent exploration studies how multiple agents discover unknown environments, poorly visited regions of a state space, or task-relevant intermediate structures while operating under partial observability, sparse rewards, communication limits, safety constraints, or finite interaction budgets. In the literature represented here, the term covers at least four closely related problem classes: complete exploration of physical environments by robots, coordinated exploration in cooperative MARL, task-driven exploration under communication or abstraction constraints, and more recent extensions to LLM-mediated optimization and scientific search [2208.06949] [1910.05512] [2403.14780] [2603.28959]. The common difficulty is not merely visiting novel states, but coordinating who explores what, when, and with what information so that redundancy, unsafe interactions, and uninformative detours are reduced. ## 1. Problem setting and main formulations A large fraction of the literature formulates multi-agent exploration as a Dec-POMDP or CTDE problem. In the sparse landmark-complex setting, exploration is modeled as a Dec-POMDP with joint state space, joint action space, partial observations, and a team reward that combines simplex discovery, communication cost, collision penalties, completion bonus, and time penalty [2209.11794]. SMMAE likewise starts from a fully cooperative Dec-POMDP under CTDE, while MAexp formalizes multi-agent exploration as a Decentralized Partially Observable Markov Decision Process with shared team reward and continuous-control policies [2301.02083] [2404.12824]. LEMAE uses the standard Dec-POMDP for cooperative agents with a sparse extrinsic reward that is nonzero only at the success state [2410.02511]. Embodied exploration papers often specialize this generic formulation to complete coverage of unknown space. In MACE, exploration ends once every voxel is labeled free or occupied, or no path to any unknown voxels remains [2208.06949]. In MUI-TARE, the loop continues until all surfaces are covered, while the additional complication is that agents may have unknown initial positions and their sub-maps must be merged robustly [2209.10775]. Reward-free exploration theory abstracts away rewards entirely: the problem becomes learning an accurate transition model of a finite-horizon tabular MDP so that an $\epsilon$-optimal policy can later be computed for any reward function [2602.01453]. The term “MACE” is used for two different contributions in this corpus. “MACE: Multi-Agent Autonomous Collaborative Exploration of Unknown Environments” is a voxel-based robotic exploration framework with centralized map merging and goal assignment [2208.06949]. “Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing” is a decentralized MARL method that approximates global novelty by communicating local novelty and adds a hindsight weighted mutual-information reward [2402.02097]. | Paradigm | Representative papers | Exploration object | |---|---|---| | Embodied mapping and coverage | MACE, MUI-TARE, MANTM, BAMAX, MAexp | voxels, sub-maps, topological nodes, hex cells, point clouds | | Cooperative MARL | EITI/EDTI, CMAE, Mixed Curiosity, SIM, MESE, SMMAE, FoX, PIMAEX, novelty-sharing MACE | joint states, sub-states, formations, peers’ novelty or returns | | Task-driven and semantic exploration | map compression, LEMAE, multi-agent LLM BO, MACC | path-relevant uncertainty, key states, weighted search criteria, blackboard hypotheses | | Reward-free exploration | phased cooperative exploration | transition kernels | ## 2. Spatial representation and map construction Representation determines what counts as “known,” “unknown,” and “worth exploring.” MACE adopts a regular 3D voxel grid: the workspace \(W \subset \mathbb{R}^3\) is divided into cubic voxels of side \(\Delta\), with \(\Delta=0.3\) m in the reported system, and LiDAR ray tracing labels voxels as free, occupied, or unknown [2208.06949]. Border voxels are then extracted by the condition \[ \text{state}(v)=\text{free} \quad\text{and}\quad \exists\,u\in N(v):\;\text{state}(u)=\text{unknown}, \] clustered by connectivity, and converted into potential goals via centroids and nearest-centroid voxels. This yields a frontier-like mechanism entirely inside a voxel representation. MUI-TARE uses a different decomposition. Each agent builds sub-maps segmented into smaller “segments,” and candidate overlaps are encoded in a factor graph \(G=(V,E)\) whose inner-connection weights \(\omega_{i,j}\) quantify overlap quality [2209.10775]. Exploration is therefore coupled to an explicit sub-map merging problem: repeating another agent’s past trajectory is not a fixed conservative rule, but an adaptive verification process driven by overlap quality and the verification gain \[ G(i,j,t)= \frac{\omega^t_{i,j}-\omega^{t_0}_{i,j}+C_\epsilon}{\|t-t_0\|+\epsilon}. \] This changes exploration from pure coverage into coverage under uncertain multi-map registration. MANTM replaces metric-heavy exchange by a topological graph containing visited “Main Nodes” and frontier-like “Ghost Nodes” [2311.00252]. Main nodes are created from panoramic RGB-D embeddings using a cosine threshold, ghost nodes are placed uniformly around each main node, and merged graphs are formed by fusing nearby main nodes in a common global frame. The paper explicitly argues that topological maps consist only of nodes and edges with abstract but essential information and are less influenced by scene structures. A plausible implication is that multi-agent exploration can trade metric precision for representation stability and lower communication cost. Other works push representation further toward task structure. In the sparse landmark-complex setting, observations add 0-, 1-, and 2-dimensional simplices to an abstract simplicial complex, so exploration quality is measured by discovered simplices rather than occupancy coverage [2209.11794]. MAexp models environments as continuous 2D/3D point clouds rather than discrete grids, with locally adaptive density and GPU-parallelized point-cloud filtering [2404.12824]. BAMAX, by contrast, represents each agent’s state on an \(N\times N\) hexagonal grid through six feature channels, including explored cells, unexplored cells, own location, other-agent locations, walls, and a local six-neighbor radar [2411.08400]. Collectively, these works show that “exploration” is representation-relative: frontier voxels, topological ghost nodes, simplices, point-cloud regions, and unexplored hex cells induce different coordination and planning problems. ## 3. Planning, coordination, and safety in embodied exploration Once a representation is fixed, the central question becomes how multiple agents are coordinated without collisions, deadlock, or excessive duplication. MACE organizes its system into a Local Module running onboard each agent and a Global Module acting as a central hub [2208.06949]. Local mapping, single-agent planning, and control are decentralized, whereas map merging and goal assignment are centralized. Agents plan synchronously at 10 Hz, goals are assigned at 5 Hz, and time-aware Safe Corridors are embedded in a MIQP so that intra-agent collision safety and safety from static obstacles are guaranteed. The corridors are made time-dependent by adding separating half-spaces derived from other agents’ predicted positions at each horizon step. MUI-TARE addresses a different coordination bottleneck: deciding when a suspected overlap between two agents’ sub-maps should be accepted, rejected, or actively verified [2209.10775]. If the overlap quality \(\omega_{i,j}\) exceeds a threshold, the merge is accepted immediately; otherwise, the selected agent backtracks to the overlap and follows a transformed look-ahead segment of the other agent’s trajectory. After relative poses are known, cooperative planning is posed as the min-max multi-depot vehicle routing problem, \[ \min_{\tau_{1..K_m}^{\text{glob}}}\max_{j=1..K_m}\{\mathrm{dist}(\tau_j^{\text{glob}})\}, \] so that agents in the same merged sub-map are planned jointly rather than independently. Here exploration efficiency and merge robustness are treated as a coupled optimization problem. Hierarchical or two-stage planners recur in several systems. MANTM’s Hierarchical Topological Planner selects a main node coarsely and a ghost node finely through graph attention and MAPPO training [2311.00252]. MAexp explicitly separates “where-to-go” from “how-to-go”: an attention-based Multi-Agent Target Generator produces a coarse region and a fine target offset, while a Single-Agent Motion Planner based on the Dynamic Window Approach executes robot-compatible motion under Ackermann constraints [2404.12824]. The task-driven map-compression framework also centralizes action selection, but its objective is not full coverage; sensors are routed by solving local discounted MDPs whose rewards are derived from uncertainty along the Seeker’s planned path [2403.14780]. Backtracking appears as an explicit control primitive in BAMAX [2411.08400]. When an agent is judged “stuck,” the method uses A* on the explored free-space graph to find the nearest node bordering an unexplored hex and feeds the resulting path back as override actions until frontier contact is re-established. This contrasts with MACE’s predictive safe-corridor coordination and MUI-TARE’s verification-driven repetition: backtracking here is not an incidental recovery behavior but a named component of the exploration algorithm. ## 4. Intrinsic objectives and coordinated exploration in MARL In cooperative MARL, exploration is commonly reshaped through intrinsic rewards, auxiliary policies, or explicit coordination objectives. Influence-based exploration introduced two canonical forms: EITI, which uses conditional mutual information to reward states where one agent affects other agents’ transition dynamics, and EDTI, which uses Value of Interaction to measure how one agent’s behavior changes another agent’s expected return [1910.05512]. PIMAEX generalizes this line by defining a social influence reward with three terms weighted by \(\alpha,\beta,\gamma\), and its specialized reward ties an agent’s return to how much it influences peers’ combined extrinsic and intrinsic rewards; the paper identifies the \(\beta\) term as the main novel term [2501.01266]. The decentralized novelty-sharing MACE introduces weighted mutual information, \[ \omega I(A_t^i;Z_t^j\mid o_t^i) = \E\!\left[z\log\frac{p(a,z\mid o_t^i)}{p(a\mid o_t^i)p(z\mid o_t^i)}\right], \] and turns it into a hindsight intrinsic reward based on actually observed accumulated novelty \(z_t^j\) [2402.02097]. Curiosity-style methods differ mainly in what novelty they measure. The Mixed Curiosity Module uses a two-headed predictor per agent: one head predicts the agent’s next local observation and the other predicts the next joint observation, and the intrinsic reward is the sum of individual and joint prediction errors with \(\alpha=\beta=1\) in the reported experiments [2210.16468]. Strangeness-driven exploration defines observation-level strangeness from autoencoder reconstruction error and global state-level strangeness from state-prediction error, then trains a separate exploration Q-function \(Q^e\) on the mixed reward \(r_t^{\rm ext}+\beta r_t^e\) while preserving a goal Q-function trained only on extrinsic reward [2212.13448]. MESE selects a low-diversity sub-state via conditioned entropy and applies RND only to that sub-state, thereby turning exploration into a team reward over coordinates deemed hard to reach cooperatively [2306.06382]. A second group of methods alters the search space or the exploration policy itself. CMAE projects the global state into restricted spaces, ranks them by normalized entropy, samples a low-entropy projection, and gives all agents a shared goal chosen as the rarest projected state in a minibatch [2107.11444]. FoX defines a formation-based equivalence relation \(\sim_{\mathcal F}\) over the exploration space, uses count-based novelty over formations, and adds a mutual-information-style formation-awareness reward so that agents infer current formations from local observations [2308.11272]. SMMAE trains an independent exploration policy for each agent to maximize local state-space coverage and adaptively adjusts each agent’s exploration probability according to the stability of the joint team policy, approximated through cross-entropy and mutual-information surrogates [2301.02083]. A recurring misconception is that coordinated exploration is adequately handled by naive independent novelty bonuses or \(\epsilon\)-greedy noise. Several of these papers explicitly argue otherwise. The Mixed Curiosity paper states that straightforward multi-agent extensions provide either individual or collective novelty only and therefore do not provide a distinct but collaborative intrinsic reward signal [2210.16468]. PIMAEX reports that vanilla PPO fails to learn in the Consume/Explore environment while peer-incentivized variants outperform PPO+RND [2501.01266]. The novelty-sharing MACE reports that IPPO+r\(_{\text{loc}}\) fails to solve any sparse task because it does not induce coordination [2402.02097]. The broad pattern is that effective exploration rewards are rarely scalar novelty terms attached independently to each agent; they are typically interaction-aware, abstraction-aware, or explicitly shared. ## 5. Communication, abstraction, and semantic guidance Communication is often the hidden bottleneck in multi-agent exploration. MACE keeps communication minimal by sending local voxel grids and planned trajectories while centralizing only global map consistency and goal assignment; the local voxel grid can be built in \(\le 1\) ms via GPU [2208.06949]. MUI-TARE reports ROS multimaster communication at approximately 786 KB/s per agent and emphasizes that adaptive merge overhead occurs mainly at overlap events, keeping average bandwidth and CPU load low [2209.10775]. MANTM argues that topological maps reduce communication traffic because agents exchange node and edge information rather than full metric submaps [2311.00252]. The task-driven compression framework makes communication itself a decision variable through abstraction indices \(\theta\), compressed occupancies \(o=A^\theta x\), variances \(V\), and bit cost \(n^\theta=k^\theta n_m+n_i\) [2403.14780]. The decentralized novelty-sharing MACE goes further and communicates only a scalar local novelty \(u_t^i\) each step, using \(\sum_j u_t^j\) as an approximation to unavailable global novelty [2402.02097]. Abstraction is also used to make guidance task-relevant rather than merely novel. LEMAE asks an LLM to generate symbolic key states and discriminator functions \(\mathcal F_i(s)\), then uses Subspace-based Hindsight Intrinsic Reward \[ r_I^{\kappa_i}(t) = \|\Phi_i(s_t)-\Phi_i(\kappa_i)\| - \|\Phi_i(s_{t+1})-\Phi_i(\kappa_i)\| \] to densify reward in the subspace relevant to each key state [2410.02511]. Its Key State Memory Tree stores observed transitions between key states and modulates exploration randomness according to branch depth. The key claim is not generic “semantic reasoning,” but that the LLM is used only a few times per task to produce reusable symbolic discriminators. Two recent extensions generalize multi-agent exploration beyond standard RL environments. In multi-agent LLM Bayesian optimization, a Strategy Agent outputs normalized weights over exploitation, informativeness, diversity, and representativeness, and a Generation Agent proposes candidates conditioned on the resulting scalarized acquisition function [2603.28959]. In MACC, independently managed LLM-based agents interact through an Incentive-Driven Blackboard that records submissions, reproduction attempts, performance, reproducibility flags, and collaboration impact, while a parameterized incentive mechanism allocates rewards [2603.03780]. These frameworks preserve the central structure of multi-agent exploration—division of labor, shared memory, incentive shaping, and redundancy control—even though the explored object is now a candidate set or a scientific hypothesis space rather than a physical environment. ## 6. Empirical patterns, theoretical limits, and open problems Empirical results in embodied systems consistently report gains from explicit coordination. MACE, evaluated in AirSim on a \(30\times30\times3\) m world with 90 cylindrical obstacles and up to 4 agents, reports that flight distance and exploration time scale approximately as \(1/M\), while the minimum inter-agent distance always exceeds the collision radius, yielding Safety ratio \(>1\) [2208.06949]. MUI-TARE reports being up to 50% more efficient than baselines on average while merging sub-maps robustly, with planner runtimes remaining within real-time budget [2209.10775]. MANTM reports at least 26.40% fewer steps than planning-based baselines and at least 7.63% fewer than RL-based competitors in unseen scenarios [2311.00252]. MAexp reports sampling speed approximately 40 times faster than existing platforms through point-cloud representation and GPU parallelization [2404.12824]. BAMAX reports faster coverage and less backtracking than DFS, BFS, Collaborative DFS, and Collaborative BFS on hex grids from \(10\times10\) to \(60\times60\), and on \(\mathcal G_{60}\) it is described as 38% faster than the next best method [2411.08400]. MARL results show a similar pattern: methods that shape exploration around interaction structure or search abstractions outperform undirected baselines. PIMAEX reports that each single-term peer-influence variant outperforms PPO+RND and that the \(\beta\)-only variant achieves the best joint return with markedly lower standard deviation at inference [2501.01266]. The novelty-sharing MACE reports reliable convergence to 100% success in Pass, SecretRoom, and MultiRoom within roughly 1k–5k PPO updates, while ablations that remove either the \(z\)-weight or the log-ratio degrade performance [2402.02097]. FoX reports substantial gains on sparse SMAC and Google Research Football scenarios, including strong improvements on the larger-agent settings [2308.11272]. CMAE reports that only CMAE solves Pass, Secret-Room, and Push-Box within 3 M steps in the sparse setting considered there [2107.11444]. These results do not imply a single dominant exploration principle; rather, they show that several different coordination devices—shared goals, influence bonuses, formation abstractions, sub-state selection, or peer incentives—can each be effective when aligned with the structure of the task. At the theoretical end, reward-free cooperative exploration exposes a sharp adaptivity limit. For a tabular finite-horizon MDP, MR.MARFE achieves an \(\epsilon\)-approximation of the dynamics using \(\rho=H\) phases and \(\widetilde O(S^6H^6A/\epsilon^2)\) agents per phase [2602.01453]. The lower bound shows that any algorithm restricted to \(\rho