Multi-Agent Contextual Exploration (MACE)

• Multi-Agent Contextual Exploration (MACE) is a research line addressing coordinated exploration in decentralized, context-aware multi-agent systems.
• It combines local novelty signals with mechanisms like novelty sharing, safe corridor planning, and structured peer selection to enhance performance in sparse and complex environments.
• MACE formulations range from decentralized MARL and robotic mapping to multi-agent LLM interaction, highlighting the need for context-driven, cooperative exploration strategies.

Multi-Agent Contextual Exploration (MACE) denotes a family of multi-agent exploration problems in which agents must decide what to probe, whom to coordinate with, and how to adapt exploration to context such as sparse rewards, partial observability, local novelty, peer diversity, task structure, or environmental geometry. Recent work uses the term both as a general label for exploration under multi-agent context and as the name of specific methods, including a novelty-sharing method for decentralized cooperative MARL, a safe-corridor framework for autonomous exploration of unknown environments, and a lightweight peer-selection framework for multi-agent LLM interaction (Jiang et al., 2024, Toumieh et al., 2022, Choi et al., 13 Jul 2026). This suggests that MACE is best understood as a research line rather than a single algorithmic template.

1. Scope and terminological usage

The acronym “MACE” appears in multiple subfields, with different operational meanings.

Usage of MACE	Setting	Defining mechanism
MACE (Jiang et al., 2024)	Decentralized cooperative MARL	Local novelty sharing, approximate global novelty, weighted mutual information, hindsight intrinsic reward
MACE (Toumieh et al., 2022)	Multi-agent autonomous exploration of unknown environments	Mapping, safe corridor generation, multi-agent planning
MACE (Choi et al., 13 Jul 2026)	Multi-agent LLM interaction	Structured peer selection via per-agent contextual bandits

In decentralized MARL, MACE is introduced as “a simple yet effective multi-agent coordinated exploration method” that addresses unavailable global novelty and the need for coordinated exploration by communicating only local novelty and using weighted mutual information as an intrinsic signal (Jiang et al., 2024). In robotics, MACE is a framework for “multi-agent collaborative exploration of unknown environments” that integrates mapping, safe corridor generation, and planning, with exploration ending when all voxels are discovered as free or occupied, or when no path can be found for the remaining undiscovered voxels (Toumieh et al., 2022). In multi-agent LLM systems, MACE is a “lightweight framework that explicitly promotes exploration through structured peer selection,” motivated by the finding that modern LLM agents “fail to explore each other” and instead exhibit “myopic and polarized interaction patterns” (Choi et al., 13 Jul 2026).

This multiplicity of usage has a substantive consequence. The shared concern is not a particular architecture, but the exploration problem itself: agents must convert local observations or local beliefs into coordinated exploratory behavior under uncertainty.

2. Formal problem formulations

The formalizations used in MACE-related work vary with domain, but they are consistently built around decentralized information and context-dependent action choice.

In multi-agent LLM interaction, the exploration problem is formalized as a partially observable stochastic game,

$$\mathcal{G} = \langle \mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{P}, \mathcal{Q}, \mathcal{R} \rangle,$$

where latent states include environment, task, and agents’ unknown capabilities, actions correspond to peer selection, and each agent observes only direct interactions (Choi et al., 13 Jul 2026). In generic multi-agent exploration platforms, the environment is cast as a Dec-POMDP,

$\langle n, S, A, O, R, P, \gamma, h \rangle,$

with objective

$$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$

which is the form adopted by MAexp for continuous-action robot exploration (Zhu et al., 2024). In contextual games under unknown constraints, each player observes a context $z^t$ and must choose actions from an unknown feasible set

$$\mathcal{A}_i^f(z) := \{ a_i \in \mathcal{A}_i \mid g_{i,m}(a_i, z) \leq 0, ~ \forall m \in [M] \},$$

so exploration must simultaneously learn reward structure and feasibility (Maddux et al., 2023).

These formulations differ in semantics—peer selection, motion planning, contextual play—but share a common burden: local signals are biased or incomplete, while the relevant exploratory object is often global, relational, or latent. That gap is explicit in decentralized MARL, where “the novelty of global states is unavailable, while the novelty of local observations is biased,” and in LLM interaction, where peer capability is latent and must be inferred through repeated querying (Jiang et al., 2024, Choi et al., 13 Jul 2026).

3. Intrinsic reward, novelty sharing, and coordinated exploration signals

A central strand of MACE research modifies the exploration signal rather than the transition dynamics. The recurrent claim is that naive single-agent exploration mechanisms do not transfer directly to multi-agent settings.

“Curiosity-Driven Multi-Agent Exploration with Mixed Objectives” argues that straightforward extensions that use either individual novelty only or collective novelty only fail to provide “a distinct but collaborative intrinsic reward signal” (Reyes et al., 2022). Its Mixed Curiosity Module uses two prediction heads, one for an agent’s own next observation and one for the next joint observation, and defines intrinsic reward as

$$i_t^n = \|\hat{o}_{t+1}^{n,\text{indiv}} - o_{t+1}^n\|_2^2 + \|\hat{o}_{t+1}^{n,\text{joint}} - o_{t+1}\|_2^2.$$

In cooperative navigation with sparse rewards, COMA+MCM achieved $0.89 \pm 0.04$, $0.68 \pm 0.25$, $0.82 \pm 0.14$, and $0.66 \pm 0.25$ on the 2-agent Same, 4-agent Same, 2-agent Diff, and 4-agent Diff settings, respectively, whereas methods based only on individual or only on joint novelty were scenario-dependent (Reyes et al., 2022).

The decentralized MARL method named MACE takes a different route. Each agent communicates only a single floating-point number per timestep, its local novelty $\langle n, S, A, O, R, P, \gamma, h \rangle,$0, and each agent sums all received novelties as a proxy for global novelty:

$\langle n, S, A, O, R, P, \gamma, h \rangle,$1

It then introduces weighted mutual information,

$\langle n, S, A, O, R, P, \gamma, h \rangle,$2

and converts it into a hindsight intrinsic reward that encourages an agent to take actions that later enable other agents’ novelty (Jiang et al., 2024). The shaped reward is

$\langle n, S, A, O, R, P, \gamma, h \rangle,$3

The paper reports superior performance in GridWorld, Overcooked, and SMAC under sparse rewards, and ablations indicate that both global novelty approximation and hindsight influence reward are necessary for best results (Jiang et al., 2024).

Other reward-shaping approaches identify narrower exploratory targets. MESE selects a sub-state by conditioned entropy and estimates entropy with a particle-based estimator, then applies Random Network Distillation only to the selected sub-state:

$\langle n, S, A, O, R, P, \gamma, h \rangle,$4

It is presented as a plug-and-play module and substantially improves MAPPO on several SMAC tasks (Tao et al., 2023). CMAE instead chooses a shared exploration goal from projected state spaces via normalized entropy and trains agents to reach that goal in a coordinated manner; on sparse-reward MPE tasks it solves all tasks within 3M environment steps, whereas Q-learning, count-based, EITI, and EDTI baselines fail completely (Liu et al., 2021). PIMAEX ties social influence directly to a peer’s intrinsic return, aiming to reward an agent for influencing another agent to obtain novel states; in the Consume/Explore environment, PIMAEX-$\langle n, S, A, O, R, P, \gamma, h \rangle,$5 achieves the highest joint returns and the lowest variance among the evaluated variants (Kölle et al., 2 Jan 2025).

Taken together, these results make a recurring point: local novelty alone is often insufficient, but purely collective novelty creates credit-assignment problems. The dominant response has been to combine local signals with shared, relational, or hindsight-based structure.

4. Structure-aware and hierarchical exploration

A second strand of MACE research makes exploration contextual by exploiting structure in the state space, the task family, or the interaction graph.

MESA is a “meta-exploration method for cooperative multi-agent learning” that first identifies a high-reward joint state-action subspace from training tasks and then learns a set of diverse exploration policies to cover that subspace (Zhang et al., 2024). Valuable joint state-action pairs are aggregated into a global buffer $\langle n, S, A, O, R, P, \gamma, h \rangle,$6, sparse rewards can be relabelled by

$\langle n, S, A, O, R, P, \gamma, h \rangle,$7

and the learned exploration policies can be integrated with any off-policy MARL algorithm at test time (Zhang et al., 2024). The paper reports significantly better performance in sparse-reward particle and multi-agent MuJoCo environments and generalization to more challenging tasks.

LEMAE introduces task-relevant guidance from an LLM by grounding linguistic knowledge into symbolic key states through discriminator functions, then using Subspace-based Hindsight Intrinsic Reward,

$\langle n, S, A, O, R, P, \gamma, h \rangle,$8

and organizing exploration with a Key State Memory Tree (Qu et al., 2024). The reported result is up to 10x acceleration on MPE and SMAC, together with reduced redundant exploration and compatibility with QMIX, IPPO, and QPLEX (Qu et al., 2024).

In robotic exploration, MANTM replaces dense metric sharing with an online topological map composed of main nodes and ghost nodes, and uses a Hierarchical Topological Planner with graph neural networks for main-node selection and ghost-node assignment (Yang et al., 2023). In Habitat, it reduces steps by at least 26.40% over planning-based baselines and by at least 7.63% over RL-based competitors in unseen scenarios (Yang et al., 2023). Multi-agent option discovery pursues a related structural idea in the joint observation space: attention is used to find collaborative agent sub-groups, and HA-MSAC learns multi-agent options that minimize expected cover time for each sub-group before integrating them with a high-level policy (Chen et al., 2022).

The common premise is that exploration becomes more effective when the system identifies the part of the state-action space that matters—sub-states, key states, joint subspaces, topological frontiers, or agent sub-groups—rather than treating the entire state space as equally worth probing.

5. Robotic systems, planning methods, and simulation platforms

MACE is also a systems topic in robotics and simulation, where exploration quality depends on mapping fidelity, safety constraints, and sampling efficiency.

The robotics framework titled “MACE: Multi-Agent Autonomous Collaborative Exploration of Unknown Environments” combines local and global voxel mapping, frontier-style goal assignment, Jump Point Search, Distance Map Planner, and time-aware Safe Corridors (Toumieh et al., 2022). Its local planning problem is posed as an MIQP over jerk-controlled trajectories constrained to lie in time-indexed safe corridors, and the reported safety ratio is always greater than 1 for teams of 2, 3, and 4 agents in the simulator (Toumieh et al., 2022). The exploration ends when the mapped volume is complete or no feasible path remains.

Decentralized ergodic trajectory optimization addresses similar objectives with a distributed optimal-control formulation. Each agent optimizes a cost containing ergodicity, control energy, and inter-agent distance penalty,

$\langle n, S, A, O, R, P, \gamma, h \rangle,$9

while communicating only with neighboring agents (Gkouletsos et al., 2021). The reported effect is reduced task completion time and more efficient control-energy distribution through collaboration.

MAexp reframes the problem at the platform level. It uses point clouds rather than grid maps, supports six state-of-the-art MARL algorithms—IPPO, ITRPO, MAPPO, MATRPO, VDPPO, and VDA2C—and decomposes exploration into a Multi-Agent Target Generator for “where-to-go” and a Single-Agent Motion Planner for “how-to-go” (Zhu et al., 2024). Its sampling speed is reported as approximately 40 times faster than existing platforms, with per-step sampling times of $$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$0 s, $$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$1 s, $$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$2 s, and $$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$3 s for $$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$4, respectively (Zhu et al., 2024).

This systems literature emphasizes a different but complementary aspect of MACE: exploration is not only a question of reward design, but also of representation, communication load, planning tractability, and deployment realism.

6. LLM-mediated contextual exploration, theory, and limitations

Recent work extends MACE into settings where agents are themselves LLMs or where LLMs provide strategic guidance.

“Multi-Agent LLMs Fail to Explore Each Other” states that current LLM agents often make premature commitments to specific peers, leading to high regret, and introduces MACE as a decomposition into independent per-agent contextual bandit problems with features for response diversity, peer distinctiveness, historical performance, interaction round, and bias (Choi et al., 13 Jul 2026). Each agent uses a LinUCB-style optimistic rule,

$$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$5

and regret is bounded sublinearly for the exploring policy, while the non-exploring regret lower bound grows with diversity:

$$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$6

The corollary states that the value of exploration increases with agent diversity (Choi et al., 13 Jul 2026).

A related decomposition appears in multi-agent LLM Bayesian optimization. There, a strategy agent outputs interpretable weights $$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$7 over exploitation, informativeness, diversity, and representativeness, and a generation agent proposes candidates according to

$$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$8

The stated motivation is that single-agent LLM prompting suffers from “cognitive overload,” whereas separating strategic control from candidate generation makes exploration–exploitation trade-offs explicit, observable, and adjustable (Carbonati et al., 30 Mar 2026).

Context calibration is another emerging theme. CERMIC is a communication-free MARL framework in which each agent infers a dynamic intention graph over peer behavior and calibrates curiosity through an information-bottleneck objective,

$$J(\theta) = \mathbb{E}\left[\sum_{t=0}^{h-1} \gamma^t R(s(t), a(t)) \,\Big|\, s(0), \pi\right],$$9

with intrinsic reward defined as Bayesian information gain. The reported result is that CERMIC-augmented agents outperform baselines and SoTA exploration methods on 12/16 sparse-reward benchmark scenarios in VMAS, MeltingPot, and SMACv2 (Pan et al., 25 Sep 2025).

Several limitations recur across the literature. The decentralized novelty-sharing MACE requires a fully-connected communication network during training, uses communication only at training time, and ideally relies on on-policy data for posterior estimation (Jiang et al., 2024). LEMAE’s symbolic grounding depends on interpretable task descriptions and awaits further multimodal advances for highly non-symbolic domains (Qu et al., 2024). In sparse-reward SMAC, CMAE improves substantially over baselines, yet on 3s_vs_5z-sparse no method succeeds (Liu et al., 2021). In nonlinear multi-agent MCTS, NonZero addresses joint-action explosion through interaction-guided local deviations and proves an action-dimension-free local-regret bound,

$z^t$0

but its formulation also underlines how quickly exploration becomes intractable if coordination structure is not exploited (Tang et al., 1 May 2026).

A common misconception is that exploration in multi-agent systems can be recovered by simply adding more randomness or more effort. The literature repeatedly rejects that view. Some methods emphasize mixed novelty rather than naive curiosity, some emphasize subspaces or key states rather than full-state novelty, and some show that structured peer selection is necessary because prompting alone does not induce reliable exploration (Reyes et al., 2022, Qu et al., 2024, Choi et al., 13 Jul 2026). Another misconception is that local novelty is an adequate proxy for global progress; the novelty-sharing MACE paper explicitly identifies local novelty bias as a core problem, and mixed-objective curiosity methods report that purely individual or purely collective signals are insufficiently robust across scenarios (Jiang et al., 2024, Reyes et al., 2022).

Across these variants, MACE research converges on a stable technical lesson: effective multi-agent exploration is rarely undirected. It is typically mediated by context—task structure, peer behavior, communication constraints, learned topologies, or explicit diversity estimates—and its success depends on making that context operational in the exploration policy itself.