Multi-Agent LLMs Fail to Explore Each Other

Abstract: Exploration is essential for reliable autonomy in multi-agent systems, yet it remains unclear whether LLM agents can explore effectively when interacting with one another. We show that modern LLM agents fail to do so, often exhibiting myopic and polarized interaction patterns that lead to suboptimal coordination and increased regret. We formalize this challenge as the Multi-Agent Exploration problem, modeling it as a partially observable stochastic game (POSG) problem in which agents must probe peers to infer their capabilities and identify effective interaction strategies. To address this, we introduce Multi- Agent Contextual Exploration (MACE), a lightweight framework that explicitly promotes exploration through structured peer selection. Across both contextual and parametric diversity settings, MACE substantially improves exploration behavior and downstream task performance. We further show theoretically that the value of exploration increases with agent diversity. Overall, our results highlight a fundamental limitation of current LLM agents and underscore the importance of explicitly guided exploration for reliable multi-agent autonomy. Code will be released in https://github.com/deeplearning-wisc/mace

• The paper demonstrates that LLM agents consistently fail in systematic exploration, leading to premature peer selection and increased cumulative regret.
• It introduces the MACE framework, a contextual bandit-inspired method that explicitly incentivizes diverse peer exploration in multi-agent environments.
• Empirical results and theoretical analysis show that improved exploration correlates with enhanced performance in heterogeneous settings.

Multi-Agent LLMs: Empirical and Theoretical Evidence for Systematic Exploration Failures

Introduction and Motivation

Despite their growing adoption in autonomous, decentralized settings, modern LLM agents exhibit critical deficiencies in multi-agent environments: most notably, a failure to explore their peers’ capabilities adequately. This paper introduces a rigorous empirical and theoretical investigation into the exploration behavior of LLM agents when embedded in multi-agent systems. The results indicate that prevailing LLM architectures, irrespective of scale or family, consistently display myopic, polarized selection behaviors—committing prematurely to interaction partners and accumulating substantial cumulative regret. This failure directly impedes system-level performance, particularly under conditions of context or capability heterogeneity.

The authors formalize the Multi-Agent Exploration problem via the lens of POSGs, wherein agents must probe peer capabilities under partial observability and evolving population structure. To address these deficits, they introduce the Multi-Agent Contextual Exploration (MACE) framework, grounded in contextual bandit theory and equipped with explicit relational exploration incentives.

Empirical Characterization of Exploration Deficits

The empirical motivation centers on a delegation bandit scenario: an LLM agent must repeatedly choose between two peers of unknown reliability, tasked with maximizing correct outcomes over multiple rounds. Benchmarking Qwen2.5-7B-Instruct, GPT-4, and GPT-5 in this setting reveals convergence to a sharp bimodal selection histogram—agents nearly always fixate on a single peer early, regardless of relative utility, and fail to recover when initial allocations are suboptimal.

Figure 1: Histogram of the number of peer A selection.

Quantitative analysis across parameter sweeps ($p_A$ vs $p_B$ variation) and CDFs of peer selection (see Figure 2; Figure 3) show that the LLMs’ selection strategies collapse to the extremes and rarely behave akin to adaptive exploration policies like UCB. This phenomenon is robust to model scale and underlying architecture.

Problem Formalization: Multi-Agent Exploration as POSG

The multi-agent context amplifies these deficits. Agents operate under strict partial observability, peer heterogeneity (parametric and contextual), and dynamically evolving interaction graphs (Figure 4). The interactions are modeled as POSGs, with each agent's policy depending on decentralized, local observation histories. Cumulative regret with respect to optimal (in-hindsight) peer policies is the core metric, quantifying the system’s failure to identify effective peer collaborations in the presence of capability and context variability.

Figure 4: The Multi-Agent Exploration problem. At round $t$, each agent needs to select a peer to interact with, and update its response in the subsequent interaction round.

Algorithmic Solution: Multi-Agent Contextual Exploration (MACE)

MACE reframes peer selection as a contextual bandit problem: each agent's choice of peer corresponds to pulling an arm, with expected rewards modeled linearly over relational feature vectors. These features encapsulate response diversity, peer distinctiveness, empirical success rates, and round indices, offering rich signals about peer under-exploration in the current interaction context (see Figure 5).

Figure 5: Contextual Diversity.

Unlike simple UCB or greedy strategies, MACE leverages LinUCB-style optimism—selecting the peer with the highest anticipated reward plus a calibrated uncertainty bonus over the current relational context. This enables substantially finer control over exploration/exploitation trade-offs and discourages premature commitment to suboptimal partners.

Experimental Results: MACE Enhances Exploration and Performance

Experiments are conducted in two canonical heterogeneity settings:

1. Contextual Diversity (HotpotQA distractor mode): Agents sharing the same backbone, each with a different context passage subset.
2. Parametric Diversity (heterogeneous LLM pool: GPT-5, Qwen2.5-7B, Llama-8B, Mistral-7B): Agents differ in underlying model capabilities but share global task context.

In both cases, MACE induces markedly more distributed peer selection (cf. In-Context Exploration, which over-commits aggressively), increases peer coverage, and generates policies that are robust across both exploration and exploitation phases.

Figure 6: Comparison of exploration and exploitation performance for each interaction round. Shaded are the standard errors across participating agents; standard error $*0.2$ are shown for Math500 and GPQA to avoid visual clutter. Also, note that the variance in the 0-th round is induced by GPT-5.

Cumulative regret analyses (Figure 7) show that:

• In-Context Exploration often underperforms even random peer selection in heterogeneous settings.
• MACE achieves significant regret and performance improvements, with these gains persisting even when parameters are frozen during exploitation.
• Explicit exploration is critical; prompting-based reasoning is insufficient for robust behavior.

Figure 7: Cumulative regret offset across tasks and phases: MACE-TD vs MACE. The average cumulative reward offset (Baseline -- Random) across interaction steps are shown (lower the better). The horizontal dashed line at Offset =0 is the Random baseline, and the shaded areas indicate the standard error across participating agents.

Theoretical Analysis: Regret Guarantees and Diversity Dependence

Rigorous analysis demonstrates that MACE achieves $O(\sqrt{T \log T})$ regret, while any purely greedy (non-exploring) policy suffers at least $\Omega(\delta T)$ regret, where $\delta$ quantifies capability diversity in the agent pool.

Figure 8: Illustration of Theorem. The theoretical regret bounds (left) align with the empirical behavior of Qwen2.5-7B-Instruct on the GPQA benchmark (right).

Key implication: the benefit of exploration grows linearly with agent diversity. In environments where agents or contexts are highly heterogeneous, the cost of myopic partner selection compounds, and systems relying solely on explicit behavioral prompts or majority-vote aggregation protocols are especially vulnerable to failure.

Analysis of Extended Frameworks and Ablations

MACE’s benefits persist under transfer to unseen benchmarks (generalizable feature learning). Temporal-Difference (TD) extensions (MACE-TD) potentially induce emergent cooperation via diversity of peer coverage (Figure 9), further supporting the case for explicit exploration strategies in complex agent societies.

Figure 9: The trend of peer coverage throughout interaction rounds.

Sensitivity analysis with respect to the exploration bonus parameter $\alpha$ (Figure 10) confirms the need for careful calibration—but shows robust trends across datasets and agent pools.

Implications, Limitations, and Future Directions

The findings have direct implications for the engineering of scalable, reliable open-agent ecosystems, including digital workforces, autonomous tool orchestration, and mixed human-AI teams. Without explicit algorithmic exploration, even very strong LLMs are liable to premature partner-lock-in, leading to groupthink, reduced adaptability, and lower systemic utility.

Key limitations include scaling behavior beyond medium-agent populations ($N$ very large), impact in settings with severe communication and feedback constraints, and the need for scalable, decentralized exploration mechanisms.

Anticipated future directions include:

• Extending exploration frameworks for very-large $N$, dynamic population settings.
• Investigating emergent specialization, division of labor, and cooperative structures in large-scale LLM agent societies.
• Integrating exploration with alignment, fairness, and system-level safety protocols for deployment in critical, high-stakes domains.

Conclusion

This work provides detailed empirical and formal evidence that existing LLM agents consistently fail at reliable peer exploration in multi-agent contexts, with consequences for regret, performance, and coordination. The MACE framework, by introducing principled, contextual exploration incentives, substantially improves both exploration metrics and downstream outcomes across diverse settings. Theoretical results establish that the value of such exploration scales with system diversity. As multi-agent LLM deployments become mainstream, explicit exploration mechanisms will be prerequisite for robust, adaptive, and trustworthy collective autonomy.