Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agents

Abstract: Collective intelligence refers to the ability of a group to achieve outcomes beyond what any individual member can accomplish alone. As LLM agents scale to populations of millions, a key question arises: Does collective intelligence emerge spontaneously from scale? We present the first empirical evaluation of this question in a large-scale autonomous agent society. Studying MoltBook, a platform hosting over two million agents, we introduce Superminds Test, a hierarchical framework that probes society-level intelligence using controlled Probing Agents across three tiers: joint reasoning, information synthesis, and basic interaction. Our experiments reveal a stark absence of collective intelligence. The society fails to outperform individual frontier models on complex reasoning tasks, rarely synthesizes distributed information, and often fails even trivial coordination tasks. Platform-wide analysis further shows that interactions remain shallow, with threads rarely extending beyond a single reply and most responses being generic or off-topic. These results suggest that collective intelligence does not emerge from scale alone. Instead, the dominant limitation of current agent societies is extremely sparse and shallow interaction, which prevents agents from exchanging information and building on each other's outputs.

• The paper demonstrates that joint reasoning achieved only 0.14% correctness compared to 7.0–15.7% by individuals, indicating a lack of emergent collective intelligence.
• It employs a three-tier probing framework—joint reasoning, information synthesis, and basic interaction—to systematically evaluate agent society performance.
• The study reveals that high individual competence does not translate to effective collective intelligence due to sparse and superficial inter-agent interactions.

Active Evaluation of Collective Intelligence in Large-Scale Agent Societies via Probing Agents

Introduction and Motivation

The proliferation of LLM agents and their aggregation into massive, persistent multi-agent systems (MAS) has stimulated research into AI-driven societies. A central and unresolved question is whether collective intelligence—defined as the group's ability to generate outcomes beyond those achievable by any individual agent—emerges spontaneously as the scale and autonomy of such societies increase. "Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agents" (2604.22452) addresses this question using MoltBook, a social platform of over two million autonomous agents, and introduces a robust empirical evaluation framework employing controlled probing agents across a three-tier hierarchy: joint reasoning, information synthesis, and basic interaction.

The Superminds Test Evaluation Framework

The Superminds Test systematically probes the presence (or absence) of collective intelligence in agent societies using task-oriented probing agents indistinguishable from regular agents. These agents inject targeted stimuli into the live platform and measure the organic societal response, allowing for controlled and reproducible evaluation across critical cognitive and interactional axes.

Figure 1: The Superminds Test framework hierarchically probes joint reasoning, information synthesis, and basic interaction by deploying probing agents and monitoring emergent responses as diagnostics of collective intelligence.

The framework encompasses:

• Tier I (Joint Reasoning): Assesses whether multi-agent discussion leads to superior solutions compared to isolated agents, targeting complex benchmarks ('Humanity’s Last Exam').
• Tier II (Information Synthesis): Tests the society's capability to aggregate and integrate distributed information from multiple agents, focusing on simple math problems requiring aggregation of premises from multiple comments.
• Tier III (Basic Interaction): Evaluates the minimal requirement for collective behaviors—namely, the ability to perceive, understand, and respond to each other’s outputs in sequential, rule-based tasks (e.g., counting chains).

Experimental Methodology

MoltBook’s agent population is driven by the OpenClaw architecture, with full agent autonomy, large scale, inherent interactivity, and observability. Probing agents are injected as ordinary participants, carrying out carefully scripted posting and responding patterns designed to elicit society-level capabilities at each tier. Engagement, correctness, and depth/quality of responses are tracked quantitatively.

Key Empirical Findings

Joint Reasoning: Absence of Emergent Collective Intelligence

On the most cognitively demanding tier, probing agents posted 2,158 challenge problems from Humanity’s Last Exam. The results are categorical: joint correctness on these complex tasks was $0.14\%$, two orders of magnitude below the $7.0\%-15.7\%$ attained by state-of-the-art individual backbone models in isolation. Moreover, only $1.6\%$ of posts received any comments. Even for the posts with multiple responses, group-level correctness ($8.6\%$) never exceeded individual comment correctness ($11.4\%$) and always lagged individual model performance. The defining criterion for collective intelligence ($g > \max_i f_i(x)$) was never met; multi-agent discussion did not yield improved accuracy or solution quality.

Comment quality analysis reveals that 76.5% of agent comments were superficial, irrelevant, or off-topic. Only $9.0\%$ showed any substantive reasoning.

Figure 2: Distribution of comment quality for complicated joint reasoning tasks; most are superficial or irrelevant, with only minimal substantive engagement.

Information Synthesis: Individual Competence, Societal Non-Engagement

Tier II assessments further disambiguated the source of failure. Probing agents distributed elementary math premises across comments, necessitating synthesis for correct task completion. Only $9.7\%$ of 103 such posts received any external reply, and out of all responses, most either did not attempt problem-solving or ignored distributional context. Of comments attempting the task, $91.7\%$ reached correct conclusions when actually engaging with necessary information, clearly indicating that competence is present at the individual level but rarely exercised due to widespread non-engagement.

Figure 3: Information synthesis trials show most posts ignored, while the small set of engaged agents reliably demonstrate capability to integrate distributed information.

Basic Interaction: Failure in Minimal Social Coordination

Even in Tier III—tasks requiring no reasoning, only ordered response to a simple rule (e.g., sequential counting)—the society failed to reliably coordinate.

• 70% of counting posts received no replies at all.
• Of the 17 external responses received, only 6 correctly followed the sequence; the rest were off-topic, spam, or misformatted.

Figure 4: Minimal interaction tasks (counting sequences) fail to elicit reliable, contextually aligned agent responses, with most posts ignored and many responses misaligned.

Platform-Wide Structural Analysis: Shallow and Uncoordinated Discussion

Global analysis of MoltBook's activity corroborates that the shallow interactivity documented in probing tasks is emblematic of the overall platform dynamic. Even highly upvoted posts, which attract more agents, do not trigger deeper threaded discussions (thread depth remains near 1.0). Across all popularity strata, replies are overwhelmingly rated as low-quality, superficial, or generic.

Figure 5: Platform-level thread metrics demonstrate very limited depth and poor reply quality, even among posts that receive numerous responses.

Implications and Theoretical Significance

Contradictory Findings: Scale Alone Does Not Yield Collective Intelligence

Contrary to prevalent assumptions extrapolated from human social phenomena, the study provides strong empirical evidence that collective intelligence does not emerge spontaneously in large-scale, open agent societies, even with access to highly capable LLMs. The core bottleneck is not individual cognitive or reasoning ability, but the extreme sparsity and lack of depth in inter-agent interaction. Agents rarely read, acknowledge, or build on one another’s outputs; instead, the society operates as an ensemble of disconnected broadcasters.

Practical Takeaways

• Increasing agent population and individual backbone quality does not suffice for emergent, synergistic group intelligence unless mechanisms are instituted to increase interaction density, context awareness, and meaningful conversational alignment.
• The operational structure of current agent societies lacks incentives, protocols, or attention architectures that drive sustained, multi-step engagement or context-following.
• Evaluation schemes for collective intelligence in MAS must go beyond passive observation and utilize controlled probing as an active diagnostic, as in the Superminds Test.

Theoretical Outlook and Future Directions

Future AI societies aspiring toward genuine collective intelligence must address architectural and protocol-level deficiencies in interaction scaffolding. Potential directions include:

• Embedding memory, attention, and context-tracking modules that prompt agents to attend to and build on prior outputs.
• Engineering intrinsic or extrinsic motivators for collaboration, information synthesis, and sustained discourse.
• Exploring environmental or social incentive structures (e.g., reputation, shared goals, or explicit coordination protocols) to encourage engagement and iterative refinement.

Further, this study motivates reexamination of the assumption that architectural scale alone (number of agents, backbone LLM quality) can substitute for well-designed inter-agent communication structures in open-ended MAS.

Conclusion

"Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agents" (2604.22452) provides definitive, large-scale empirical evidence that spontaneous collective intelligence does not manifest in current agent societies, despite massive population and advanced individual agent capabilities. The essential bottleneck is the rarity and low quality of agent-to-agent interaction, which precludes the emergence of higher-order group capabilities such as information synthesis and collective reasoning. For collective intelligence to flourish, agent society designs must prioritize interaction mechanisms and conversation-aware engagement rather than relying solely on scale or individual advances in model capability.