Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View (2310.02124v3)

Abstract: As NLP systems are increasingly employed in intricate social environments, a pressing query emerges: Can these NLP systems mirror human-esque collaborative intelligence, in a multi-agent society consisting of multiple LLMs? This paper probes the collaboration mechanisms among contemporary NLP systems by melding practical experiments with theoretical insights. We fabricate four unique societies' comprised of LLM agents, where each agent is characterized by a specifictrait' (easy-going or overconfident) and engages in collaboration with a distinct `thinking pattern' (debate or reflection). Through evaluating these multi-agent societies on three benchmark datasets, we discern that certain collaborative strategies not only outshine previous top-tier approaches, but also optimize efficiency (using fewer API tokens). Moreover, our results further illustrate that LLM agents manifest human-like social behaviors, such as conformity and consensus reaching, mirroring foundational social psychology theories. In conclusion, we integrate insights from social psychology to contextualize the collaboration of LLM agents, inspiring further investigations into the collaboration mechanism for LLMs. We commit to sharing our code and datasets\footnote{\url{https://github.com/zjunlp/MachineSoM}.}, hoping to catalyze further research in this promising avenue.

Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View

The paper investigates the potential of NLP systems, specifically LLMs, to mimic human-like collaborative intelligence within a multi-agent context. Drawing from social psychology, the paper introduces and evaluates the structures of LLM-based societies, each characterized by particular traits and thinking patterns, to explore effective collaboration methods.

Core Investigation

Central to this inquiry are four fabricated multi-agent societies, where each society consists of three agents, with each agent embodying one of two traits: easy-going or overconfident. These traits guide the agents in adhering to one of two distinct thinking patterns during collaborative tasks: debate or reflection. The discussion-focus of the former, "debate," fosters interactive exchange, potentially leading to consensus. Conversely, "reflection" offers a more introspective approach where agents revise their responses based on accumulated insights from prior interactions.

Experimental Setup

The paper evaluates these societies across three benchmark datasets: MATH, MMLU, and Chess Move Validity. Each dataset poses distinct challenges:

1. MATH assesses advanced mathematical and scientific reasoning.
2. MMLU focuses on extensive multi-task language understanding, covering high school level topics.
3. Chess Move Validity requires agents to predict the next legitimate move in a game of chess.

In these evaluations, agents demonstrate varied strategic collaboration, iteratively engaging in either debating or reflecting over several rounds.

Key Findings

1. Performance Variation Among Strategies: Collaborative strategies leveraging diverse permutations of thinking patterns show a marked range in efficiency. Specifically, strategies rooted in consistent debate outperform others, enhancing accuracy. Conversely, mere increase in either agent numbers or collaborative rounds does not guarantee improved results. Instead, the interplay between agent count and collaborative strategy often dictates efficacy.
2. Impact of Agent Traits: Interestingly, agents with easy-going traits did not provide a stark advantage over those with overconfident traits. This finding suggests that collaboration efficacy might not hinge solely on the nature of individual agent traits.
3. Social Behavior Manifestation: Similar to human societies, agents exhibit behaviors such as conformity and consensus-building, mirroring foundational social psychology theories. This not only affirms the intuitive design of the agent traits and thinking patterns but also raises potential avenues for development in the field of human-AI interaction.

Theoretical and Practical Implications

Theoretically, this research bridges the gap between machine learning and social psychology, advocating for an interdisciplinary approach to understand and enhance collaborative AI systems. It suggests that strategic collaboration rooted in human social behaviors could be key to unlocking collective intelligence in LLM societies.

Practically, the results prompt further inquiries into optimal configurations for multi-agent systems, beyond mere expansion in scale. They hint at the potential for these systems to support complex problem-solving tasks through more refined social dynamics.

Future Directions

This exploration lays groundwork for future efforts to amplify AI systems' collaborative intelligence by integrating insights from social and psychological sciences into technology development. Emphasizing small-group collaboration strategies and diverse individual characteristics may enhance AI implementations across fields such as autonomous systems, human-AI interaction, and beyond. The paper marks a critical move towards constructing AI societies capable of mimicking sophisticated human social behaviors, encouraging further innovation along these lines.