CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments

Abstract: Multi-agent systems (MAS) built on LLMs have shown growing promise, with their effectiveness resting on agents' ability to coordinate through text-based channels much as human teams do. Yet recent study suggests that MAS often falter not because agents lack individual task-solving ability, but because they lack collaborative competence: the capacity to establish common ground, maintain shared task understanding, balance individual and collective incentives, and repair misalignment as interaction unfolds. Decades of research in Computer-Supported Cooperative Work have characterized these requirements for human teams coordinating under constrained communication, yet existing MAS evaluations focus mainly on task outcomes or single-agent proficiency in reasoning, planning, and tool use. To enable a systematic analysis of agents' collaborative competence in MAS, we introduce CollabSim, a configurable simulation framework that combines a theory-grounded definition of collaborative capabilities, controlled manipulation of interaction conditions, and action-level probing of agents' internal states. Experiments across four LLMs show that CollabSim can capture condition effects, separate model performance patterns, and reveal task-dependent effects of agent design.

• The paper presents CollabSim, a simulation framework enabling fine-grained examination of collaborative competence in LLM multi-agent systems using CSCW paradigms.
• The study systematically manipulates communication bandwidth, information visibility, and group size to reveal their effects on task performance and cooperation.
• The results underscore the importance of process-level analysis in diagnosing coordination, incentive-alignment, and grounding failures in LLM interactions.

CollabSim: A Formal Framework for Assessing LLM Agents' Collaborative Competence

Motivation and Theoretical Foundations

As LLM-driven multi-agent systems (MAS) are increasingly leveraged for tasks requiring complex teamwork, the ability of agents to coordinate through natural language emerges as a critical bottleneck. Empirical evidence highlights that recent MAS failures are predominantly due to weak collaborative competence, rather than individual deficiencies in reasoning, planning, or tool use. These failures manifest as breakdowns in establishing common ground, maintaining shared task understanding, negotiating trade-offs between personal and collective incentives, and repairing misalignment. While decades of Computer-Supported Cooperative Work (CSCW) literature have deeply characterized such human collaboration challenges, most evaluation protocols for MAS remain myopically outcome-driven and overlook nuanced process-level dynamics.

CollabSim is introduced to address this gap by providing a controlled, theory-grounded simulation environment that enables systematic probing of collaborative competence in LLM agents. It integrates validated experimental paradigms from CSCW—such as those evaluating common ground, interdependent negotiation, social dilemmas, and distributed information pooling—to create rigorous, reproducible multi-agent testbeds.

Framework Architecture and Experimentation Methodology

CollabSim is architected to support fine-grained process-level analysis through the following components:

• Interaction and Control Layers: The interaction layer standardizes agent perception and action via a unified Agent Context Protocol, while the control layer (Experiment Controller) orchestrates experiment configuration, state updates, action validation, and per-turn cognitive probing.
• Configurable Experimental Variables: Researchers can systematically manipulate key interaction conditions:
  • Communication Bandwidth: Constraints on message length and frequency, mirroring bandwidth and channel richness studies in CSCW.
  • Information Visibility: Controls over agent access to global versus private state information, e.g., through awareness dashboards or live workspaces.
  • Group Size: Variable team sizes to gauge how collaboration scales and coordination overheads emerge.
• Probing Module: After every action, agents are queried about their mental model of the task state, perceived partner intentions, and self-reasoning. Quantitative metrics derived include self-reported confidence and inter-agent alignment (SBERT-based cosine similarity).
• Task Paradigms: Four tasks, each mapping to a classic CSCW scenario:
  • Shape Factory: Resource and order completion via negotiation under specialization and cost asymmetry.
  • DayTrader: Social-dilemma investment balancing group and individual rewards.
  • Hidden Profile: Distributed information pooling for optimal candidate selection.
  • Map Task: Referential grounding and communication without shared visual context.

Experimental Evaluation and Metrics

CollabSim was validated with state-of-the-art LLMs—Qwen3.6-35B-A3B, Llama-4-Maverick-17B-128E-Instruct-FP8, GPT-5.5, Claude 4.6 Sonnet—each instantiated as either persona-based or theory-informed agents. Rigorous experimentation examined model, agent design, and interaction condition effects across the four tasks.

Metrics captured at outcome, process, and probing levels include (by task): task-level wealth, outcome accuracy, order fulfillment, negotiation/trade acceptance, cooperation rates, information mention rates, communication efficiency, revision rates, and evolving confidence/alignment scores.

Key Empirical Findings

Impact of Interaction Conditions

• Communication Bandwidth: Limiting bandwidth reliably depressed coordination and cooperation metrics across all tasks. For example, in DayTrader, cooperation rates collapsed to near zero under restrictions. Agents failed to prioritize high-value exchanges, exposing deficiencies in collaborative grounding behaviors.
• Information Visibility: Enhanced visibility increased engagement (e.g., trade acceptances, drawing revisions) but only improved outcomes when additional action directly addressed the task's core demands (e.g., large gains in Map Task route accuracy for GPT-5.5 under canvas visibility).
• Group Size: Larger groups yielded higher aggregate wealth in Shape Factory (due to greater trading opportunity) but also lower order fulfillment, highlighting coordination strain. In DayTrader, increased group size paradoxically improved agents' reported alignment/confidence but lowered cooperative investment, suggesting agents converged on individual defection.

Model and Agent Design Sensitivity

• Model Backbone: Proprietary LLMs (GPT-5.5, Claude 4.6) generally outperformed open-source models but exhibited divergent strengths (e.g., high cooperative investment in DayTrader for Claude, high task accuracy in Hidden Profile for GPT-5.5). Open-source models (Qwen3.6, Llama-4) underperformed and were more sensitive to exact task instantiation and condition.
• Agent Design: Theory-informed agents—equipped with explicit prompts to actively construct shared mental models, align on goals, and ground communication—produced measurable deltas in process/outcome metrics. However, the sign and magnitude of these effects were highly task-dependent, and explicit theory did not guarantee superior performance (e.g., theory-informed agents in Hidden Profile under bandwidth constraint degraded accuracy for several backbones).

Process-Level Failure Modes

Through qualitative trace analysis, low performance was attributed to:

• Coordination Failures: Fragmented group strategies or fallback to individualistic heuristics, especially in large teams.
• Incentive-Alignment Failures: Groups aligning on suboptimal, individually rational strategies in social dilemmas.
• Grounding Deficiencies: Inadequate clarification, confirmation, or repair in referential and asymmetric information contexts.

Importantly, divergence between self-reported confidence (probing) and actual coordinated action was observed, directly reflecting real-world patterns of collaborative illusion.

Implications and Future Directions

CollabSim demonstrates that evaluating collaborative competence in LLM agents demands process-level instrumentation, not just downstream task metrics. The controlled manipulation of interaction variables, combined with turn-level cognitive probing, uncovers how breakdowns arise from failures in shared mental model maintenance, incentive negotiation, and dynamic realignment—phenomena not captured by conventional benchmarks.

Theoretically, CollabSim positions the study of LLM-based MAS collaboration closer to human team research, enabling more rigorous transference and comparison between artificial and human organizational dynamics.

Practically, the framework enables diagnosis of specific failure modes and benchmarking incremental advances in agent architecture, prompt engineering, and model capability. It is extensible to a larger family of tasks and interactional configurations, supporting comparative studies across agent populations, architectures, and protocols.

Future directions involve expanding the repository of collaborative paradigms (e.g., more complex survival and resource-sharing scenarios), exploring richer probing modalities (e.g., multi-faceted mental models, plan prediction), and integrating memory, planning, or reward-shaping strategies to mediate collaboration in large and dynamic teams.

Conclusion

CollabSim provides the LLM research community with a compositional, theoretically principled, and empirically validated platform for evaluating agent collaborative competence. It demonstrates that process-level analyses are essential for understanding and advancing MAS coordination. The results challenge reductionist benchmarks based solely on task completion and substantiate the need for nuanced, theory-driven frameworks in AI-driven collaboration research.

---

Reference: "CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments" (2606.06399)