CollabSim: A CSCW Multi-Agent Simulator

Updated 6 July 2026

CollabSim is a CSCW-grounded simulation framework designed to investigate collaborative competence among LLM agents through controlled multi-agent experiments.
It operationalizes CSCW constructs like common ground, information visibility, and group size as configurable experimental variables to diagnose coordination breakdowns.
Its dual-layer architecture, featuring an interaction layer and a control layer, standardizes protocols and supports causal analysis of multi-agent collaboration.

Searching arXiv for CollabSim and closely related collaborative simulation frameworks to ground the article in current research. Search query: CollabSim collaborative competence LLM agents CSCW multi-agent experiments CollabSim is a CSCW-grounded simulation framework for investigating the collaborative competence of LLM agents through controlled multi-agent experiments. It is designed for settings in which agents coordinate through text-based channels and may fail not because they lack individual task-solving ability, but because they cannot reliably establish common ground, maintain shared task understanding, balance individual and collective incentives, or repair misalignment as interaction unfolds. In contrast to outcome-centric multi-agent benchmarks and single-agent proficiency tests, CollabSim treats collaboration as a process-level phenomenon and operationalizes classic CSCW constructs as configurable experimental variables, probing instruments, and task-specific measures (Chen et al., 4 Jun 2026).

1. Conceptual basis and research problem

CollabSim addresses a specific gap in the evaluation of LLM-based multi-agent systems. The framework is motivated by the claim that many current systems are assessed primarily through task outcomes or isolated reasoning ability, even though collaborative failure often arises from breakdowns in interaction rather than deficiencies in individual competence. The target construct is collaborative competence, defined conceptually as the capacity to maintain common ground, preserve shared task understanding, balance personal and group incentives, and repair misalignment during collaboration (Chen et al., 4 Jun 2026).

The framework is explicitly grounded in CSCW. The paper ties its design to Clark’s common ground and grounding theory, shared mental model theory, CSCW studies of media richness, information visibility, and team structure, and work on misalignment repair and descriptive coordination. Within this perspective, collaboration is not reducible to message production or final task completion. It is a structured interactional achievement shaped by communication constraints, awareness mechanisms, and the distribution of information across agents (Chen et al., 4 Jun 2026).

A central implication is that multi-agent evaluation must expose the interaction process. CollabSim therefore emphasizes what agents do, how interaction evolves, and what agents report about the current situation, partner intent, and intended next action. This suggests a methodological shift from judging whether a team succeeded to analyzing how coordination was achieved or lost.

2. Formal experiment model and system architecture

CollabSim formalizes a collaboration experiment as

$\mathcal{E} = (\mathcal{A}, \mathcal{S}, \{\mathcal{X}_i\}_{i=1}^n, \{\mathcal{O}_i\}_{i=1}^n, T),$

where $\mathcal{A}=\{a_1,\ldots,a_n\}$ is the set of agents, $\mathcal{S}$ is the state space, $\mathcal{X}_i$ is the action space for agent $a_i$ , $\mathcal{O}_i$ is the observation space for agent $a_i$ , and $T$ is the termination condition. At each turn $t$ , the state is $s^t \in \mathcal{S}$ , each agent receives an observation $\mathcal{A}=\{a_1,\ldots,a_n\}$ 0, selects an action $\mathcal{A}=\{a_1,\ldots,a_n\}$ 1, and the environment transitions to $\mathcal{A}=\{a_1,\ldots,a_n\}$ 2 (Chen et al., 4 Jun 2026).

The architecture has two layers. The interaction layer governs perception and action. Its key abstraction is the Agent Context Protocol, which provides each agent with a public update reflecting the current shared state filtered through visibility rules, plus a private state containing agent-specific information. This design allows disparate collaborative tasks to share a common interface while preserving asymmetries in knowledge and role structure (Chen et al., 4 Jun 2026).

The control layer orchestrates experiments through a config-driven loop. The Experiment Controller reads a .yml configuration, initializes state, action spaces, roles, prompts, and conditions, constructs observations, queries agents for actions, validates actions against legal action spaces, applies state transitions, triggers the probing module after each action, and logs all events in structured JSON. This makes CollabSim reproducible and suitable for controlled comparison across models, agent designs, and environmental conditions (Chen et al., 4 Jun 2026).

The framework’s architecture is significant because it treats collaboration as an experimentally manipulable system property rather than an incidental by-product of prompting. A plausible implication is that CollabSim functions as an evaluation instrument as much as a simulator: it standardizes interaction, intervention, and trace collection in a way that supports causal analysis of condition effects.

3. Controlled manipulation of interaction conditions

A defining feature of CollabSim is its translation of CSCW-inspired constructs into explicit experimental knobs. The paper identifies three condition categories: communication bandwidth, information visibility, and group size (Chen et al., 4 Jun 2026).

Communication bandwidth controls how much and how often agents can communicate. It is used to model channel constraint and media richness effects, including fewer or shorter messages, lower frequency, and more constrained exchange. Information visibility controls which shared-state facts are visible beyond private observations. In Shape Factory, an awareness dashboard can reveal teammates’ money, production number, order progress, and specialty; in Map Task, the guide can see the follower’s canvas progress. Group size varies the number of agents and is used to study how coordination overhead changes as teams scale (Chen et al., 4 Jun 2026).

The condition table in the paper associates these manipulations with specific tasks. Shape Factory includes baseline, bandwidth, group size, and visibility conditions. DayTrader includes baseline, bandwidth, and group size. Hidden Profile includes baseline and bandwidth. Map Task includes baseline, bandwidth, and visibility (Chen et al., 4 Jun 2026).

This experimental structure distinguishes CollabSim from benchmarks that hold communication and observability fixed. The framework assumes that collaboration quality is contingent on interaction conditions, not merely on model capability. That assumption is borne out by the reported experiments, which show systematic condition effects across tasks and models.

4. Tasks, probes, and evaluation metrics

CollabSim instantiates four CSCW-inspired tasks, each targeting a distinct collaborative mechanism. Shape Factory studies resource coordination under interdependence: each agent has a specialty shape that is cheaper to produce, orders never include one’s own specialty, and agents must negotiate trades and coordinate production. DayTrader studies incentive balancing in a social dilemma: each round, agents choose individual or group investment; individual investment yields private returns, group investment is tripled and shared, and top earners receive a bonus. Hidden Profile studies information pooling under asymmetry: each agent holds only partial evidence, and the optimal answer is unavailable from any single private document. Map Task studies referential grounding in asymmetric spatial communication: the guide sees the full route, while the follower sees only landmarks and must reconstruct the route from text (Chen et al., 4 Jun 2026).

The DayTrader incentive structure includes an explicit bonus rule. At the end of each decision phase, the round leader or leaders receive a bonus of \$90, split evenly among winners: $\mathcal{A}=\{a_1,\ldots,a_n\}$ 3 This mechanism increases tension between private and collective rationality (Chen et al., 4 Jun 2026).

The framework’s probing module is an important methodological contribution. After every action, it queries agents about three dimensions: perceived task state, perceived teammate intent, and self-reasoning or intended action. Example prompts include “At this moment, how do you assess the current situation?”, “At this moment, what do you think the other participants are trying to do?”, and “At this moment, what do you plan to do?” The wording is adapted for Hidden Profile and Map Task according to their collaboration structure (Chen et al., 4 Jun 2026).

From these probes, CollabSim derives a self-reported confidence score about shared task understanding and a pairwise response similarity measure using SBERT cosine similarity. The latter functions as a proxy for internal alignment or common ground. The framework therefore evaluates collaboration at three levels: outcome metrics, process metrics, and probing metrics (Chen et al., 4 Jun 2026).

The paper specifies task-level process measures as follows.

Task	Outcome metrics	Process metrics
Shape Factory	average wealth	Trade accept rate; Order fulfillment rate; Message-trade ratio
DayTrader	average wealth / net return	Cooperation rate; Average group pool size; Total messages
Hidden Profile	final vote accuracy	Vote change rate; Mention rate of the key candidate; Average message length
Map Task	route drawing accuracy	Communication efficiency; Drawing revision rate; Total messages

For Hidden Profile, the appendix defines the key-candidate mention rate as the fraction of messages that both contain a paraphrase of Candidate C’s private clues and refer to Candidate C via a standalone letter “C.” This metric operationalizes whether agents surface distributed evidence rather than settling prematurely on a conclusion (Chen et al., 4 Jun 2026).

5. Experimental setup and empirical findings

The paper evaluates CollabSim on four LLM backbones: Qwen3.6-35B-A3B, Llama-4-Maverick-17B-128E-Instruct-FP8, GPT-5.5, and Claude 4.6 Sonnet. For each model, it compares two agent designs: persona-based agents, which receive a demographic or persona prompt plus task instructions but no explicit collaboration theory, and Collaboration-Theory-Informed agents, which are instructed with CSCW concepts including common ground, shared mental models, transactive memory, and Gricean maxims (Chen et al., 4 Jun 2026).

Several condition effects are reported. Limited communication bandwidth reduces cooperation across tasks: lower cooperation in DayTrader, reduced message-trade ratio in Shape Factory, lower mention rate of key candidate information in Hidden Profile, and lower communication efficiency in Map Task. Increased information visibility raises engagement, with higher trade acceptance in Shape Factory and higher drawing revision rates in Map Task, but does not always improve final outcomes. Group size produces mixed effects: in Shape Factory, larger groups increase wealth but reduce order fulfillment; in DayTrader, larger groups can increase confidence while lowering real cooperation, especially for theory-informed agents at larger sizes (Chen et al., 4 Jun 2026).

Backbone-specific patterns are also prominent. GPT-5.5 is reported as the most consistently strong overall and top-tier in several tasks, though not uniformly best. Claude 4.6 is very strong in DayTrader, particularly as group size grows, but performs poorly in Hidden Profile, failing in both conditions. Qwen3.6 is highly condition-sensitive, improving in some constrained or visibility-enhanced settings but collapsing in others, with especially poor performance in some Shape Factory settings. Llama-4 is the weakest overall, often reaching zero fulfillment or very low task performance in harder collaborative settings (Chen et al., 4 Jun 2026).

A notable result is that theory-informed prompting does not uniformly improve collaboration. It often helps in Shape Factory by improving acceptance rates, and in baseline DayTrader by increasing cooperation across all backbones. Yet it can also hurt: in Hidden Profile under limited bandwidth, theory-informed prompting can reduce accuracy dramatically, even from 100% to 0% for some models; in DayTrader with 6 agents, it can sharply reduce wealth and cooperation for Llama-4 (Chen et al., 4 Jun 2026).

The probing analysis reveals additional structure. Agents’ grounding confidence generally rises over time. Average alignment is highest in DayTrader, then Hidden Profile, then Shape Factory, and lowest in Map Task. Across three of four tasks, task-state alignment is highest and partner-intent alignment is lowest. Group size affects alignment differently by task: in Shape Factory, alignment decreases as group size grows; in DayTrader, alignment increases with group size, but this may reflect convergence on defection rather than better collaboration (Chen et al., 4 Jun 2026).

These findings undermine two common assumptions. First, high outcome performance does not guarantee collaborative competence. Second, higher alignment is not inherently beneficial, because alignment may form around a strategically harmful equilibrium. The framework therefore treats outcome, process, and internal-state measures as complementary rather than interchangeable.

CollabSim belongs to a broader class of collaborative simulation and evaluation systems, but its emphasis is distinctive. VirT-Lab focuses on flexible, customizable, large-scale multi-agent team simulation in spatial and temporal environments, with explicit support for space, time, roles, goals, communication, and evaluation (Almutairi et al., 6 Aug 2025). ReCoLLAB studies cooperative ad-hoc teammate modeling by classifying teammate types from short interaction prefixes and routing to best-response policies, with retrieval grounding improving robustness under partial observability (Wallace et al., 5 Dec 2025). MoCoMR generates synthetic but realistic collaborative mixed-reality logs by modeling speaking, gaze, and locomotion and assembling them into group interaction graphs (Romero et al., 12 Mar 2025). CollabBench evaluates and trains collaborative LLM agents in grounded cooperative games under diverse simulated player profiles, balancing efficiency with affective adaptation (Qian et al., 4 Jun 2026).

Relative to these systems, CollabSim is centered less on scalable simulation, teammate-type inference, multimodal behavior synthesis, or benchmark training, and more on controlled experimental diagnosis of collaborative competence. This suggests that its natural role is methodological: it provides a framework for asking why multi-agent collaboration succeeds or fails under manipulated interaction conditions, rather than primarily a platform for deploying realistic team environments or training socially adaptive policies.

The paper also identifies several limitations. Task coverage is limited to four paradigmatic CSCW tasks, though future additions could include Desert Survival Task and passcode game. Model and agent coverage is limited to four LLMs and two agent designs; future work could extend to reasoning-tuned models, smaller open-source models, and memory-augmented or planner-based agents. Metric granularity is representative rather than exhaustive, leaving room for measures such as repair frequency, turn-taking balance, finer grounding behaviors, and more detailed collaboration traces. The probing module relies on self-reports, which may diverge from actual decision processes, although the paper treats this discrepancy itself as diagnostically meaningful (Chen et al., 4 Jun 2026).

A recurrent misconception is that multi-agent systems fail mainly because constituent models are weak reasoners. CollabSim argues instead that failure often lies in collaborative process: failure to coordinate around a shared goal, failure to balance individual and group incentives, or failure to ground task-relevant information. Another misconception is that adding collaboration theory to prompts should monotonically improve team performance. The reported results do not support that view; the effect of theory guidance is task-dependent and condition-dependent (Chen et al., 4 Jun 2026).

In this sense, CollabSim reframes evaluation of LLM-based multi-agent systems around a CSCW premise: collaboration is an interactional accomplishment shaped by common ground, visibility, structure, and repair. Its contribution lies in turning that premise into an experimental methodology with formal task structure, controlled condition manipulation, action-level probes, and process-sensitive metrics.