Collaborative Gym Framework

• Collaborative Gym is a framework for human-agent collaboration defined by asynchronous, tripartite interactions among agents, humans, and environments.
• It formalizes tasks as POMDPs with role-conditioned APIs to evaluate both task outcomes and communication processes.
• Empirical studies show collaborative agents with situational planning outperform fully autonomous agents in Travel Planning, Related Work, and Tabular Analysis.

Searching arXiv for the primary paper and closely related “gym”-style collaboration frameworks to ground the article. arXiv search query: "Collaborative Gym human-agent collaboration 2024"

Collaborative Gym, or Co-Gym, is a framework for enabling and evaluating human-agent collaboration in settings where language-model agents must work with humans rather than replace them outright. It is defined by asynchronous, tripartite interaction among agents, humans, and task environments, and by an evaluation regime that measures both outcomes and processes of collaboration. In reported real-user evaluations, collaborative agents outperformed fully autonomous counterparts in Travel Planning, Tabular Analysis, and Related Work, with win rates of 86%, 74%, and 66%, respectively (Shao et al., 2024).

1. Definition and conceptual scope

Collaborative Gym was introduced to study cases in which fully autonomous agents are insufficient because humans retain latent preferences, domain expertise, or a need for control (Shao et al., 2024). The framework therefore treats collaboration not as an auxiliary feature but as the primary problem setting. Its central premise is that many agent deployments are fundamentally joint activities in which humans and agents share authority over tools, intermediate artifacts, and final decisions.

Co-Gym differs from conventional single-agent benchmarks by making the environment explicitly multi-party. It is not simply an agent–environment loop. Instead, the environment is shared by an LM-backed agent and a human, both of whom can communicate, inspect public state, access private state, and modify shared artifacts. This arrangement yields what the framework describes as mixed authority: both human and agent can act, both can pause, and both can redirect the task.

A common misconception is that Collaborative Gym is merely a chat interface wrapped around tools. The framework is more structured than that. Each task is formalized as a partially observable Markov decision process, and the collaboration protocol includes role-conditioned actions, public and private observations, event-driven notifications, and explicit meta-actions for communication and waiting (Shao et al., 2024).

2. Formal model and asynchronous architecture

Each Co-Gym task is modeled as a POMDP

$$(\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \mathcal{U}, \mathcal{O}),$$

where $\mathcal{S}$ is the environment state, $\mathcal{A}$ is the task-specific action space, $\mathcal{T}$ is the transition function, $\mathcal{R}$ is the reward, $\mathcal{U}$ is the instruction space, and $\mathcal{O}$ is the observation space (Shao et al., 2024). Observations may contain public components, such as a shared editor or notebook, and private components, such as a role-specific search window.

The environment API is role-conditioned: $\mathcal{S}$5 Here, role identifies the acting party, action is either a task action or a collaboration action, and private indicates whether the resulting update is visible only to the acting role or should be broadcast more widely (Shao et al., 2024).

Two collaboration meta-actions are built into the framework: SendTeammateMessage and WaitTeammateContinue. The first exposes communication as part of the action space; the second allows an agent or human to yield control without changing task state. This design avoids strict turn-taking and supports asynchronous interaction. The environment can notify participants when public state changes, when private state changes, when a teammate message arrives, or when inactivity triggers a timeout event (Shao et al., 2024).

The runtime architecture consists of an EnvNode, an AgentNode, and, in simulated settings, a SimulatedHumanNode. Communication is implemented with Redis channels. EnvNode applies task actions through env.step(role, action), updates chat history for SendTeammateMessage, ignores WaitTeammateContinue in terms of task state, and then publishes role-specific observations back to the participants. This event-driven design is central to Co-Gym’s asynchronous character and distinguishes it from synchronous multi-agent frameworks (Shao et al., 2024).

3. Task environments and shared workspaces

Co-Gym was instantiated with three task families in both simulated and real-world settings: Travel Planning, Related Work writing, and Tabular Analysis (Shao et al., 2024). All three center on a shared workspace that both human and agent can inspect and edit, but each uses a different tool ecology.

In Travel Planning, the shared artifact is an itinerary in a public editor. The action set includes CitySearch, AttractionSearch, RestaurantSearch, FlightSearch, AccommodationSearch, DistanceMatrix, and EditorUpdate. The simulated backend uses TravelPlanner’s database, whereas the real setting uses Google Maps and Places APIs (Shao et al., 2024).

In Related Work, the shared state includes a public paper library and a shared editor. The action set includes SearchPaper, LibraryAddPaper, LibraryDropPaper, LibraryToDraft, and EditorUpdate. The simulated setting uses embedding-based retrieval over computer-science arXiv papers indexed before October 2024; the real setting uses the arxiv.py API (Shao et al., 2024).

In Tabular Analysis, the public workspace contains tabular data, notebook cells with outputs, and an editor for the report. The action set is minimal—JupyterExecuteCell and EditorUpdate—but the task is technically demanding because analysis quality depends on both code execution and interpretation (Shao et al., 2024).

The three instantiated tasks differ in artifact type, tool granularity, and dependence on human expertise. Travel Planning exposes hidden preferences and constraints; Related Work depends heavily on topic-specific judgement; Tabular Analysis couples executable reasoning with narrative synthesis. A concise summary appears below.

Task	Shared workspace and core actions	Real-user win rate vs autonomous
Travel Planning	Shared editor; CitySearch, AttractionSearch, RestaurantSearch, FlightSearch, AccommodationSearch, DistanceMatrix, EditorUpdate	86%
Related Work	Shared library and editor; SearchPaper, LibraryAddPaper, LibraryDropPaper, LibraryToDraft, EditorUpdate	66%
Tabular Analysis	Shared notebook and editor; JupyterExecuteCell, EditorUpdate	74%

These environments are deliberately heterogeneous. This suggests that Co-Gym is intended less as a benchmark for one narrow capability than as an instrumentation layer for studying collaboration under different mixtures of search, writing, coding, and preference elicitation.

4. Agent design and evaluation framework

Co-Gym evaluates three agent configurations: a Fully Autonomous Agent, a Collaborative Agent, and a Collaborative Agent with Situational Planning (Shao et al., 2024). The Fully Autonomous Agent can only take task-specific actions. The Collaborative Agent is given the collaboration acts in addition to task actions, but in practice tends to underuse them. The Collaborative Agent with Situational Planning adds explicit reasoning over whether to communicate, act, or wait.

Situational Planning operates in two stages. First, a Scratchpad dictionary is updated through actions such as ADD_NOTE, DELETE_NOTE, EDIT_NOTE, or DO_NOTHING. Second, a planning prompt requires the model to choose among three options: send a message, take a task action, or do nothing. Only after this plan selection is the corresponding message or task action generated (Shao et al., 2024). This architecture is designed to improve memory, coordination, and control-sharing.

Co-Gym’s outcome evaluation combines delivery and quality. Delivery Rate records whether a non-empty output is produced within the step budget. Task Performance is a task-specific scalar in $[0,1]$, computed either by scripts or by human or LM judgement. The framework defines

$\text{Collab Score} = \mathbbm{1}_{\text{Delivered}} \times \text{TaskPerformance}.$

Undelivered trajectories therefore receive zero collaboration score (Shao et al., 2024).

Process evaluation is equally central. Initiative-taking is measured by classifying utterances that advance task execution or establish mutual understanding. If $p_i$ is the proportion of initiative utterances by participant $\mathcal{S}$0, initiative entropy is

$\mathcal{S}$1

High $\mathcal{S}$2 indicates balanced initiative; low values indicate dominance or passivity (Shao et al., 2024).

Controlled autonomy is measured with two counts. $\mathcal{S}$3 is the number of agent confirmation questions that elicit a human response confirming or disconfirming the proposed action. $\mathcal{S}$4 is the number of human halting messages, such as requests to stop or redirect agent behavior. In real-user studies, the framework also records overall satisfaction on a 1–5 Likert scale focused on communication quality and usefulness (Shao et al., 2024).

5. Empirical findings and observed failure modes

In simulated evaluation, collaborative agents often showed lower delivery rates than fully autonomous agents but higher task performance among delivered cases, indicating that collaboration increases task complexity while improving the quality of successful outcomes (Shao et al., 2024). The strongest configuration was the Collaborative Agent with Situational Planning. In Travel Planning and Related Work, it produced statistically significant gains in task performance over same-model autonomous baselines for multiple LLMs. In Tabular Analysis, improvements were smaller but still present for the strongest models.

In real-user evaluation, GPT-4o with Situational Planning achieved human-preferred outcomes across all three tasks. Travel Planning reached a human-rated collaboration score of 0.788 and a win rate of 86% over the autonomous baseline; Tabular Analysis reached 0.804 and 74%; Related Work reached 0.604 and 66% (Shao et al., 2024). Reported overall satisfaction was 3.78 for Travel Planning, 4.06 for Tabular Analysis, and 3.06 for Related Work, all on a 1–5 scale (Shao et al., 2024). The process metrics also varied by task: initiative entropy was highest in Travel Planning, lower in Related Work, and intermediate in Tabular Analysis, reflecting different balances of agent proactivity and human oversight.

The framework also produced a detailed failure taxonomy. Communication failures were the most frequent, appearing in 65% of real trajectories and 80% of simulated trajectories. Situational Awareness failures appeared in 40% of real and 47% of simulated trajectories; Planning failures in 39% and 43%; Environment Awareness failures in 28% and 13%; Personalization failures in 16% and 11% (Shao et al., 2024). Typical errors included working silently, failing to ask clarifying questions, repeating earlier queries, ignoring user constraints, misusing available tools, and failing to carry preferences forward through a session.

These findings support two broader conclusions. First, human-agent collaboration can materially improve final outcomes relative to full autonomy. Second, the performance gap depends on process quality: communication capabilities, situational awareness, and the balance between autonomy and human control remain the dominant bottlenecks (Shao et al., 2024).

6. Relation to adjacent “gym” frameworks and broader significance

Collaborative Gym sits within a broader family of “gym”-style research infrastructures, but its contribution is unusually explicit about human participation as a first-class component. Relative to OpenAI Gym and standard POMDP benchmarks, it introduces role-conditioned stepping, public and private observations, and explicit communication acts (Shao et al., 2024). Relative to PettingZoo and other Markov-game frameworks, it replaces synchronous turn structure with asynchronous, event-driven collaboration. Relative to autonomous web-agent benchmarks such as WebArena and SWE-bench, it treats human oversight as part of the environment rather than an external evaluator (Shao et al., 2024).

The term “collaborative gym” has also been used more broadly across other domains. In human–robot collaboration, "Human-Robot Gym" (Thumm et al., 2023) provides eight safe RL tasks with a provable safety shield for human safety. In optical-tweezer microrobotics, "Interactive OT Gym" (Tan et al., 27 May 2025) combines haptic teleoperation, RL, and context-aware shared control for cooperative cell manipulation. In social simulation, "NegotiationGym" (Mangla et al., 5 Oct 2025) provides multi-agent negotiation and cooperation episodes with agent-specific utility functions and self-optimization hooks. This suggests that “collaborative gym” has become a general research pattern: a structured environment in which collaboration itself, rather than isolated performance, is the object of study.

Co-Gym’s specific significance lies in formalizing that pattern for human–LM teamwork. It offers a task environment abstraction, a collaboration protocol, and a dual evaluation regime for outcome and process (Shao et al., 2024). Its limitations are equally clear: only three task families were instantiated, simulated humans remain only approximations of real collaborators, and current collaborative agents still fail frequently on communication, planning, and personalization. The framework’s future direction follows directly from those limits: more task coverage, stronger collaborative agents, improved memory and planning, and broader real-world deployment (Shao et al., 2024).