CPS-TaskForge: Generating Collaborative Problem Solving Environments for Diverse Communication Tasks (2408.08853v2)

Abstract: Teams can outperform individuals; could adding AI teammates further bolster performance of teams solving problems collaboratively? Collaborative problem solving (CPS) research commonly studies teams with two agents (human-human or human-AI), but team research literature finds that, for complex tasks, larger teams are more effective. Progress in studying collaboration with more than two agents, through textual records of team interactions, is hindered by a major data challenge: available CPS corpora are predominantly dyadic, and adapting pre-existing CPS tasks to more agents is non-trivial. We address this data challenge by developing a CPS task generator, CPS-TaskForge, that can produce environments for studying CPS under a wide array of conditions, and releasing a CPS task design checklist grounded in the theoretical PISA 2015 CPS framework to help facilitate the development of CPS corpora with more agents. CPS-TaskForge takes the form of a resource management (tower defense) game, and different CPS tasks can be studied by manipulating game design parameters. We conduct a case study with groups of 3-4 humans to validate production of diverse natural language CPS communication in a game instance produced by CPS-TaskForge. We discuss opportunities for advancing research in CPS (both with human-only and human-AI teams) using different task configurations. We will release data and code.

CPS-TaskForge: Generating Collaborative Problem Solving Environments for Diverse Communication Tasks

The paper "CPS-TaskForge: Generating Collaborative Problem Solving Environments for Diverse Communication Tasks" introduces a novel tool for generating environments conducive to the paper of Collaborative Problem Solving (CPS) across various conditions—CPS-TaskForge. This tool aims to address the existing data challenges in CPS research, particularly the paucity of open data involving more than dyadic interactions.

CPS-TaskForge is predicated on the notion that CPS tasks with more complex team compositions can yield richer insights, especially with the integration of AI teammates. The primary innovation is a CPS task generator that produces environments using a resource management game paradigm (specifically, a tower defense game). This platform can manipulate different CPS task design parameters, therefore allowing researchers to paper CPS in diverse settings. Furthermore, the authors provide a CPS task design checklist grounded in the PISA 2015 CPS framework which facilitates the systematic development of CPS environments.

Key Contributions:

1. Identification of Research Gaps:
   • The paper underscores the necessity of research on teams with more than two agents. It argues that existing empirical CPS studies largely focus on dyadic interactions, thereby neglecting the diversity and complexity of larger team dynamics.
   • Furthermore, it emphasizes that datasets involving human-AI teams are limited, primarily due to the lack of adaptable, scalable task generators.
2. Introduction of CPS-TaskForge:
   • CPS-TaskForge is capable of generating a variety of CPS task environments. The primary task environment is a tower defense game which involves planning and resource management—both critical aspects of CPS.
   • The tool allows for the adjustment of various CPS design parameters such as team composition, communication modality, task difficulty, resource allocation, and more, thereby enabling detailed and nuanced studies of CPS processes.
3. Case Study Validation:
   • The authors conducted a case paper with groups of 3-4 humans, demonstrating the tool’s ability to generate diverse and natural language CPS communication.
   • The paper explored various CPS skills, including social skills (maintaining communication, sharing information) and cognitive skills (planning, executing actions).

Numerical Results and Observations:

1. Diverse Strategies and CPS Skills:
   • Participants employed different strategies for resource management, showcasing the task environment's flexibility and its capacity to elicit varied CPS behaviors. One team might focus on concentrated defenses, while another might distribute resources across the map.
   • Communication was rich, with cognitive skills employed 49% of the time and planning activities comprising 29% of team interactions, supporting the tool’s efficacy in generating meaningful CPS data.
2. Positive Participant Reception:
   • Participants noted the task's engaging nature, with many expressing interest in playing outside of the paper context, validating the design requirement for the task to be entertaining and motivation-inducing.

Implications and Future Directions:

The development of CPS-TaskForge has significant theoretical and practical implications. Theoretically, the tool enhances our understanding of team dynamics and collaborative problem-solving processes in multi-agent settings, including human-AI teams. Practically, it provides a scalable and adaptable platform for researchers to systematically explore various facets of CPS.

Future Developments:

• Incorporation of more sophisticated game mechanics to simulate even more diverse CPS scenarios.
• Extension to support different communication modalities and longitudinal studies to observe CPS processes over time.
• Exploration of human-AI interaction paradigms, particularly focusing on trust, reliance, and efficacy in hybrid teams.

Concluding Remarks:

CPS-TaskForge represents a substantial step forward in CPS research, offering a versatile and theoretically grounded platform for generating CPS task environments. This tool opens new avenues for studying collaborative efforts in more complex, larger team settings and promotes a deeper integration of AI in team problem-solving activities.