CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2201.06796v2)

Abstract: Large LMs offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs' generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3's capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities, and reveal its contribution as a writing "collaborator" under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs' promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.

• The paper introduces the CoAuthor dataset, capturing 1445 sessions of human-AI interactions to assess collaborative writing performance.
• The paper demonstrates GPT-3’s proficiency in enhancing textual fluency and generating diverse vocabularies, improving both creative and argumentative content.
• The paper reveals variable collaboration patterns, where randomized AI suggestions foster greater engagement and offer insights for personalized writing tools.

Insights from the CoAuthor Dataset: Exploring Human-AI Collaborative Writing

The paper "CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring LLM Capabilities" introduces the CoAuthor dataset, which seeks to enhance understanding of AI's role in collaborative writing environments. By amassing and examining interaction data between writers and AI models, specifically GPT-3, the study underscores the importance of datasets in revealing the multifaceted capabilities of LMs under human-computer interaction (HCI).

The CoAuthor dataset comprises interactions between 63 participants and four GPT-3 instances, spanning 1445 writing sessions. By focusing on creative and argumentative writing tasks, the dataset evaluates GPT-3's language, ideation, and collaborative capabilities. Such a dataset is a valuable resource for illuminating the nuances of human-AI collaboration, particularly in interactive settings.

Key Findings on AI Capabilities

1. Language Capabilities: GPT-3 demonstrates proficiency in generating grammatically sound and fluent text. The paper reports fewer grammatical errors in sentences written by both GPT-3 and humans compared to those penned by humans alone. Furthermore, combining human and AI contributions resulted in notably more diverse vocabulary.
2. Ideation Abilities: By analyzing the generation and reuse of new ideas, the study finds that GPT-3 contributes creatively to writing. About 13% of GPT-3's suggestions in creative writing included novel named entities, with 20% of these being reused by writers, indicating GPT-3's role in enhancing ideational breadth.
3. Collaboration Patterns: The dataset explores how writers engage with AI. It notes significant variability across writers in terms of mutuality (level of interaction) and equality (balance of content). Randomness in AI-generated suggestions also influenced the degree of collaboration, with higher randomness generally correlating with increased interaction in argumentative tasks.

Methodological Contributions

The CoAuthor dataset was meticulously designed to cover diverse contexts, support subjective interpretations, and capture the writing process rather than merely its outcomes. These aspects render it a resourceful tool for assessing AI-generated interactions in versatile environments.

The paper suggests reconfigurable instances of writing sessions, encouraging future research to tailor AI capabilities according to specific design objectives. This flexibility supports long-term applicability and extension of the dataset, accommodating rapid advancements in LMs.

Implications and Future Directions

The implications of the CoAuthor study are profound both theoretically and practically:

• Theoretical Insights: It illuminates the nuanced capabilities of GPT-3 and similar LMs in the context of collaborative writing, advancing scholarly discourse on AI's role in augmenting human creativity and productivity.
• Practical Applications: Interaction designers can leverage insights from this dataset to create more effective AI writing tools, enhancing user engagement and productivity. Understanding variability in AI-writer collaboration can inform personalized writing assistants that cater to individual user preferences.
• Future Research: Speculation on future studies could explore more intricate nuances of collaborative dynamics, assess longitudinal changes in user behavior, and expand datasets to encompass other writing genres and tasks.

The CoAuthor dataset, as presented in this paper, serves as a valuable benchmark and resource, fostering a deeper comprehension of the intricacies and potential of human-AI creative partnerships in writing. By bridging gaps between HCI and NLP perspectives, it invites further interdisciplinary exploration into optimizing AI's impact on human productivity and innovation in creative endeavors.