This paper explores the use of Natural Language Generation (NLG) in creative writing through a paper involving 13 professional writers who used Wordcraft, an AI-powered text editor. The paper investigates the potential of NLG to impact creative writing, particularly in brainstorming, generating story details, world-building, and research assistance. The authors highlight the challenges of building AI tools for experienced writers, noting that current NLG technologies struggle to preserve style and authorial voice and lack a deep understanding of story content.
The paper involved commissioning 13 published writers from diverse backgrounds to use Wordcraft over an extended period, collecting qualitative feedback through journals and interviews. Wordcraft is a text editor with NLG-powered controls, including:
- Story seeds: Generates story ideas based on user prompts.
- Continuation: Proposes continuations to the text.
- Elaborate selection: Expands upon selected text.
- Generate text from a custom prompt: Generates text based on a user-specified prompt.
- Replace selection: Suggests alternative text for selected text (fill-in-the-blank).
- Rewrite selection: Rewrites selected text according to a specified property (e.g., "to be Shakespearean").
All of these controls are supported using in-context learning techniques, prompting a LLM with a few examplars of the task in question so that the LLM is capable of performing the task on the user's input. LaMDA was used as the underlying LLM. The prompts are formulated as a sequence of conversational turns.
The paper aimed to understand how professional writers perceive and interact with state-of-the-art language generation tools for creative writing. Unlike previous research that primarily used amateur writers and contrived user studies, this work evaluated NLG applications with a diverse audience in terms of background and expertise over an eight-week period. Participants were encouraged to incorporate the tools into their writing process as they saw fit, aligning with industry norms for commissioned works.
The paper revealed various emergent workflows and limitations. Participants approached the chatbot with diverse goals, including using it as a research assistant, a beta reader, a writing partner, and a brainstorming tool. Examples of queries participants wrote to the chatbot are shown in Table 1 in the paper.
The paper identifies several initial hopes and expectations of the writers, including using the tool as a brainstorming partner, expanding upon existing ideas, generating background details, refining story arcs, designing and keeping track of world elements, producing rewrites, generating humor, facilitating access to information, and exploring the LLM's boundaries in representing marginalized ideas and groups.
The paper discusses several limitations that participants encountered, including the difficulty of maintaining a consistent style and voice, the tendency of suggestions to revert to tropes and repetition, the fickleness of working with LLMs, the failure of the chatbot to produce meaningful and opinionated conversation, and the model's superficial natural language understanding (NLU).
The authors discuss the need for taste and intentionality in LLMs, the tradeoff between safety, sensibility, and good writing, the diversity of writers and their needs, and the question of who the target audience for AI-assisted writing tools should be. The authors suggest that developers of AI writing tools need to focus on the parts of writing that are most time-consuming and least enjoyable, and that the audience for these tools should be involved in the conversation on how the tools are developed.
The paper identifies several themes that suggest important research directions for the fields of Natural Language Processing (NLP) and Human-Computer Interaction (HCI). These include:
- The need for LLMs to exhibit taste and intentionality in their generations, moving beyond simple next-word prediction to incorporate higher-level narrative goals.
- The tradeoff between safety and sensibility, and the importance of allowing for transgression and risk-taking in creative writing tools.
- The diversity of writers and their needs, and the importance of flexibility and personalization in AI-assisted writing tools.
Participants found that their collaboration with Wordcraft worked best when they assumed the same attitude, taking Wordcraft’s disconnected and often batty suggestions as a given, then trying to make sense of them as a story.
The paper concludes by emphasizing that while AI-powered writing is unlikely to replace writers anytime soon, it holds promise for making parts of the creative writing process easier, faster, and more fun. The authors call for involving the audience for these tools in the conversation on how they are developed and suggest that future research should focus on the parts of writing that are most time-consuming and least enjoyable.