Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset (1909.05358v1)

Abstract: A significant barrier to progress in data-driven approaches to building dialog systems is the lack of high quality, goal-oriented conversational data. To help satisfy this elementary requirement, we introduce the initial release of the Taskmaster-1 dataset which includes 13,215 task-based dialogs comprising six domains. Two procedures were used to create this collection, each with unique advantages. The first involves a two-person, spoken "Wizard of Oz" (WOz) approach in which trained agents and crowdsourced workers interact to complete the task while the second is "self-dialog" in which crowdsourced workers write the entire dialog themselves. We do not restrict the workers to detailed scripts or to a small knowledge base and hence we observe that our dataset contains more realistic and diverse conversations in comparison to existing datasets. We offer several baseline models including state of the art neural seq2seq architectures with benchmark performance as well as qualitative human evaluations. Dialogs are labeled with API calls and arguments, a simple and cost effective approach which avoids the requirement of complex annotation schema. The layer of abstraction between the dialog model and the service provider API allows for a given model to interact with multiple services that provide similar functionally. Finally, the dataset will evoke interest in written vs. spoken language, discourse patterns, error handling and other linguistic phenomena related to dialog system research, development and design.

Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

The paper introduces Taskmaster-1, a dialog dataset created with the aim of advancing data-driven approaches to dialog system development. With the increasing prevalence of voice-based personal assistants such as Apple’s Siri, Microsoft Cortana, Amazon Alexa, and Google Assistant, there is a pressing need for realistic and diverse dialog data to enhance natural language understanding (NLU) and natural language generation (NLG) capabilities in these systems.

Dataset Composition and Collection Methods

Taskmaster-1 encompasses 13,215 task-based dialogs across six domains: ordering pizza, auto repair appointments, ride service setup, movie ticket booking, ordering coffee drinks, and making restaurant reservations. Uniquely, the dataset includes both spoken dialogs using the Wizard of Oz (WOz) approach and written self-dialogs. The WOz approach involves two participants—the user and an agent—where the agent is a trained handler interacting as a digital assistant via text converted to speech, thus simulating an automated system. This approach prioritizes natural expressions from users within a controlled interaction environment.

Conversely, the self-dialogs involve a single individual writing both sides of the conversation, encapsulating both user and assistant roles. This method has shown high content richness, facilitating scenario diversity at a fraction of the cost of spoken dialog collection.

Characteristics and Strengths of Taskmaster-1

Among the notable attributes, Taskmaster-1 covers a substantial array of realistic conversational flows and entity-rich interactions, showing a wider array compared to traditional datasets like MultiWOZ. The dataset features an average of 23 utterances per dialog, promoting rich context in interactions. Furthermore, Taskmaster-1 dialogues are labeled with API calls and arguments, which simplifies the annotation process compared to more intricate schemas, reducing the burden on annotators while maintaining utility in practical applications.

Comparative Analysis and Baseline Models

Through quantitative analysis, the self-dialogs exhibit higher perplexity and lower BLEU scores in comparison to MultiWOZ, indicating greater difficulty in modeling due to increased variability and diversity. A similar analysis between the self-dialogs and the two-person dialogs shows that while the latter contain richer interaction in terms of natural discourse, the former allows for expansive scenario exploration at a lower cost.

The paper provides several baseline models on the self-dialog corpus, including powerful architectures such as LSTM with attention, convolutional seq2seq models, and Transformer networks. The experiments reveal a strong correlation between BLEU scores and human ranking judgments, with the Transformer model outperforming others in automatic metrics.

Theoretical and Practical Implications

The Taskmaster-1 dataset's focus on realistic yet diverse task-based dialogs serves as a pivotal resource for the development and enhancement of dialog systems. By providing a robust benchmark complete with API-based annotations, the dataset facilitates research in dynamic dialog modeling, error handling, and contextual NLU/NLG. Future development in AI can leverage Taskmaster-1 to refine multi-domain conversational capabilities, ultimately advancing the integration of AI in everyday task fulfiLLMent scenarios.

In conclusion, Taskmaster-1 presents an invaluable contribution to the field of dialog system research, emphasizing realistic dialog generation and cross-domain functionality crucial for the next generation of automated assistants.