Wizard of Wikipedia: Knowledge-Powered Conversational agents (1811.01241v2)

Abstract: In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequence models typically "generate and hope" generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context. Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding. To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses. Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction.

Knowledge-powered Conversational Agents: A Summary

The paper authored by Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston from Facebook AI Research, titled "Knowledge-powered Conversational Agents," addresses the challenges and design of intelligent agents that leverage external knowledge to enhance open-domain dialogue capabilities. The paper's primary contribution is the creation of a large-scale dataset, Wizard of Wikipedia, which grounds dialogue in factual knowledge sourced from Wikipedia. Furthermore, the paper proposes several architectures and models to utilize this dataset effectively for generating knowledgeable and engaging conversational responses.

Introduction and Motivation

Current state-of-the-art sequence-to-sequence models in dialogue systems exhibit limited abilities in leveraging long-term explicit knowledge. These models typically generate responses based on the input sequence using information encoded implicitly within their parameters. The authors argue that intelligent agents must harness external knowledge bases to conduct meaningful and informative conversations. The paper sets out to collect a dataset that allows the measurement and improvement of such knowledge-grounded dialogues.

Dataset Collection: Wizard of Wikipedia

To build a suitable benchmark, the authors crowd-sourced a vast dataset of human-human dialogues, with one participant acting as a "wizard"—a knowledgeable interlocutor who has access to relevant Wikipedia snippets—and the other participant as an "apprentice" engaging in general conversation. The dataset encompasses 1365 topics spanning a broad range of subject matters, with 201,999 utterances annotated to indicate the grounding knowledge used by the wizard. This setup ensures that the dataset is rich in diverse contexts and verifiable knowledge usage.

Models and Methods

The paper details several architecture designs that utilize the Wizard of Wikipedia dataset to enhance dialogue systems:

1. Knowledge Retrieval: An information retrieval (IR) system is used to fetch relevant sentences from Wikipedia based on the dialogue context. This serves as input for our next processing stages.
2. Knowledge Attention: Transformer-based models are employed to perform fine-grained attention over the retrieved knowledge, determining the most relevant pieces of information for the given dialogue context.
3. Utterance Prediction: The final model stage involves generating the dialogue response. The paper distinguishes between two types of models: retrieval models that select responses from a predefined set of candidates and generative models that create responses word-by-word.

Retrieval Models

The Retrieval Transformer Memory Network includes several innovations:

• Combining Memory Network approaches for knowledge retrieval with Transformer architectures for encoding context and generating responses.
• Pre-training on datasets like Reddit to enhance performance on the unseen test topics.

Generative Models

Two primary variants of the generative Transformer Memory Networks are investigated:

• End-to-end: Where the system learns to attend to knowledge and generate a response within a single model.
• Two-stage: Where knowledge selection and response generation are handled as separate but sequentially dependent tasks. The two-stage model exhibited improved resilience to knowledge selection errors via a novel training technique termed "knowledge dropout."

Evaluation and Results

Automatic metrics and human evaluations are used to assess the models’ performance:

• Knowledge Selection Task: Evaluates accuracy in selecting appropriate knowledge snippets.
• Full Task Evaluation: Performance is measured in terms of recall and F1 scores when integrating knowledge into meaningful dialogue generation.

The results from the automatic evaluation show that the addition of external knowledge sources significantly enhances model performance. The human evaluation indicates that retrieval models tend to be more engaging, while generative models are adept at incorporating and leveraging new knowledge, especially in novel topics.

Implications and Future Directions

The work laid out in this paper has significant theoretical and practical implications. Integrating explicit knowledge components into dialogue systems advances the field towards more intelligent and contextually aware AI agents. The Wizard of Wikipedia dataset provides a robust benchmark for future research in knowledge-grounded dialogues. Future developments should focus on improving retrieval accuracy, integrating multi-task learning across dialogue and QA tasks, and further refining the balance between retrieval and generative model strengths to maximize both engagingness and informativeness in dialogue systems.

In summary, the research presents a comprehensive approach to enhancing conversational AI by integrating external knowledge sources, offering substantial improvements and laying the groundwork for future advancements in dialogue systems.