Towards Exploiting Background Knowledge for Building Conversation Systems (1809.08205v1)

Abstract: Existing dialog datasets contain a sequence of utterances and responses without any explicit background knowledge associated with them. This has resulted in the development of models which treat conversation as a sequence-to-sequence generation task i.e, given a sequence of utterances generate the response sequence). This is not only an overly simplistic view of conversation but it is also emphatically different from the way humans converse by heavily relying on their background knowledge about the topic (as opposed to simply relying on the previous sequence of utterances). For example, it is common for humans to (involuntarily) produce utterances which are copied or suitably modified from background articles they have read about the topic. To facilitate the development of such natural conversation models which mimic the human process of conversing, we create a new dataset containing movie chats wherein each response is explicitly generated by copying and/or modifying sentences from unstructured background knowledge such as plots, comments and reviews about the movie. We establish baseline results on this dataset (90K utterances from 9K conversations) using three different models: (i) pure generation based models which ignore the background knowledge (ii) generation based models which learn to copy information from the background knowledge when required and (iii) span prediction based models which predict the appropriate response span in the background knowledge.

Exploiting Background Knowledge for Conversation Systems

The paper "Towards Exploiting Background Knowledge for Building Conversation Systems" presents a novel dataset and an exploration of models to enhance dialog systems by leveraging background knowledge. This paper addresses a fundamental limitation in existing dialog system datasets: the absence of explicit background knowledge linked to conversational utterances. This lack has resulted in models that treat conversation merely as a sequence-to-sequence generation task, which diverges considerably from the human approach of using background knowledge during conversations.

Novel Dataset

The authors introduce a new dataset comprising movie chats drawn from a variety of structured and unstructured background knowledge resources such as plots, reviews, comments, and fact tables. The dataset includes 90K utterances drawn from 9K conversations, pertaining to 921 movies. Each conversation response in their dataset is explicitly linked to sections of background knowledge, emulating the natural human tendency to reference previously acquired information during discussions. This dataset is built via crowdsourcing, with workers instructed to incorporate background knowledge into conversations, thus significantly reducing the typical noise seen in datasets extracted from online forums.

Model Evaluation

The paper evaluates three distinct paradigms of conversation models:

1. Generation-based Models: The Hierarchical Recurrent Encoder-Decoder (HRED) does not explicitly incorporate background knowledge but serves as a baseline for comparison.
2. Generate-or-Copy Models: Adaptation of the pointer-generator network, enabling models to either copy relevant segments of provided resources or generate new responses.
3. Span Prediction Models: Utilization of models like Bi-directional Attention Flow (BiDAF) which predict spans in documents relevant to conversational context, adapted from QA tasks like SQuAD.

Results and Observations

The paper's experimental results highlight several key insights:

• Span prediction models outperform the generation-based approaches in coherence and relevance, signifying the need for more structured exploitation of background knowledge rather than generation from scratch.
• The performance of span prediction models like BiDAF demonstrates limitations in handling longer document spans due to computational constraints, indicating areas for improvement in scalability.
• The generate-or-copy paradigm, while useful, showed limitations due to noise in dataset resources when irrelevant information was included, suggesting a need for better filtering and selection mechanisms in future models.

Implications and Future Directions

The authors suggest that the dataset and outlined models paving ways for new approaches in conversation systems by effectively integrating background knowledge. The insights from this paper direct future research towards hybrid models that can seamlessly integrate the summarize-or-copy capabilities with more sophisticated language generation techniques.

Additionally, the approach of creating datasets with conversations linked explicitly to background knowledge represents a significant step towards domain-specific dialogue models. Such models could be beneficial in practical applications like troubleshooting bots, educational assistance systems, and e-commerce interfaces where domain-specific knowledge is vital for meaningful interaction.

This paper invites the research community to fundamentally rethink conversation modeling, pushing for systems that rely on external information sources to drive coherent and contextually rich dialogues. Continued development in this direction could significantly enhance AI interactions and create more engaging user experiences.

Overall, while the paper highlights both promising results and areas needing improvement, its contributions in dataset creation and model evaluation offer substantive advancements in AI-powered dialog systems.