Zero-Shot Relation Extraction via Reading Comprehension (1706.04115v1)

Abstract: We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot. This reduction has several advantages: we can (1) learn relation-extraction models by extending recent neural reading-comprehension techniques, (2) build very large training sets for those models by combining relation-specific crowd-sourced questions with distant supervision, and even (3) do zero-shot learning by extracting new relation types that are only specified at test-time, for which we have no labeled training examples. Experiments on a Wikipedia slot-filling task demonstrate that the approach can generalize to new questions for known relation types with high accuracy, and that zero-shot generalization to unseen relation types is possible, at lower accuracy levels, setting the bar for future work on this task.

• The paper introduces a novel formulation of relation extraction as a question-answering task using natural-language queries.
• It employs crowdsourced question templates and a large dataset to adapt a reading comprehension model for zero-shot learning.
• The method generalizes to unseen entities and paraphrases, demonstrating promising results with an F1 score of 41%.

Zero-Shot Relation Extraction via Reading Comprehension

This paper presents a novel approach to relation extraction (RE) by framing it as a reading comprehension task, thereby enabling zero-shot learning of new relation types at test time. The core idea is to represent each relation extraction task as answering natural-language questions. By associating one or more question templates with each relation type, the paper proposes that relation extraction can leverage recent advances in neural reading comprehension models.

Methodology

The approach involves reducing relation extraction to answering questions, where each relation type is mapped to a parametric natural-language question. For instance, the relation $educated\_at(x, y)$ could be represented by questions like “Where did $x$ paper?” Given an entity $x$ and a sentence containing $x$, the process involves extracting answers that provide the required relational information.

This formulation allows the deployment of neural reading comprehension techniques for relation extraction. Particularly, the method facilitates zero-shot learning, where the model can handle new, previously unseen relation types. At test time, only the question templates need to be provided for new relations, allowing the trained reading comprehension model to extract relevant data points.

Data Collection

The authors constructed a large reading comprehension dataset by leveraging distant supervision from the WikiReading dataset, which aligns Wikidata relations with corresponding Wikipedia texts. To generate question templates for each relation, they employed crowdsourcing techniques, ultimately creating a dataset of over 30 million question-sentence-answer examples.

Negative examples—where a question has no answer in the given sentence—are included to train the model to determine question answerability. These examples are generated by mismatching question types with sentence contents.

Model Adaptation

A state-of-the-art reading comprehension model, BiDAF, is adapted to handle the relation extraction task. It is extended to determine the absence of an answer in given text by learning a threshold that indicates whether the model is sufficiently confident in the extracted information.

Experimental Evaluation

Experiments are conducted to evaluate the performance of the proposed method in scenarios with unseen entities, unseen question templates, and unseen relations. Results indicate:

• The model generalizes well to new entities and texts.
• The ability to interpret and respond to unseen paraphrases shows successful question template generalization.
• There is notable but limited zero-shot capability in extracting unseen relations, with an F1 score reaching 41%.

The model's success hinges on its capacity to establish type-based cues and paraphrase detection, learned across various observed relations.

Implications and Future Directions

Reducing relation extraction to a query-based reading comprehension task potentially broadens the applicability of information extraction methodologies. As advancements in machine reading and QA continue, this method could be refined and extended to other complex tasks within NLP and AI applications. However, the challenge of zero-shot RE highlights areas for further research, specifically enhancing model capabilities in handling unseen relation patterns and examples.

The work represents an incremental step in the evolution of natural language understanding, laying the groundwork for smarter and more adaptive information extraction systems in diverse domains. Future exploration could focus on enhancing paraphrasing capabilities and integration with larger, more diverse datasets for robust zero-shot learning.