DeepType: Multilingual Entity Linking by Neural Type System Evolution (1802.01021v1)

Abstract: The wealth of structured (e.g. Wikidata) and unstructured data about the world available today presents an incredible opportunity for tomorrow's Artificial Intelligence. So far, integration of these two different modalities is a difficult process, involving many decisions concerning how best to represent the information so that it will be captured or useful, and hand-labeling large amounts of data. DeepType overcomes this challenge by explicitly integrating symbolic information into the reasoning process of a neural network with a type system. First we construct a type system, and second, we use it to constrain the outputs of a neural network to respect the symbolic structure. We achieve this by reformulating the design problem into a mixed integer problem: create a type system and subsequently train a neural network with it. In this reformulation discrete variables select which parent-child relations from an ontology are types within the type system, while continuous variables control a classifier fit to the type system. The original problem cannot be solved exactly, so we propose a 2-step algorithm: 1) heuristic search or stochastic optimization over discrete variables that define a type system informed by an Oracle and a Learnability heuristic, 2) gradient descent to fit classifier parameters. We apply DeepType to the problem of Entity Linking on three standard datasets (i.e. WikiDisamb30, CoNLL (YAGO), TAC KBP 2010) and find that it outperforms all existing solutions by a wide margin, including approaches that rely on a human-designed type system or recent deep learning-based entity embeddings, while explicitly using symbolic information lets it integrate new entities without retraining.

DeepType: Multilingual Entity Linking by Neural Type System Evolution

The paper "DeepType: Multilingual Entity Linking by Neural Type System Evolution" introduces a novel approach to Entity Linking (EL) that integrates symbolic information into the reasoning process of neural networks. The method, termed DeepType, aims to resolve the difficulty arising from the integration of structured and unstructured data in EL tasks, particularly when dealing with multiple languages.

Overview of DeepType Approach

DeepType employs a neural type system to explicitly integrate symbolic knowledge from ontologies like Wikidata into neural network models. This integration is achieved by reframing the design problem into a mixed integer problem, which results in a type system that constrains neural network outputs to respect a symbolic structure. The solution involves a two-step algorithm: first, a heuristic search or stochastic optimization to define the type system; second, fitting classifier parameters using gradient descent.

Key Contributions

The primary contributions of this work include:

1. Automated Type System Design: The paper introduces a system for automatically designing type systems without human intervention, tailored for target tasks such as EL.
2. Improved Entity Linking: By using type constraints, the complexity of disambiguation in EL tasks is reduced from $O(N^2)$ to $O(N)$, allowing for the incorporation of new entities without retraining.
3. Multilingual Capability: The type system designed for English generalizes well to other languages, including French, German, and Spanish. This cross-lingual application is validated through experiments showing near identical performance across languages.
4. Enhanced Named Entity Recognition (NER): DeepType provides improved performance over baselines in NER tasks, indicating cross-domain transfer potential. It demonstrates new state-of-the-art results on the OntoNotes development set.

Experimental Results

Entity Linking Performance

DeepType was tested on several datasets, including WikiDisamb30, CoNLL (YAGO), and TAC KBP 2010, outperforming existing solutions by significant margins. The system achieved disambiguation accuracies close to 99.0% on CoNLL (YAGO) and 98.6% on TAC KBP 2010, using Oracle predictions. These results imply that EL task performance can be elevated significantly with more accurate type prediction.

NER Task Transfer

Transfer learning experiments using DeepType as a pretraining mechanism for NER tasks on CoNLL 2003 and OntoNotes 5.0 showed noticeable improvements in performance metrics. Such improvements underscore the potential for symbolic information to influence neural network-based models positively across various domains.

Implications and Future Directions

The DeepType approach suggests several future research directions:

• Generalization to Other Problems: Exploring the applicability of DeepType for tasks beyond EL, such as other NLP applications where symbolic structure is integral.
• Hierarchical Type Systems: Investigating whether hierarchical type systems can bridge the accuracy gap between current models and Oracle predictions.
• Relaxing Conditional Independence: Evaluating additional performance gains by relaxing assumptions made by the type classifier’s conditional independence.

In conclusion, DeepType represents an advancement in bridging symbolic knowledge with neural network reasoning, offering improved capabilities in multilingual contexts and suggesting potential benefits for a broader range of applications in AI. The approach leverages ontology-based symbolic information to constrain neural models, paving the way for enhanced entity linking and cross-domain transfer capabilities. The release of associated code further facilitates exploration and application of these methodologies in diverse research settings.