Neural Attentive Bag-of-Entities Model for Text Classification (1909.01259v2)

Abstract: This study proposes a Neural Attentive Bag-of-Entities model, which is a neural network model that performs text classification using entities in a knowledge base. Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine simple high-recall entity detection based on a dictionary, to detect entities in a document, with a novel neural attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities. We tested the effectiveness of our model using two standard text classification datasets (i.e., the 20 Newsgroups and R8 datasets) and a popular factoid question answering dataset based on a trivia quiz game. As a result, our model achieved state-of-the-art results on all datasets. The source code of the proposed model is available online at https://github.com/wikipedia2vec/wikipedia2vec.

• The paper presents the NABoE model that fuses entity recognition and neural attention to improve text classification performance.
• The methodology leverages dictionary-based entity extraction combined with attention weighting to address linking and salience challenges.
• Experimental results demonstrate state-of-the-art accuracy, achieving 86.8% on 20NG and 97.1% on R8 while excelling in factoid question answering.

Neural Attentive Bag-of-Entities Model for Text Classification

The presented paper introduces the Neural Attentive Bag-of-Entities (NABoE) model, aimed at enhancing text classification performance by leveraging entities from a knowledge base (KB). This approach departs from traditional word-based models, utilizing the clarity and semantic specificity that entities offer.

Model Overview

The NABoE model integrates entity recognition with a novel neural attention mechanism. Entities, identified via a dictionary-based approach from a KB, such as Wikipedia, are used to represent documents. The attention mechanism selectively weights these entities based on their relevance and semantic clarity, addressing both entity linking and entity salience in a unified manner. This reduces disambiguation errors and enhances the focus on entities central to the text's semantics.

Experimental Validation

The model's performance was empirically validated using two standard datasets for text classification: the 20 Newsgroups (20NG) and R8 datasets. Additionally, a factoid question answering dataset was employed to demonstrate NABoE's applicability to more varied linguistic challenges. In all cases, NABoE achieved state-of-the-art results.

Numerical Results

On the 20NG dataset, the NABoE-full model surpassed baseline methods with an accuracy of 86.8%, highlighting the efficacy of combining word and entity-based document representations. In the R8 dataset, the accuracy reached 97.1%. The results indicate that integrating entity signals with traditional word-based methods captures nuanced semantic information.

In factoid question answering, NABoE similarly excelled, with notable accuracy enhancements in both the history and literature categories compared to competitors. The NABoE-full model achieved 94.9% accuracy in history and 98.5% in literature, underscoring its capability in complex semantic interpretation tasks.

Implications and Future Directions

The NABoE model's approach highlights the potential benefits of incorporating structured KB information into text processing tasks. Its ability to focus on pertinent entities enriches semantic representation, which can be particularly advantageous in applications requiring precise understanding, like information retrieval or semantic search.

Future research might explore integrating global coherence in entity selection to further refine the attention mechanism. Additionally, adapting this model to other NLP tasks could reveal broader applications, potentially extending to areas such as machine translation or dialogue systems.

In summary, this work advances text classification by effectively employing KB entities, demonstrating substantial improvements in capturing and utilizing semantic insights from textual data. The model opens avenues for further innovations in combining KB-derived data with traditional text analysis techniques.