Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach (1909.00161v1)

Abstract: Zero-shot text classification (0Shot-TC) is a challenging NLU problem to which little attention has been paid by the research community. 0Shot-TC aims to associate an appropriate label with a piece of text, irrespective of the text domain and the aspect (e.g., topic, emotion, event, etc.) described by the label. And there are only a few articles studying 0Shot-TC, all focusing only on topical categorization which, we argue, is just the tip of the iceberg in 0Shot-TC. In addition, the chaotic experiments in literature make no uniform comparison, which blurs the progress. This work benchmarks the 0Shot-TC problem by providing unified datasets, standardized evaluations, and state-of-the-art baselines. Our contributions include: i) The datasets we provide facilitate studying 0Shot-TC relative to conceptually different and diverse aspects: the topic'' aspect includessports'' and politics'' as labels; theemotion'' aspect includes joy'' andanger''; the situation'' aspect includesmedical assistance'' and ``water shortage''. ii) We extend the existing evaluation setup (label-partially-unseen) -- given a dataset, train on some labels, test on all labels -- to include a more challenging yet realistic evaluation label-fully-unseen 0Shot-TC (Chang et al., 2008), aiming at classifying text snippets without seeing task specific training data at all. iii) We unify the 0Shot-TC of diverse aspects within a textual entailment formulation and study it this way. Code & Data: https://github.com/yinwenpeng/BenchmarkingZeroShot

• The paper presents a unified benchmarking framework that standardizes datasets and evaluations for zero-shot text classification.
• It recasts classification as an entailment problem by generating hypotheses from label definitions, improving prediction for unseen labels.
• Experimental results highlight an ensemble of entailment-trained models outperforming traditional methods across topic, emotion, and situation detection.

Benchmarking Zero-shot Text Classification: Datasets, Evaluation, and Entailment Approach

The paper "Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach" addresses the underexplored area of zero-shot text classification (0shot-tc). This field endeavors to assign appropriate labels to text across various domains and aspects such as topic, emotion, and situation without relying on task-specific labeled data. The authors present standardized datasets and evaluations to facilitate uniform progress and comparison across the field.

Key Contributions

The paper makes several significant contributions:

1. Unified Datasets: The work compiles datasets that encompass diverse aspects beyond mere topical categorization, including emotion and situation detection, thereby broadening the scope of 0shot-tc research.
2. Evaluation Framework: Two evaluation methods are introduced:
   • Label-partially-unseen: Training on some labels and testing on the full set.
   • Label-fully-unseen: Classifying without seeing any task-specific training data, pushing the boundaries of text classification models.
3. Entailment Approach: The authors recast 0shot-tc as a textual entailment problem, simulating human-like decision-making in text interpretation. This approach employs hypothesis generation using label names and their definitions to enhance the model's understanding of label semantics.

Experimental Findings

The research presents experimental results comparing baseline methods (e.g., Word2Vec, ESA) and trained models (e.g., BERT fine-tuned on entailment datasets like MNLI, FEVER, and RTE). The findings suggest:

• Textual Entailment Superiority: Models trained on textual entailment tasks generally outperform traditional methods, especially in unseen label prediction scenarios.
• Diverse Performance Across Aspects: The entailment models exhibit varying levels of success across different classification tasks (topic, emotion, situation), indicating the need for task-specific tuning.
• Ensemble Approach: An ensemble of models trained on different entailment datasets yields the best performance, demonstrating the potential for further improvements in capturing label semantics.

Implications of the Research

Practically, this work paves the way for developing more adaptable and generalizable text classification systems, vital for applications like intent recognition and sentiment analysis where labeled data is scarce. Theoretically, the articulation of 0shot-tc within a textual entailment framework introduces new challenges and opportunities in understanding the semantic relationship between texts and labels.

Future Directions

While this paper provides a foundational standard for 0shot-tc research, future developments could explore:

• Enhancing hypothesis generation techniques to improve the naturalness and accuracy of label descriptions.
• Developing more sophisticated models to handle abstract or less frequent classes.
• Expanding the datasets to include more diverse and complex domains.

In conclusion, this paper contributes significantly to the field of natural language understanding by offering a coherent framework for benchmarking zero-shot text classification, thus inviting further research to advance machine understanding capabilities.