Retrieval Augmented Zero-Shot Text Classification (2406.15241v2)

Abstract: Zero-shot text learning enables text classifiers to handle unseen classes efficiently, alleviating the need for task-specific training data. A simple approach often relies on comparing embeddings of query (text) to those of potential classes. However, the embeddings of a simple query sometimes lack rich contextual information, which hinders the classification performance. Traditionally, this has been addressed by improving the embedding model with expensive training. We introduce QZero, a novel training-free knowledge augmentation approach that reformulates queries by retrieving supporting categories from Wikipedia to improve zero-shot text classification performance. Our experiments across six diverse datasets demonstrate that QZero enhances performance for state-of-the-art static and contextual embedding models without the need for retraining. Notably, in News and medical topic classification tasks, QZero improves the performance of even the largest OpenAI embedding model by at least 5% and 3%, respectively. Acting as a knowledge amplifier, QZero enables small word embedding models to achieve performance levels comparable to those of larger contextual models, offering the potential for significant computational savings. Additionally, QZero offers meaningful insights that illuminate query context and verify topic relevance, aiding in understanding model predictions. Overall, QZero improves embedding-based zero-shot classifiers while maintaining their simplicity. This makes it particularly valuable for resource-constrained environments and domains with constantly evolving information.

• The paper introduces QZero, a training-free method that augments query embeddings using Wikipedia-derived context for zero-shot classification.
• It leverages a two-step pipeline to retrieve relevant categories and reformulate queries, enhancing both word and contextual embedding models via cosine similarity.
• Validation on six datasets shows accuracy gains up to 13%, offering improved performance and computational efficiency in resource-constrained settings.

Retrieval Augmented Zero-Shot Text Classification

The paper "Retrieval Augmented Zero-Shot Text Classification" introduces QZero, a novel approach for enhancing zero-shot text classification by leveraging retrieval-augmented learning. The primary aim is to address the challenge posed by the inherent lack of rich contextual information in query embeddings which often hinders zero-shot classification performance.

Methodology

QZero is designed as a training-free mechanism to bolster the quality of query embeddings without necessitating model retraining. The approach operates through a retrieval system that augments the query with relevant contextual information from a comprehensive knowledge corpus, specifically Wikipedia. QZero employs a two-step pipeline for the reformulation of queries before classification:

1. Retrieval of Categories: For any given input query, the retrieval system identifies and fetches relevant Wikipedia articles. The categories associated with these articles are then extracted.
2. Query Reformulation: The obtained categories are used to reformulate the initial query. For static word embedding models, keywords from these categories are extracted and weighed according to their frequency. For contextual models, the categories are concatenated to form the reformulated query.

This enhanced query is then embedded and compared to class label embeddings using cosine similarity for zero-shot classification tasks.

Results

The efficacy of QZero was tested on six diverse text classification datasets, including AG News, DBPedia, Yahoo Answers, Yummly, TagMyNews, and Ohsumed. Using QZero yielded notable improvements in classification accuracy across all datasets and models. Noteworthy results include:

• AG News: All models saw accuracy gains of at least 4.17%, except for the TE-3-large model, which experienced a minor drop of 1.57%. Word2Vec coupled with QZero improved performance significantly, even outperforming the contextual TE-3-small model by 3.4%.
• Ohsumed: TE-3-large and Word2Vec models achieved an accuracy boost of at least 5.00%, underscoring QZero's potential in the medical domain despite the general Wikipedia corpus being the only knowledge source.
• TagMyNews: Word2Vec achieved a significant improvement of 13.00% in accuracy, and TE-3-large saw a 6.61% increase.

QZero successfully enhances smaller embedding models to achieve performance levels comparable to their larger counterparts, translating to substantial computational savings. This dual utility of enriching the query context while avoiding costly retraining is particularly advantageous for resource-constrained settings.

Analysis and Implications

The use of both dense (Contriever) and sparse (BM25) retrievers highlights QZero's adaptability and robustness across retrieval paradigms. The dense retriever, particularly effective in domains aligned with its training data, excelled in tasks related to news topics and Wikipedia, while the sparse retriever proved more effective for extensive medical text in the Ohsumed dataset.

An additional analysis uncovers QZero's capacity to provide meaningful insights into the context of queries, showcasing the categories and keywords relevant to the query. These insights aid in verifying the pertinence to specific topics and understanding model predictions more profoundly.

Future Directions

Despite the documented successes, limitations remain. The potential noise introduced by uninformative categories in large contextual models like TE-3-large presents an area for refinement. Furthermore, exploring QZero's application beyond embedding models, such as its utility in generative or natural language inference models, stands as an intriguing avenue for further exploration.

Conclusion

The QZero approach represents a substantial step forward in zero-shot text classification, enhancing performance without the computational burden associated with model retraining. By leveraging retrieval-augmented learning and the vast knowledge within Wikipedia, QZero significantly improves embedding-based classification models' accuracy and utility. This methodology holds promise for more efficient and interpretable zero-shot classification, especially in dynamically evolving or resource-limited environments.