Large Language Models For Text Classification: Case Study And Comprehensive Review (2501.08457v1)

Abstract: Unlocking the potential of LLMs in data classification represents a promising frontier in natural language processing. In this work, we evaluate the performance of different LLMs in comparison with state-of-the-art deep-learning and machine-learning models, in two different classification scenarios: i) the classification of employees' working locations based on job reviews posted online (multiclass classification), and 2) the classification of news articles as fake or not (binary classification). Our analysis encompasses a diverse range of LLMs differentiating in size, quantization, and architecture. We explore the impact of alternative prompting techniques and evaluate the models based on the weighted F1-score. Also, we examine the trade-off between performance (F1-score) and time (inference response time) for each LLM to provide a more nuanced understanding of each model's practical applicability. Our work reveals significant variations in model responses based on the prompting strategies. We find that LLMs, particularly Llama3 and GPT-4, can outperform traditional methods in complex classification tasks, such as multiclass classification, though at the cost of longer inference times. In contrast, simpler ML models offer better performance-to-time trade-offs in simpler binary classification tasks.

• The paper demonstrates that LLMs outperform traditional ML in complex text classification tasks, notably in nuanced scenarios like employee reviews.
• The study highlights the critical role of prompt engineering, where techniques such as Few-Shot combined with Chain of Thought significantly enhance outcomes.
• Findings reveal that larger models and effective quantization boost accuracy, while models like RoBERTa offer efficient trade-offs with reduced inference times.

Text Classification with LLMs: Performance, Techniques, and Trade-offs

The paper provides a comprehensive evaluation of LLMs in the context of text classification, contrasting their performance with traditional machine learning algorithms across two distinct classification tasks. This exploration is pivotal, as LLMs continue to play an increasingly significant role in natural language processing tasks.

Focusing on text classification, the paper examines two scenarios: a multiclass classification of employee reviews based on working location and a binary classification of news articles into fake or real. These tasks were designed to examine the ability of LLMs to handle both simple and complex categorization tasks. The LLMs under review span a variety of architectures and sizes, including encoder-only models like RoBERTa, and decoder-only models such as GPT-4 and the Mistral and Llama series.

Key Findings

1. Model Performance:
   • LLMs generally outperform traditional machine learning models in complex classification tasks, particularly in scenarios that require nuanced understanding and contextual interpretations, such as the employee review classification task.
   • RoBERTa achieved notable scores, closely following top-performing LLMs. Meanwhile, simpler ML models like Naive Bayes and Support Vector Machines (SVM) exhibited competitive results in less complex, binary tasks.
2. Impact of Prompting Techniques:
   • Prompt engineering significantly influences LLM performance. Certain techniques, such as Chain of Thought (CoT) and Few-Shot (FS) prompting, effectively enhance performance by providing additional contextual information. Particularly, FS+COT+RP+NA emerged as a leading strategy in facilitating better task performance.
   • Conversely, techniques like Role-playing combined with Naming Assistant occasionally detracted from performance, indicating a potential for prompts to distract or mislead models, highlighting the intricacy of prompt design.
3. Model Scaling and Quantization:
   • Larger models like Llama3 70B consistently deliver superior performance, underscoring the advantage of scale in LLM capabilities. Despite this, quantized models demonstrated efficacy comparable to non-quantized versions, affirming that quantization techniques can preserve model performance while reducing computational demands.
4. Practical Trade-offs:
   • The time required for LLM inference is substantial, especially for more extensive models. When considering the performance-to-time ratio, RoBERTa provides an optimal balance, achieving competitive F1-scores with significantly reduced inference times.
   • In high-speed scenarios, traditional ML models emerged as efficient alternatives, clearly outperforming complex LLMs in binary tasks with remarkably low latency.

Implications and Future Directions

The findings suggest crucial considerations for the practical application of LLMs in real-world text classification tasks. While LLMs excel in understanding complex linguistic nuances, their deployment must weigh computational cost, speed, and task complexity. The nuanced trade-offs between model scale, prompt effectiveness, and performance metrics emphasize the need for a context-driven selection of models and techniques.

Future research could extend this paper by exploring additional datasets across varying domains, thereby verifying the robustness of these findings. Moreover, further exploration into adaptive prompting strategies could yield more universally effective designs, potentially bridging the performance gap observed across different models and architectures.

In conclusion, this paper advances our understanding of LLM applicability in text classification, demonstrating that, with strategic prompt and model selection, LLMs stand poised to transform text classification paradigms, provided their deployment is contextually nuanced and resource-conscious.