Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

When LLMs are Unfit Use FastFit: Fast and Effective Text Classification with Many Classes (2404.12365v1)

Published 18 Apr 2024 in cs.CL, cs.AI, cs.IR, and cs.LG

Abstract: We present FastFit, a method, and a Python package design to provide fast and accurate few-shot classification, especially for scenarios with many semantically similar classes. FastFit utilizes a novel approach integrating batch contrastive learning and token-level similarity score. Compared to existing few-shot learning packages, such as SetFit, Transformers, or few-shot prompting of LLMs via API calls, FastFit significantly improves multiclass classification performance in speed and accuracy across FewMany, our newly curated English benchmark, and Multilingual datasets. FastFit demonstrates a 3-20x improvement in training speed, completing training in just a few seconds. The FastFit package is now available on GitHub and PyPi, presenting a user-friendly solution for NLP practitioners.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Stance classification of context-dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251–261, Valencia, Spain. Association for Computational Linguistics.
  2. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807.
  3. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.
  4. Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.
  5. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  6. A large-scale dataset for argument quality ranking: Construction and analysis.
  7. Zero-shot topical text classification with LLMs - an experimental study. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9647–9676, Singapore. Association for Computational Linguistics.
  8. Mistral 7b.
  9. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert.
  10. Supervised contrastive learning.
  11. An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv:1909.02027.
  12. Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
  13. Selective in-context data augmentation for intent detection using pointwise v-information. In Conference of the European Chapter of the Association for Computational Linguistics.
  14. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. ArXiv, abs/2205.05638.
  15. Lost in the middle: How language models use long contexts.
  16. Benchmarking natural language understanding services for building conversational agents. arXiv preprint arXiv:1903.05566.
  17. Roberta: A robustly optimized bert pretraining approach.
  18. Making llms worth every penny: Resource-limited text classification in banking. Proceedings of the Fourth ACM International Conference on AI in Finance.
  19. In-context learning for text classification with many labels. ArXiv, abs/2309.10954.
  20. Gpt-4 technical report.
  21. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.
  22. Multitask prompted training enables zero-shot task generalization. ArXiv, abs/2110.08207.
  23. Mpnet: Masked and permuted pre-training for language understanding.
  24. Ul2: Unifying language learning paradigms.
  25. Llama 2: Open foundation and fine-tuned chat models.
  26. Efficient few-shot learning without prompts.
  27. Finetuned language models are zero-shot learners.
  28. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
  29. Qaid: Question answering inspired few-shot intent detection.
  30. Bertscore: Evaluating text generation with bert.
  31. Revisiting few-sample {bert} fine-tuning. In International Conference on Learning Representations.

Summary

  • The paper presents FastFit, a few-shot text classification method that dramatically speeds up training (3–20x faster) while delivering superior accuracy.
  • It leverages batch contrastive training and token-level similarity metrics to effectively distinguish subtle differences among many classes.
  • FastFit offers easy integration with Hugging Face, is accessible via GitHub and PyPi, and demonstrates robust multilingual performance on diverse benchmarks.

Overview of FastFit: A Fast and Accurate Few-Shot Classification Method

Introduction

FastFit represents a notable step in addressing the challenge of few-shot classification, particularly when classes exhibit subtle semantic differences. By integrating techniques like batch contrastive learning and token-level similarity scoring, this method significantly outpaces its predecessors in both performance and efficiency.

Key Contributions

FastFit introduces several notable advancements in few-shot text classification:

  • Batch Contrastive Training: Utilizes intra-batch examples as negative samples, improving the separation of text classes efficiently.
  • Token-Level Similarity Metrics: Enhances the granularity of text comparison, leading to better discrimination between closely related classes.
  • Rapid Training Capabilities: Demonstrates a training speed 3-20x faster than comparative methods, enabling model readiness in seconds.

Technical Implementation

Implementation details highlight FastFit's accessibility and ease of integration:

  • Availability: The package is accessible via GitHub and PyPi, simplifying its adoption in practical applications.
  • Integration with Hugging Face: FastFit extends from the Hugging Face trainer, ensuring compatibility and customization according to user or task-specific requirements.

Evaluation on FewMany Benchmark

FastFit has been rigorously evaluated on the FewMany benchmark:

  • Consists of diverse datasets featuring at least 50 classes each, covering domains from intent detection to product classification.
  • Demonstrated superior performance in both English and multilingual settings, markedly outperforming other existing few-shot learning approaches.

Experimental Insights

Detailed experiments furnish extensive validation of FastFit's strengths:

  • Comparison with Baselines: FastFit shows heightened accuracy in multi-class few-shot scenarios compared to both standard classifier setups and other few-shot learning packages.
  • Efficiency in Training: Substantial improvements in training times were documented, providing substantial practical benefits in terms of efficiency.

Multilingual Capabilities

FastFit extends its applicability to multilingual datasets, an essential feature given the global use of AI solutions:

  • Evaluated using the Amazon Multilingual MASSIVE dataset, observing consistent performance across languages.

Versatility and Future Application

The flexibility of FastFit, demonstrated across different model sizes and in multilingual contexts, suggests its utility extends beyond its current applications. It portends well for future adaptations and optimizations that may further enhance its applicability and performance in an even broader array of few-shot and NLP tasks. Additionally, improvements in upstream processes like tokenization and pre-training could yield further gains.

Overall, FastFit presents a robust solution to the challenges of few-shot classification in NLP, combining speed, accuracy, and practicality. As AI continues to evolve, such tools will be pivotal in enabling efficient model training and deployment across various languages and domains.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown