Simple-Sampling and Hard-Mixup with Prototypes to Rebalance Contrastive Learning for Text Classification (2405.11524v1)
Abstract: Text classification is a crucial and fundamental task in natural language processing. Compared with the previous learning paradigm of pre-training and fine-tuning by cross entropy loss, the recently proposed supervised contrastive learning approach has received tremendous attention due to its powerful feature learning capability and robustness. Although several studies have incorporated this technique for text classification, some limitations remain. First, many text datasets are imbalanced, and the learning mechanism of supervised contrastive learning is sensitive to data imbalance, which may harm the model performance. Moreover, these models leverage separate classification branch with cross entropy and supervised contrastive learning branch without explicit mutual guidance. To this end, we propose a novel model named SharpReCL for imbalanced text classification tasks. First, we obtain the prototype vector of each class in the balanced classification branch to act as a representation of each class. Then, by further explicitly leveraging the prototype vectors, we construct a proper and sufficient target sample set with the same size for each class to perform the supervised contrastive learning procedure. The empirical results show the effectiveness of our model, which even outperforms popular LLMs across several datasets.
- AI@Meta. Llama 3 model card. 2024.
- Deep over-sampling framework for classifying imbalanced data. In ECMLPKDD, pages 770–785, 2017.
- What is the effect of importance weighting in deep learning? In ICML, pages 872–881, 2019.
- Learning imbalanced datasets with label-distribution-aware margin loss. NeurIPS, 2019.
- A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109, 2023.
- Smote: synthetic minority over-sampling technique. JAIR, 16:321–357, 2002.
- A simple framework for contrastive learning of visual representations. In ICML, pages 1597–1607, 2020.
- Dual contrastive learning: Text classification via label-aware data augmentation. arXiv preprint arXiv:2201.08702, 2022.
- Class-balanced loss based on effective number of samples. In CVPR, pages 9268–9277, 2019.
- Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
- A holistic lexicon-based approach to opinion mining. In WSDM, pages 231–240, 2008.
- Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020.
- Understanding back-translation at scale. In EMNLP, pages 489–500, 2018.
- SimCSE: Simple contrastive learning of sentence embeddings. In EMNLP, 2021.
- Supervised contrastive learning for pre-trained language model fine-tuning. In ICLR, 2021.
- Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729–9738, 2020.
- Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2020.
- Lora: Low-rank adaptation of large language models. In ICLR, 2021.
- Detecting out-of-distribution data through in-distribution class prior. In ICML, 2023.
- Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020.
- Supervised contrastive learning. NeurIPS, pages 18661–18673, 2020.
- Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pages 1746–1751, 2014.
- Learning question classifiers. In COLING, 2002.
- A survey on text classification: From shallow to deep learning. arXiv preprint arXiv:2008.00364, 2020.
- Dice loss for data-imbalanced nlp tasks. In ACL, pages 465–476, 2020.
- Focal loss for dense object detection. In ICCV, pages 2980–2988, 2017.
- Multi-timescale long short-term memory neural network for modelling sentences and documents. In EMNLP, pages 2326–2335, 2015.
- Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
- Deep attention diffusion graph neural networks for text classification. In EMNLP, pages 8142–8152, 2021.
- Few-shot node classification on attributed networks with graph meta-learning. In SIGIR, pages 471–481, 2022.
- Local and global: Temporal question answering via information fusion. In IJCAI, 2023.
- Time-aware multiway adaptive fusion network for temporal knowledge graph question answering. In ICASSP, 2023.
- A simple but effective approach for unsupervised few-shot graph classification. In WWW, 2024.
- Improved graph contrastive learning for short text classification. In AAAI, 2024.
- Long-tail learning via logit adjustment. In ICLR, 2020.
- V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, pages 565–571, 2016.
- Training language models to follow instructions with human feedback. In NeurIPS, 2022.
- Contrastive learning with hard negative samples. In ICLR, 2020.
- Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
- Supervised prototypical contrastive learning for emotion recognition in conversation. In EMNLP, 2022.
- Training convolutional networks with noisy labels. In ICLR, 2015.
- Not all negatives are equal: Label-aware contrastive loss for fine-grained text classification. In EMNLP, pages 4381–4394, 2021.
- Pte: Predictive text embedding through large-scale heterogeneous text networks. In SIGKDD, pages 1165–1174, 2015.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
- Attention is all you need. In NeurIPS, 2017.
- Attention-based lstm for aspect-level sentiment classification. In EMNLP, pages 606–615, 2016.
- Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In EMNLP, pages 6382–6388, 2019.
- Data noising as smoothing in neural network language models. In ICLR, 2017.
- Self-taught convolutional neural networks for short text clustering. Neural Networks, 88:22–31, 2017.
- On multi-domain long-tailed recognition, generalization and beyond. In ECCV, 2022.
- Graph convolutional networks for text classification. In AAAI, pages 7370–7377, 2019.
- Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
- Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
- mixup: Beyond empirical risk minimization. In ICLR, 2018.
- Deep long-tailed learning: A survey. arXiv preprint arXiv:2110.04596, 2021.
- Empowering long-tail item recommendation through cross decoupling network (cdn). In SIGKDD, 2023.
- Imbalanced label distribution learning. In AAAI, 2023.
- Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In CVPR, pages 9719–9728, 2020.
- Balanced contrastive learning for long-tailed visual recognition. In CVPR, pages 6908–6917, 2022.
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.