Dual Contrastive Learning: Text Classification via Label-Aware Data Augmentation (2201.08702v1)

Abstract: Contrastive learning has achieved remarkable success in representation learning via self-supervision in unsupervised settings. However, effectively adapting contrastive learning to supervised learning tasks remains as a challenge in practice. In this work, we introduce a dual contrastive learning (DualCL) framework that simultaneously learns the features of input samples and the parameters of classifiers in the same space. Specifically, DualCL regards the parameters of the classifiers as augmented samples associating to different labels and then exploits the contrastive learning between the input samples and the augmented samples. Empirical studies on five benchmark text classification datasets and their low-resource version demonstrate the improvement in classification accuracy and confirm the capability of learning discriminative representations of DualCL.

Dual Contrastive Learning: Text Classification via Label-Aware Data Augmentation

The paper presents a novel supervised contrastive learning framework, Dual Contrastive Learning (DualCL), designed to enhance text classification tasks by leveraging label-aware data augmentation. The research tackles the limitations inherent in the direct application of contrastive learning, which has demonstrated efficacy in unsupervised settings, to supervised learning contexts.

Core Contributions of the Paper

The primary innovation introduced by the paper is the DualCL framework, which addresses the disjunction between traditional supervised contrastive learning and the generation of directly usable classifiers in supervised tasks. At its core, DualCL simultaneously learns two distinct types of representations: input sample feature representations, and parameter representations that act as classifiers for those features. This alignment allows for a more direct translation of learned representations into actionable classifier parameters without requiring additional processing steps or external classifier algorithms.

The following key components mark significant contributions of the paper:

1. Dual Representations: DualCL incorporates a novel mechanism where each training sample is associated with both a feature representation and a classifier representation. This dual representation allows the model to not only capture feature alignments within the data but also formulate implicit linear classifiers in the latent space.
2. Label-Aware Data Augmentation: The paper introduces a method for generating label-aware input representations, effectively creating multiple perspectives of the input data enriched with label information. This process helps in augmenting the input data without truly adding new samples and facilitates the duality in learning both features and classifiers concurrently.
3. Dual Contrastive Loss: Two interdependent contrastive loss functions are developed. These losses govern the interactions between input feature representations and classifier representations, ensuring that they are aligned with correct labels while being distinct for different class labels.
4. Theoretical Justification: The paper provides a theoretical foundation illustrating that minimizing the dual contrastive loss is tantamount to maximizing the mutual information between the inputs and the labels. This insight offers a deeper understanding of how DualCL effectively optimizes representation learning in supervised contexts.

Empirical Validation

The efficacy of the DualCL framework is validated through empirical studies across five benchmark text classification datasets: SST-2, SUBJ, TREC, PC, and CR. The experimental results establish that DualCL consistently outperforms baseline models employing cross-entropy loss and existing contrastive learning adaptations, such as supervised contrastive LLMing techniques.

Notably, the results demonstrate superior performance in low-resource scenarios, a common challenge in text classification tasks where labeled data can be more scarce. This advantage is attributed to the framework's ability to exploit the label-aware augmentation, inherently generating richer and more informative representations from limited data points.

Implications and Future Directions

The implications of DualCL extend beyond improved text classification accuracy. By embedding classifier parameter learning within the representation learning phase, the paper hints at a paradigm where models become inherently more versatile and efficient in leveraging supervised learning signals. The framework's success raises interesting questions about its potential application in other machine learning domains where dual representations might yield similar benefits, such as image or graph classification.

Speculatively, the refined approach to contrastive learning embodied by DualCL could inspire modifications in the training of larger, domain-agnostic models like those used in transfer learning settings. Given the dual representation structure, future developments could focus on expanding this framework to accommodate more complex model architectures and varied data modalities. Extending the label-aware augmentation to handle multi-label or hierarchical labeling scenarios offers an exciting trajectory for further research.

In conclusion, the paper presents a solid advancement in the field of text classification through supervised contrastive learning. The framework's novel integration of dual representations and its superior performance on a range of datasets underscores its contribution to developing more efficient and effective text classification models.