Contrastive Out-of-Distribution Detection for Pretrained Transformers (2104.08812v2)

Abstract: Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time. Therefore, in practice, a reliable model should identify such instances, and then either reject them during inference or pass them over to models that handle another distribution. In this paper, we develop an unsupervised OOD detection method, in which only the in-distribution (ID) data are used in training. We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones. These OOD instances can then be accurately detected using the Mahalanobis distance in the model's penultimate layer. We experiment with comprehensive settings and achieve near-perfect OOD detection performance, outperforming baselines drastically. We further investigate the rationales behind the improvement, finding that more compact representations through margin-based contrastive learning bring the improvement. We release our code to the community for future research.

• The paper introduces an unsupervised framework using contrastive learning to fine-tune pretrained transformers for enhanced out-of-distribution (OOD) detection.
• Combining contrastive learning with Mahalanobis distance in the penultimate layer achieves near-perfect OOD detection, significantly outperforming baseline methods.
• The proposed method improves robustness to OOD instances without degrading performance on in-distribution tasks, highlighting a promising approach for reliable NLP systems.

Contrastive Out-of-Distribution Detection for Pretrained Transformers

The paper "Contrastive Out-of-Distribution Detection for Pretrained Transformers" presents a focused paper on enhancing the robustness of pretrained transformers when encountering out-of-distribution (OOD) instances, which are crucial for deploying reliable NLP systems. Despite the remarkable efficacy of pretrained transformers on in-distribution (ID) data, their vulnerability to OOD instances, which cause semantic shifts, remains a significant challenge in practical applications. The paper introduces a novel unsupervised OOD detection framework that leverages contrastive learning to fine-tune transformers, aiming to improve their ability to differentiate OOD instances based on compact representation learning.

Methodology

The proposed method is centered around unsupervised OOD detection, where only ID data are used for training. A critical innovation in this work is the application of contrastive learning to transformer models. The approach involves fine-tuning these models with a contrastive loss to enhance the representation space's compactness. This increased compactness aids in distinguishing between ID and OOD instances more effectively.

To operationalize OOD detection, the method computes the Mahalanobis distance in the model's penultimate layer, a choice supported by previous research showing its efficacy in capturing distributional characteristics. The paper argues that this combination of contrastive learning and Mahalanobis distance measurement, particularly when paired with a margin-based contrastive loss, leads to superior OOD detection performance.

Experimental Evaluation

The authors conduct expansive experiments across various settings, utilizing multiple NLP datasets to benchmark against standard baselines. These experiments demonstrate that the proposed framework achieves near-perfect OOD detection, markedly outperforming existing methods like Maximum Softmax Probability and baseline energy-score-based approaches. Importantly, the results underscore that the margin-based contrastive loss yields the most significant improvements in OOD detection metrics, such as AUROC and FAR95, when compared to other scoring mechanisms.

Findings and Implications

The paper provides evidence that more compact representations through contrastive learning enhance OOD detection capabilities, suggesting a promising direction for future exploration. It emphasizes the practical utility of the proposed framework, which does not degrade ID task performance, thereby maintaining classification effectiveness while bolstering robustness against OOD instances.

Future Research and Implications

The paper suggests an intriguing avenue for future work in novel class detection, which presents a more challenging scenario where OOD instances derive from the same corpus and must be distinguished amidst a shared task environment. This challenge highlights the need for continued research to further refine OOD detection methodologies, especially considering the increasingly complex and diverse data environments faced by NLP systems.

In conclusion, this research contributes a substantial step toward more resilient NLP models capable of effectively managing the unpredictability and variability inherent in real-world data. As the field progresses, the insights and methodologies presented in this paper are likely to inform and inspire further advancements in unsupervised OOD detection strategies.