Rethinking Feature Distribution for Loss Functions in Image Classification (1803.02988v1)

Abstract: We propose a large-margin Gaussian Mixture (L-GM) loss for deep neural networks in classification tasks. Different from the softmax cross-entropy loss, our proposal is established on the assumption that the deep features of the training set follow a Gaussian Mixture distribution. By involving a classification margin and a likelihood regularization, the L-GM loss facilitates both a high classification performance and an accurate modeling of the training feature distribution. As such, the L-GM loss is superior to the softmax loss and its major variants in the sense that besides classification, it can be readily used to distinguish abnormal inputs, such as the adversarial examples, based on their features' likelihood to the training feature distribution. Extensive experiments on various recognition benchmarks like MNIST, CIFAR, ImageNet and LFW, as well as on adversarial examples demonstrate the effectiveness of our proposal.

• The paper proposes the Large-Margin Gaussian Mixture (L-GM) loss, a novel function for image classification that models training features as a Gaussian Mixture distribution to improve performance.
• Empirical results show that the L-GM loss outperforms traditional loss functions like softmax cross-entropy across various benchmark datasets including ImageNet and LFW.
• The L-GM loss enhances robustness against adversarial examples by incorporating likelihood regularization, enabling better detection and classification of inputs deviating from learned distributions.

Rethinking Feature Distribution for Loss Functions in Image Classification

The paper "Rethinking Feature Distribution for Loss Functions in Image Classification" presents a novel approach to loss functions in deep neural networks used for image classification. The authors propose the Large-Margin Gaussian Mixture (L-GM) loss, a method that refines classification tasks by presuming that features extracted from the training set adhere to a Gaussian Mixture distribution. This proposition offers a significant advancement over the traditional softmax cross-entropy loss function by integrating a classification margin and likelihood regularization to enhance both classification performance and feature distribution modeling.

Deep learning has witnessed substantial improvements in classification tasks like object recognition, face verification, and even speech recognition. However, traditional loss functions such as softmax cross-entropy possess inherent limitations in probabilistically modeling training feature distributions effectively. The proposed L-GM loss function addresses these deficiencies by assuming a Gaussian Mixture Model (GMM) for the training features, thereby offering a framework where posterior probabilities are computed based on Bayes' theorem.

The L-GM loss incorporates a margin in the classification loss expression, simplifying its calculation compared to angular distance functions used in large-margin softmax loss approaches. This incorporation places no intricate demand on distance functions because of the non-negativity of Mahalanobis distance terms within a GMM framework, providing straightforward integration of classification margins.

Moreover, the likelihood regularization term within the L-GM loss encapsulates the center loss equivalence as a special case, hinting that feature distance measurements now share coherence in both classification loss and likelihood regularization dictated by the GMM assumption. The L-GM loss hence aligns feature distribution modeling and classification margin towards maximizing generalization capabilities and achieving robustness against adversarial examples.

Empirically, the L-GM loss is demonstrated to outperform established loss functions across various datasets, including MNIST, CIFAR, ImageNet, and LFW, proving its efficacy for both small-scale and large-scale classification problems. The quantitative advancements seen across benchmark datasets underscore the flexibility and performance improvements that the L-GM loss brings to image classification.

This novel loss function is particularly advantageous for adversarial example classification. The likelihood regularization term enables a reliable evaluation of an input's feature distribution, allowing for improved detection and classification of adversarial inputs. This additional robustness highlights a significant practical implication where models can be trained to recognize inputs deviating from learned distributions effectively, thus enhancing security and reliability.

In conclusion, the L-GM loss provides a compelling alternative in the field of loss functions for image classification tasks. By aligning feature distribution modeling with classification principles and improving adversarial distinction, it opens pathways for safer neural network deployment and extends theoretical understanding within classification frameworks. This work offers a promising direction in researcher endeavors in model optimization, enabling future advances in feature distribution analysis, loss function development, and robust system design across AI applications.