Robust Optimization for Deep Regression

Abstract: Convolutional Neural Networks (ConvNets) have successfully contributed to improve the accuracy of regression-based methods for computer vision tasks such as human pose estimation, landmark localization, and object detection. The network optimization has been usually performed with L2 loss and without considering the impact of outliers on the training process, where an outlier in this context is defined by a sample estimation that lies at an abnormal distance from the other training sample estimations in the objective space. In this work, we propose a regression model with ConvNets that achieves robustness to such outliers by minimizing Tukey's biweight function, an M-estimator robust to outliers, as the loss function for the ConvNet. In addition to the robust loss, we introduce a coarse-to-fine model, which processes input images of progressively higher resolutions for improving the accuracy of the regressed values. In our experiments, we demonstrate faster convergence and better generalization of our robust loss function for the tasks of human pose estimation and age estimation from face images. We also show that the combination of the robust loss function with the coarse-to-fine model produces comparable or better results than current state-of-the-art approaches in four publicly available human pose estimation datasets.

• The paper presents a robust deep regression method using Tukey's biweight loss function to effectively reduce the influence of outliers in training data without hard thresholding.
• The authors introduce a coarse-to-fine cascade of ConvNets to progressively refine predictions using higher-resolution inputs, improving accuracy for complex regression tasks.
• Empirical evaluation on several datasets shows significant performance improvements, including reduced mean pixel error and up to 20-fold faster convergence compared to traditional L2 loss.

Overview of Robust Optimization for Deep Regression

The paper authored by Vasileios Belagiannis et al. presents a novel approach to enhancing the robustness of deep regression for computer vision tasks by incorporating Tukey's biweight function, a robust M-estimator, into Convolutional Neural Networks (ConvNets). The research addresses a significant issue prevalent in regression tasks—sensitivity to outliers—by proposing a loss function that downweights the influence of outliers on the training process. This is achieved without establishing a hard threshold between inliers and outliers, circumventing the pitfalls of $L2$ minimization that is typically sensitive to outliers.

Methodological Innovations

The authors propose a two-fold improvement to enhance regression tasks within ConvNets:

1. Tukey's Biweight Loss Function: Unlike the commonly used $L2$ loss, Tukey's biweight function effectively suppresses the influence of outliers on the training, offering faster convergence and improved generalization. Its incorporation is meticulously demonstrated through its ability to downweight training samples with unusually large errors. This feature is particularly pivotal for regression tasks like human pose estimation and age estimation from facial images. The robust loss is parameter-free due to the incorporation of the median absolute deviation, which scales the residuals' variability.
2. Coarse-to-Fine Model: The authors introduce a cascade of ConvNets, where initial predictions are refined through progressively higher-resolution input images. This strategy ensures that image regions are processed with increasing precision, improving the accuracy of regressed values. The cascade facilitates feature capture at varied resolutions, crucial for nuanced tasks like human pose detection.

Empirical Evaluation

The robustness and efficacy of these innovations are empirically validated on multiple datasets, namely PARSE, LSP, Football, and Volleyball datasets for human pose estimation, as well as an age estimation task from facial images. The findings reveal significant improvements, such as:

• A marked reduction in mean pixel error (MPE) compared to traditional $L2$ loss, illustrating better generalization across data with outliers.
• Faster convergence rates, notably achieving up to 20-fold acceleration on the PARSE dataset.
• Competitiveness with state-of-the-art models for human pose estimation, often surpassing them. The model leverages robust learning, achieving better results or comparable performance with fewer learned parameters and training epochs compared to other deep-learning architectures.

Implications and Future Directions

The implications of employing a robust loss function extend deep into practical and theoretical realms. Practically, the significant enhancement in regression tasks' robustness can lead to more reliable deployment of computer vision systems in real-world scenarios that are characterized by noisy data. Theoretically, this work broadens the scope of robust statistics within neural network training, encouraging further exploration of alternative robust estimators in deep learning.

Future work may explore:

• Extensions to other regression domains such as depth estimation or 3D reconstruction.
• Integrating these robust techniques with diverse network architectures to potentially uncover other benefits or synergies.
• Examining this loss function's adaptability in multitask learning scenarios, where tasks of varying complexity might benefit from the robust handling of outliers.

The paper proposes a compelling advance in deep regression optimization through the robust handling of outliers, evidencing significant strides in both performance and computational efficiency.