Outlier-Robust Training of Machine Learning Models

Abstract: Robust training of machine learning models in the presence of outliers has garnered attention across various domains. The use of robust losses is a popular approach and is known to mitigate the impact of outliers. We bring to light two literatures that have diverged in their ways of designing robust losses: one using M-estimation, which is popular in robotics and computer vision, and another using a risk-minimization framework, which is popular in deep learning. We first show that a simple modification of the Black-Rangarajan duality provides a unifying view. The modified duality brings out a definition of a robust loss kernel $\sigma$ that is satisfied by robust losses in both the literatures. Secondly, using the modified duality, we propose an Adaptive Alternation Algorithm (AAA) for training machine learning models with outliers. The algorithm iteratively trains the model by using a weighted version of the non-robust loss, while updating the weights at each iteration. The algorithm is augmented with a novel parameter update rule by interpreting the weights as inlier probabilities, and obviates the need for complex parameter tuning. Thirdly, we investigate convergence of the adaptive alternation algorithm to outlier-free optima. Considering arbitrary outliers (i.e., with no distributional assumption on the outliers), we show that the use of robust loss kernels {\sigma} increases the region of convergence. We experimentally show the efficacy of our algorithm on regression, classification, and neural scene reconstruction problems. We release our implementation code: https://github.com/MIT-SPARK/ORT.

• The paper introduces a unified robust loss kernel that adapts non-linear least squares and cross-entropy losses to effectively neutralize outlier effects.
• It proposes an Adaptive Alternation Algorithm that alternates weight updates to automatically tune model and outlier parameters, enhancing convergence.
• Experimental and theoretical analyses demonstrate expanded convergence regions and superior performance over standard techniques in regression and classification tasks.

Outlier-Robust Training of Machine Learning Models

In the paper "Outlier-Robust Training of Machine Learning Models," the authors present a novel framework for improving the robustness of machine learning models when training data is contaminated by outliers. The study addresses two distinct literatures—M-estimation frameworks utilized in robotics and computer vision, and noise-tolerant approaches formulated in deep learning—bridging them through a unified concept of robust loss design.

Core Contributions

1. Unified Robust Loss Kernel: The paper introduces a modified Black-Rangarajan duality that overlays a coherent scaffold across different robust loss designs. By modifying the duality originally intended for least-squares problems, the authors craft a robust loss kernel concept applicable to both non-linear least squares and cross-entropy losses. This kernel, denoted as $\sigma$, conforms to a set of conditions ensuring it acts linearly for small losses and dampens significantly at larger losses, thus neutralizing outlier effects.
2. Adaptive Alternation Algorithm: Using the developed robust loss kernel, the authors propose the Adaptive Alternation Algorithm (Adv3), which leverages an iterative framework to adjust model weights and outlier weights alternatively. The algorithm features an adaptive scheme for modifying kernel parameters, enhancing convergence without manual parameter tuning, which is a significant advancement over prior techniques like Graduated Non-Convexity (GNC).
3. Efficacy and Convergence Analysis: The paper offers a theoretical analysis showcasing the enlargement of convergence regions due to the robust loss kernel. Extensive experiments on regression and classification tasks, as well as novel view synthesis using neural radiance fields, indicate superior performance of the proposed method over traditional SGD methods and other heuristics employed to handle outliers.

Implications

The implications of this research are both profound and broad, affecting the training processes of contemporary machine learning models:

• Practical Advancements: The integration of robust loss kernels capable of cross-application into various domains simplifies the development of machine learning models that are resilient to training disturbances. Techniques derived from this research can be seamlessly applied across different data types and application scopes, from computer vision to complex robotics systems.
• Theoretical Insights: By mathematically elucidating the connections between risk minimization and robust estimation frameworks, the paper establishes a deeper theoretical foothold for understanding and developing robust training methods. This insight assists in refining the underlying assumptions typically considered during model development.
• Cross-Pollination of Techniques: The unified approach fosters cross-pollination between traditionally dichotomous research disciplines, potentially inspiring novel combinations of heuristic and analytical techniques to enhance robustness in machine learning tasks.

Future Directions

The study opens avenues for several future explorations:

• Broader Application Scenarios: Extending the applicability of the proposed robust loss kernels to non-standard machine learning challenges, such as those involving time series data or multi-modal data, could significantly enrich the robustness landscape.
• Refinement of Adaptive Methods: Further development of adaptive algorithms that incorporate uncertainty quantification and conformal predictions could lead to breakthroughs in achieving near-human-level perceptual robustness in noisy environments.
• Automated Hyperparameter Evolution: Fully automated systems that evolve and learn hyperparameters through meta-learning approaches grounded in the presented theoretical framework could revolutionize model optimization.

In conclusion, the paper distinctly contributes to the ongoing quest for robust machine learning methodologies by redefining the scope and application of robust loss designs, proposing an effective, computationally plausible adaptive algorithm, and setting a foundation for future research in outlier-robust learning. The comprehensive theory combined with practical efficacy has the potential to significantly influence both academic and industry-driven pursuits towards more reliable models.