Stochastic gradient descent for streaming linear and rectified linear systems with adversarial corruptions

Published 2 Mar 2024 in cs.LG, cs.NA, math.NA, and stat.ML | (2403.01204v2)

Abstract: We propose SGD-exp, a stochastic gradient descent approach for linear and ReLU regressions under Massart noise (adversarial semi-random corruption model) for the fully streaming setting. We show novel nearly linear convergence guarantees of SGD-exp to the true parameter with up to $50\%$ Massart corruption rate, and with any corruption rate in the case of symmetric oblivious corruptions. This is the first convergence guarantee result for robust ReLU regression in the streaming setting, and it shows the improved convergence rate over previous robust methods for $L_1$ linear regression due to a choice of an exponentially decaying step size, known for its efficiency in practice. Our analysis is based on the drift analysis of a discrete stochastic process, which could also be interesting on its own.

Abstract PDF HTML Upgrade to Chat

Authors (3)

References (42)

Citations (1)

View on Semantic Scholar

Summary

The paper presents SGD-exp, a novel algorithm that uses an exponentially decaying step size to achieve nearly linear convergence in both linear and ReLU regressions.
It establishes theoretical guarantees by tolerating up to 50% corruption under the Massart model and any corruption under the symmetric oblivious model.
Empirical insights confirm the method’s robustness and scalability on high-dimensional streaming data, outperforming traditional robust regression approaches.

Stochastic Gradient Descent (SGD) for Streaming Data under Semi-Random Corruption: A Study on Linear and ReLU Regressions

The relentless growth of online data has necessitated the development of advanced machine learning algorithms capable of handling vast, high-dimensional, and streaming data sets. In particular, robust regression methods are critical for making accurate predictions even when a significant fraction of the data is corrupted. This blog post explores a novel approach titled Stochastic Gradient Descent with Exponential Decay (SGD-exp), designed for linear and ReLU regressions under the challenging conditions of semi-random adversarial corruption, specifically within the fully streaming setting.

Background

Robust regression techniques have gained prominence for their ability to fit models to data contaminated by outliers, which could occur due to various issues like measurement errors or adversarial attacks on data integrity. Typically, robust methods aim to reduce sensitivity to outliers, thus ensuring reliable predictions despite data corruption. Traditional approaches and earlier innovations have laid the groundwork for robust learning; however, they often struggle to scale effectively with ever-increasing data complexity and volume. Stochastic algorithms like Stochastic Gradient Descent (SGD) offer a lifeline due to their scalability and efficiency in high-dimensional settings. Yet, the integration of robustness in the context of streaming data—where data is processed on-the-fly without storing historical information—poses additional hurdles.

SGD-exp: The Proposed Method

SGD-exp tackles the challenge head-on by offering a robust regression framework suitable for linear and Rectified Linear Unit (ReLU) regressions under adversarial semi-random corruption models, namely the Massart noise model and symmetric oblivious response corruption. Its key innovation lies in incorporating an exponentially decaying step size, a strategy rarely explored in this context but known for its efficiency in practical settings.

Theoretical Guarantees

SGD-exp distinguishes itself by providing the first nearly linear convergence guarantees for both linear and ReLU regression under adversarial corruption conditions. It endures up to a 50% corruption rate under the Massart model, aligning with what is theoretically the highest tolerable corruption level for this model. Additionally, it tolerates any corruption rate under the symmetric oblivious model, a notable advancement over previous approaches.

Methodology

The essence of SGD-exp's methodology is adapting the step size according to an exponential decay schedule, which contrasts with the constant or polynomial decay used in traditional SGD applications. This adaptation significantly enhances the algorithm's resilience to outliers and corrupted data, as demonstrated through rigorous mathematical analysis and empirical validation.

Empirical Insights

Experimental results underscore SGD-exp's superior performance compared to existing robust methods, particularly in scenarios of high-dimensional data with adversarial corruption. Simulations on synthetic data sets reveal its near-linear convergence rates and robustness against varying degrees of corruption. Its efficacy is further showcased in real-world applications involving complex data sets, where SGD-exp consistently achieves low error rates, underscoring its practical utility and scalability.

Future Directions

While SGD-exp represents a significant leap forward in robust regression for streaming data, it also opens avenues for further exploration. Extensions to other non-linear models, refinement of convergence analysis under diverse noise distributions, and incorporation of constraints are promising areas for future research.

Conclusion

In sum, SGD-exp emerges as a powerful tool in the field of machine learning for streaming data, bridging the gap between robustness and scalability. Its theoretical underpinnings and compelling empirical performance mark a considerable step forward in designing algorithms that can withstand the imperfections inherent in real-world data. As data-driven decision-making continues to permeate diverse sectors, the importance of such robust, scalable algorithms becomes increasingly pronounced, making SGD-exp a timely and significant contribution to the field.

Markdown Report Issue