Convex Basins in Single-Index Model Loss Landscapes: Applications to Robust Recovery under Strong Adversarial Corruption

Abstract: We study the problem of robustly learning Gaussian Single Index Models (SIMs) in the presence of heavy-tailed noise and a constant fraction of adversarially corrupted covariates and responses. Prior work on robust recovery has considered settings such as linear regression (Pensia et al., JASA 2024), strictly monotonic link functions (Awasthi et al., NeurIPS 2022), and phase retrieval (Buna and Rebeschini, AISTATS 2025). However, these techniques do not extend to generic asymmetric non-monotonic link functions such as \textsc{GeLU} and \textsc{Swish}, which arise naturally as scalar primitives in modern gated neural architectures. We close this gap by giving the first robust recovery algorithm with near-linear sample and time complexity for generic non-monotonic link functions, thereby establishing the first robust recovery guarantees for a broad family of nonlinear SIMs for which \textit{no guarantees were previously known}. Our central contribution is a new structural understanding of the Gaussian squared-loss landscape under adversarial contamination. Crucially, we prove that for a broad class of nonlinear non-monotonic SIMs, a dimension-independent, constant-radius convex basin exists around the ground truth and is efficiently reachable via robust spectral initialization even under adversarial contamination. Prior works fail to establish both guarantees simultaneously, thereby either breaking down under adversarial contamination or failing to handle generic non-monotonic link functions. Together, these structural insights yield a principled warm start for robust gradient descent that provably converges to a final estimation error of $O(σ\sqrtε)$ in $\tilde{O}(nd)$ time with $\tilde{O}(d)$ samples, where $ε$ is the contamination fraction.

• The paper introduces a robust estimator that recovers parameters in high-dimensional SIMs despite heavy-tailed noise and adversarial corruption.
• It leverages convex basins in the loss landscape for efficient initialization via robust spectral methods and Stein identities.
• The proposed algorithm achieves an error rate of O(σ√ε) in nearly-linear time, extending robust recovery to non-monotonic link functions.

Convex Basins in Single-Index Model Loss Landscapes: Applications to Robust Recovery under Strong Adversarial Corruption

Overview and Problem Setting

The paper "Convex Basins in Single-Index Model Loss Landscapes: Applications to Robust Recovery under Strong Adversarial Corruption" (2605.29497) addresses robust parameter estimation in high-dimensional Single-Index Models (SIMs) when covariates and responses are subject to heavy-tailed noise and a constant fraction of arbitrary adversarial contamination. The SIM framework is highly general, encompassing linear and logistic regression, phase retrieval, and generalized linear models (GLMs), by modeling $Y = f(X^\top \beta^\star) + \zeta$ with $f$ a known link, $X \sim \mathcal{N}(0, I)$, and $\zeta$ stochastic noise.

Standard approaches for robust high-dimensional estimation provide algorithmic and statistical guarantees solely in settings with monotonic links or highly symmetric functions, such as quadratic phase retrieval, and typically fail for generic, especially non-monotonic, nonlinear links (e.g., GeLU, Swish), which are now prevalent in deep neural architectures. Adversarial contamination further exacerbates parameter recoverability due to the induced breakdown of the loss landscape and loss of statistical identity in gradient flow.

The work closes a significant gap by characterizing the precise landscape geometry that enables robust, efficient recovery for a broad class of link functions, and introduces the first near-linear-time robust estimator for generic non-monotonic SIMs tolerating heavy-tailed noise and strong adversarial corruption.

Loss Landscape Geometry and Structural Assumptions

The analysis centers on the squared-error loss landscape

$$\mathcal{L}(\beta) = \tfrac{1}{2} \mathbb{E}\left[ (f(X^\top \beta) - Y)^2 \right]$$

and the corresponding Hessian $H(\beta)$. For $f$ affine, $\mathcal{L}(\beta)$ is globally convex, but for generic $f$—and notably for non-monotonic activations—nonconvexity is typical. The authors prove that robust recovery is enabled provided the loss exhibits a dimension-independent, constant-radius convex basin around the ground truth, and that this region is efficiently accessible even under adversarial contamination.

Two key structural link function conditions are established:

• Convex Basin Property: For a large set of $f$ (including monotonic, quadratic, GeLU, Swish), the Hessian inside a ball of radius $f$0 about the ground truth remains strongly convex, with curvature lower bounded by terms involving low-dimensional Gaussian integrals of $f$1 and its derivatives. This covers non-monotonic functions previously untreatable by robust methods.
• Expected Squared Convexity (ESC): A spectral identifiability condition, $f$2, ensures the existence of a moment matrix whose leading eigenvector is $f$3, so that robust initialization via higher-order spectral methods is possible.

Thus, the existence and accessibility of a convex basin are characterized via easy-to-check integral constraints on $f$4.

Algorithmic Approach: Robust Spectral Initialization and Descent

With the structural landscape properties, the robust recovery algorithm comprises two main stages:

1. Robust Spectral Initialization: Generalizing the phase retrieval principle, the method applies robust PCA to higher-order moment matrices (constructed via Stein identities) to compute an initialization $f$5 near $f$6. For functions with information exponent (IE) at most 2, including GeLU and Swish, these moments carry sufficient signal despite adversarial corruption, and robust PCA can extract a direction within the convex basin. The algorithm leverages the robust hypercontractive PCA of [jambulapati2024black], yielding both computational and contamination tolerance efficiency.
2. Robust Gradient Descent: From the robust initialization, projected robust gradient steps using nearly-linear time robust mean estimation further refine the estimate. The outer iterations remain within the convex basin, guaranteeing fast, stable convergence under both contamination and heavy-tailed noise.

The end-to-end complexity is $f$7 time with $f$8 samples, which is optimal up to logarithmic factors. Remarkably, the error rate achieved is $f$9, matching upper bounds for robust quadratic phase retrieval under similar contamination, despite the expanded class of link functions.

Theoretical Guarantees and Implications

Main quantitative result: For $X \sim \mathcal{N}(0, I)$0, $X \sim \mathcal{N}(0, I)$1, and $X \sim \mathcal{N}(0, I)$2 satisfying the geometric and spectral conditions above, the estimator $X \sim \mathcal{N}(0, I)$3 satisfies, with high probability,

$X \sim \mathcal{N}(0, I)$4

given $X \sim \mathcal{N}(0, I)$5 samples and $X \sim \mathcal{N}(0, I)$6-fraction strong adversarial contamination of covariates and responses.

Key claims include:

• This is the first guarantee for many non-monotonic, non-affine link functions beyond phase retrieval, particularly those such as GeLU and Swish now ubiquitously used in neural gating (including Transformers’ GLU variants).
• Both robust reachability (initialization) and local geometric convergence (descent) are simultaneous, a property not previously established in this generality.
• For all monotonic links and many non-monotonic links with IE $X \sim \mathcal{N}(0, I)$7 (detectable by second-order moments), robust sample and computational efficiency is possible.

Practically, this advances robust high-dimensional estimation for structured models directly relevant to modern neural networks, where increasingly complex, non-monotonic nonlinearities can now be efficiently and reliably trained in adversarial or heavy-tailed environments.

Connections and Outlook

The theoretical developments clarify the computational-statistical tradeoff frontier for robust estimation in nonlinear models. For links with higher information exponent ($X \sim \mathcal{N}(0, I)$8), the robust moment-tensor approach outlined herein would necessitate robust eigen-tensors and higher-order concentration, whose scalability remains an open theoretical and algorithmic problem.

The geometric convex basin principle could motivate new architectures and learning frameworks in deep learning by suggesting activation design with both expressiveness and robust learnability. The extension to non-Gaussian covariate distributions, general $X \sim \mathcal{N}(0, I)$9-estimators, or agnostic/differentially private settings, as well as robust recovery in Multi-Index Models (MIMs), are promising directions indicated by the analysis.

Conclusion

The paper establishes a comprehensive geometric and algorithmic framework for robust recovery in single-index models with broad link functions subject to strong adversarial contamination and heavy-tailed noise (2605.29497). By identifying explicit, efficiently verifiable structural properties of the loss landscape and spectral moments, it delineates the precise conditions permitting optimal and efficient robust recovery for activation functions spanning monotonic, symmetric, and asymmetric non-monotonic cases. The results substantially expand the universe of robustly learnable models and inform future research at the intersection of robust statistics, non-convex optimization, and modern neural architectures.