Adversarial Contamination Meets Hard Thresholding: An Iterative Algorithm with Signal Adaptivity and Minimax Optimality

Abstract: Pervasive data contamination -- stemming from measurement errors, outliers, or adversarial corruption -- has motivated the development of robust statistical methods. In this context, we propose a two-stage Adversarial Contamination-resistant Iterative Hard Thresholding (AC-IHT) algorithm for high-dimensional regression with contamination. Our nonconvex algorithm achieves minimax near-optimal (up to logarithmic terms) estimation by iteratively updating the coefficient vector and the contamination vector with different thresholding scales. We further demonstrate that our AC-IHT estimator is signal-adaptive: under proper signal conditions, it adaptively attains a sharper estimation rate and more accurate support recovery. Moreover, it enjoys the strong oracle property, laying a theoretical foundation for asymptotic inference. Numerical experiments confirm its superior finite-sample performance. Finally, we discuss theoretical extensions of the proposed procedure to generalized linear models and to heavy-tailed noise settings.

• The paper introduces a two-stage AC-IHT algorithm that iteratively applies hard thresholding to robustly estimate signals under adversarial contamination.
• It provides signal-adaptive rates for estimation and exact support recovery, matching minimax optimality in high-dimensional settings.
• Empirical and theoretical results confirm its efficiency, extending to GLMs and heavy-tailed noise while achieving oracle properties for inference.

Signal-Adaptive Iterative Hard Thresholding under Adversarial Contamination

Problem Formulation and Motivation

The increasing prevalence of adversarial contamination—including outliers, measurement errors, and malicious corruptions—in high-dimensional datasets necessitates robust statistical procedures with provable guarantees. The authors consider the high-dimensional linear model

$Y = X\beta^* + \sqrt{n}\theta^* + \xi,$

where $Y\in \mathbb{R}^n$ is the response vector, $X\in\mathbb{R}^{n\times p}$ the design matrix, $\beta^*$ an $s$-sparse coefficient vector, $\theta^*\in\mathbb{R}^n$ an ${\tilde o}$-sparse adversarial contamination vector, and $\xi$ sub-Gaussian noise. The multiplicative scaling of $\theta^*$ by $\sqrt{n}$ ensures homogeneity in the magnitude of the columns of $Y\in \mathbb{R}^n$0. The contamination model is sufficiently general to subsume multiple realistic scenarios, such as corrupted regression, robust principal component analysis, and settings with heterogenous treatment effects.

Traditional robust regression techniques based on convex relaxation or joint $Y\in \mathbb{R}^n$1 penalization lack sufficient adaptivity to signal strength and provide suboptimal statistical efficiency, especially in the presence of structured adversarial contamination. Additionally, existing minimax optimality results, while comprehensive in uncontaminated models, do not explicitly characterize how signal strengths of $Y\in \mathbb{R}^n$2 and $Y\in \mathbb{R}^n$3 impact estimation and support recovery under contamination.

Two-Stage AC-IHT Algorithm

The core methodological contribution is a two-stage Adversarial Contamination-resistant Iterative Hard Thresholding (AC-IHT) algorithm. This approach leverages the computational advantages and signal adaptivity properties of iterative hard thresholding, extending these benefits to the contaminated high-dimensional regime.

Stage 1 establishes an initial estimator by iteratively performing gradient updates of the loss

$Y\in \mathbb{R}^n$4

coupled with hard-thresholding separately on both $Y\in \mathbb{R}^n$5 and $Y\in \mathbb{R}^n$6. Thresholds are dynamically decayed to their targeted rates; this separation is crucial for the estimator to achieve delicate control over sparsity and bias propagation between the coefficient and contamination vectors.

Stage 2 refines the initial estimator by re-running hard-thresholded gradient iterations, but with fixed thresholds set at theoretically justified statistical levels. The decoupled handling of $Y\in \mathbb{R}^n$7 and $Y\in \mathbb{R}^n$8 allows the final estimator, $Y\in \mathbb{R}^n$9, to achieve signal-adaptive rates as well as exact support recovery under suitable beta-min and theta-min conditions.

Figure 1: Estimation accuracy and support recovery performance with increasing sample size.

Figure 2: The $X\in\mathbb{R}^{n\times p}$0 estimation error of the two-stage AC-IHT algorithm as a function of sparsity $X\in\mathbb{R}^{n\times p}$1 and contamination $X\in\mathbb{R}^{n\times p}$2 in 300 replications.

Theoretical Guarantees

The authors' main results can be summarized along several axes:

Estimation and Support Recovery Rates

Let $X\in\mathbb{R}^{n\times p}$3. Without stringent signal strength assumptions, the AC-IHT estimator achieves

$X\in\mathbb{R}^{n\times p}$4

matching known uncontaminated minimax rates when $X\in\mathbb{R}^{n\times p}$5 is small and transitioning smoothly to the contamination-dominated regime as $X\in\mathbb{R}^{n\times p}$6 increases. If a beta-min condition $X\in\mathbb{R}^{n\times p}$7 holds, the estimator attains an improved

$X\in\mathbb{R}^{n\times p}$8

reflecting signal-adaptive sharpness. Further, under both beta-min and theta-min (for contamination) conditions, the estimator converges to the oracle rate $X\in\mathbb{R}^{n\times p}$9 and achieves exact support recovery:

$\beta^*$0

with high probability.

Figure 3: Comparison of asymptotic normality for AC-IHT, IHT-$\beta^*$1, and AC-SCAD.

Asymptotic Inference

In the strong-signal regime, $\beta^*$2 satisfies the strong oracle property. That is, for any direction $\beta^*$3,

$\beta^*$4

enabling asymptotically valid post-selection inference. This property differentiates AC-IHT from previous contamination-robust estimators, which do not attain asymptotic normality.

Minimax Optimality

Lower bounds for both estimation ($\beta^*$5 loss for $\beta^*$6) and approximate support recovery match the algorithmic upper bounds (up to logarithmic factors). In particular, the joint estimation error cannot be improved over

$\beta^*$7

This validates the sharpness of the AC-IHT rates and underscores its near-minimax optimality under a broad adversarial contamination model.

Empirical Results

Comprehensive simulation studies corroborate the theoretical findings. In a range of contamination regimes and noise distributions (Gaussian, Rademacher, Uniform, and heavy-tailed), AC-IHT dominates or matches classical and contemporary benchmarks, both in estimation error and support recovery metrics. The numerical results show robust convergence to the oracle solution as $\beta^*$8 increases and favorable finite-sample properties even under substantial contamination.

Figure 4: Convergence dynamics of AC-IHT over iterations for varying $\beta^*$9 and $s$0. Each setting aggregates 100 simulations.

Figure 5: Outlier support recovery with increasing contamination level $s$1; larger $s$2 relaxes the signal threshold required for accurate identification.

Extensions: GLMs and Heavy-Tailed Regression

The AC-IHT framework generalizes beyond Gaussian linear models. In the context of generalized linear models (GLMs), the only modification required is to the gradient step, replacing squared error with negative log-likelihood and adjusting for the canonical link. The theory extends to establish analogous estimation and adaptivity rates under appropriate regularity conditions on the link function and response distribution.

The authors also formalize a connection to the regime of heavy-tailed noise. By decomposing heavy-tailed errors as the sum of a truncated “light-tailed” component and a sparse contamination vector (outlierization at large residuals), the contaminated linear model subsumes heavy-tailed settings. AC-IHT achieves minimax rates for regression when only low moments are finite, matching best-known results for robust M-estimators.

Practical and Theoretical Implications

Practical Implications:

The AC-IHT algorithm is computationally efficient, requiring only basic gradient and thresholding operations (similar to classical IHT) and does not necessitate data splitting for debiasing. Its adaptive regularization and signal separation allow the practitioner to obtain both minimax-optimal estimation and valid variable selection without tuning penalties for shrinkage bias. The ability to retain strong oracle properties—even under adversarial contamination—is highly advantageous for reliable inference in real-world high-dimensional applications where data may be corrupted in arbitrary ways.

Theoretical Significance and Future Directions:

This work provides the first explicit, sharp signal-adaptive rates for contaminated sparse regression, refines understanding of the interplay between signal strength and robust recovery, and extends the advantages of algorithmic regularization to contaminated regimes. Future work may close residual logarithmic gaps in the bounds, establish theory for adaptive threshold choice, and further relax the design and noise assumptions beyond the sub-Gaussian paradigm (noted limitations include the extension to heavy-tailed designs).

Conclusion

This paper establishes a rigorous, yet computationally practical, framework for robust high-dimensional regression under adversarial contamination. The proposed AC-IHT algorithm achieves minimax near-optimality and exhibits novel signal adaptivity, exact support recovery, and asymptotic inference properties—features not previously obtained in this setting. The approach’s generality encompasses contaminated GLMs and heavy-tailed regression, strengthening both practical methodology and the landscape of robust statistical theory.