Robust Regression with Adaptive Contamination in Response: Optimal Rates and Computational Barriers

Abstract: We study robust regression under a contamination model in which covariates are clean while the responses may be corrupted in an adaptive manner. Unlike the classical Huber's contamination model, where both covariates and responses may be contaminated and consistent estimation is impossible when the contamination proportion is a non-vanishing constant, it turns out that the clean-covariate setting admits strictly improved statistical guarantees. Specifically, we show that the additional information in the clean covariates can be carefully exploited to construct an estimator that achieves a better estimation rate than that attainable under Huber contamination. In contrast to the Huber model, this improved rate implies consistency even when the contamination is a constant. A matching minimax lower bound is established using Fano's inequality together with the construction of contamination processes that match $m> 2$ distributions simultaneously, extending the previous two-point lower bound argument in Huber's setting. Despite the improvement over the Huber model from an information-theoretic perspective, we provide formal evidence -- in the form of Statistical Query and Low-Degree Polynomial lower bounds -- that the problem exhibits strong information-computation gaps. Our results strongly suggest that the information-theoretic improvements cannot be achieved by polynomial-time algorithms, revealing a fundamental gap between information-theoretic and computational limits in robust regression with clean covariates.

• The paper introduces an estimator based on truncated regression depth that achieves minimax optimal rates under adaptive response contamination.
• It exploits clean covariates to dilute the influence of corrupted responses, offering a statistically improved rate over classical Huber models.
• Despite these advances, the work proves that no polynomial-time algorithm can attain the optimal rates, establishing a significant computational barrier.

Robust Regression with Adaptive Contamination: Statistical-Computational Gaps

Problem Setting and Motivation

This paper investigates robust regression under a contamination model in which responses can be corrupted adaptively, but covariates remain clean—a key departure from classical Huber's $\epsilon$-contamination, where both $X$ and $y$ may be adversarially perturbed. Specifically, the work considers settings with $X_i \sim \mathcal{N}(0, I_p)$ and $$y_i \mid X_i \sim (1-\epsilon)\mathcal{N}(X_i^\top \beta, \sigma^2) + \epsilon Q_{X_i}$$, allowing the contaminating response distribution to depend arbitrarily on each covariate instance. The central questions addressed are:

1. Whether leveraging clean covariates enables strictly improved minimax estimation rates, even for constant $\epsilon$,
2. Whether any polynomial-time estimator can attain these improved rates.

Theoretical Results: Statistical Optimality

Information-Theoretic Rates

The authors demonstrate a strict information-theoretic improvement over the Huber setting. For Huber contamination (joint $X, y$), the minimax $\ell_2$ estimation error rate for $\beta$ is $$\Theta\Big( \sigma \big( \sqrt{p/n} + \epsilon \big) \Big)$$, with the additive $X$0 term acting as an information-theoretic obstacle to consistent estimation at constant $X$1.

In contrast, under this clean-covariate, adaptively contaminated response model, the paper constructs an estimator based on a generalized “truncated” regression depth functional that achieves:

$X$2

This rate is shown to be minimax optimal: lower bounds are matched via a Fano-based construction utilizing simultaneous matching in total variation over exponentially many hypotheses, extending well beyond standard two-point arguments. Notably, the dependence $X$3 enables consistent estimation as $X$4, for any fixed $X$5—a phenomenon provably impossible in the classical Huber or non-uniform contamination settings.

The technical core uses the fact that, with clean $X$6, the influence of a contaminated $X$7 can be diluted by conditioning on directions $X$8 where $X$9 is large—the 1D projections with heavy tails—thereby exploiting information from the extremes of the design.

Role of the Covariate Distribution

The minimax rate exhibits explicit dependence on the marginal tail behavior of $y$0: heavier-tailed distributions induce faster rates via the tail-mass in high-magnitude directions. This is formalized for generalized Gaussian designs, interpolating between Gaussian and Laplace tails.

Computational Lower Bounds: SQ and Low-Degree Barriers

Despite this statistical separation, the authors rigorously prove that no polynomial-time estimator (formalized via SQ and low-degree polynomial models) can exploit these information-theoretic benefits.

Statistical Query Lower Bound

Via an intricate reduction to conditional Non-Gaussian Component Analysis and constructions matching $y$1 moments with standard Gaussian, the authors show that any SQ-based estimator achieving estimation error $y$2 must either:

• Make superpolynomially many (exponential in $y$3) queries,
• Or require queries of tolerance $y$4. Hence, for polynomial runtime and sample sizes polynomial in $y$5, the best achievable error is $y$6. Thus, the gap between computational and statistical minimax rates is superpolynomial, not merely quadratic as in classic robust regression under oblivious contamination.

Low-Degree Polynomial Hardness

By leveraging reductions between the SQ and low-degree polynomial frameworks, identical sample complexity lower bounds hold for polynomial-time tests of all sufficiently low degree.

Related Contamination Models and Hierarchy

The paper situates adaptive contamination within a lattice of model classes, clarifying that:

• Oblivious response contamination (noise independent of $y$7) permits even stronger statistical consistency in the large-$y$8 regime.
• Huber with bounded marginal likelihood, or adversarial additive response outliers, do not share these statistical advantages.
• The class considered here provides the maximal adversarial power consistent with retaining statistical consistency in high dimensions, provided the design remains perfectly clean.

Practical and Theoretical Implications

The results identify a regime with a clear statistical-computational gap: in high dimensions, information-theoretic minimax estimators are infeasible, while all efficiently computable estimators must suffer $y$9 error scaling. The clean-covariate assumption, even when computationally exploited, is insufficient to overcome this computational barrier, in contrast with the folklore that robust statistics become tractable when only $X_i \sim \mathcal{N}(0, I_p)$0 is corrupted.

This has practical consequences for robust machine learning in high dimensions: leveraging clean features per se does not eliminate algorithmic hardness in adversarial contamination settings unless additional structure or restrictions on corruption are imposed.

Conclusion

This work precisely characterizes the minimax estimation rates and computational complexity of robust regression with adaptively contaminated responses and clean covariates. It establishes a strong information-computation gap, proving that the additional information in the clean design regime is statistically accessible only to infeasible, non-algorithmic estimators. These results sharpen the theoretical understanding of robust estimation and the limitations of computationally efficient methods, motivating further study into models that interpolate between adversarial and oblivious contamination with computationally tractable recovery.