Contaminated Multi-task Learning with Heterogeneity: Fundamental Limits and Optimal Algorithms

Abstract: Integrating information across related tasks can improve estimation and prediction in transfer, multi-task, and federated learning, but contamination and heterogeneity make robust borrowing challenging. We study a contaminated multi-task empirical risk minimization (ERM) framework in which an $ε$ fraction of $K$ tasks, each with sample size $n$, may be arbitrarily contaminated while the remaining tasks are heterogeneous. Our goal is to estimate both the global minimizer of the average risk and the clean task-specific minimizers, thereby combining robustness and personalization. In the Gaussian mean model, we show that several common paradigms, including adaptive and robust regularization around a shared center, global matrix regularization, decomposition-based regularization, and score-based outlier-task detection, all suffer from a worst-case contamination error of order $ε\sqrt{d/n}$, which is suboptimal compared to the lower bound $ε/\sqrt{n}$. This identifies a dimension-dependent barrier for these approaches. We then establish minimax lower bounds for a general heterogeneous ERM setting and propose a computationally efficient filtering-based robust multi-task gradient descent method. Under local strong convexity, smoothness, and sub-Gaussian gradient assumptions, the proposed method attains high-probability upper bounds matching the minimax rates up to logarithmic factors over a broad regime. In particular, it removes the extra $\sqrt{d}$ contamination dependence of many regularization-based methods and score-based outlier detection, while achieving personalization to local tasks under strong heterogeneity. Simulations and a real-data analysis demonstrate strong robustness and personalization relative to a broad range of benchmark methods.

• The paper derives sharp minimax lower bounds, revealing that traditional robust methods suffer from a √d contamination barrier in high dimensions.
• It introduces a robust multi-task gradient descent (RMT-GD) algorithm that employs joint robust mean estimation with adaptive covariance filtering to achieve near-optimal rates.
• Extensive experiments demonstrate that the proposed algorithm significantly reduces global and local estimation errors, even with up to 25% contamination.

Robust Multi-task Learning under Joint Contamination and Heterogeneity: Fundamental Barriers and Minimax-Optimal Solutions

Introduction and Problem Formulation

The paper "Contaminated Multi-task Learning with Heterogeneity: Fundamental Limits and Optimal Algorithms" (2607.02681) advances the statistical theory of multi-task learning (MTL) when both heterogeneity and task-level contamination are present. Specifically, the authors consider a regime where $K$ related tasks each provide $n$ data points: while the majority are clean but potentially heterogeneous, an adversarial fraction $\epsilon$ may be arbitrarily contaminated. Both estimation of a global parameter (shared among tasks) and local task-specific parameters are considered central objectives.

This model captures realistic federated and decentralized scenarios—applications where data heterogeneity and unreliability coexist, such as hospitals with different patient populations and adversarial edge devices. The main challenge is to design procedures which simultaneously leverage information sharing (to improve sample efficiency) while avoiding breakdowns due to contamination and miscalibration due to heterogeneity.

Fundamental Barriers of Classical MTL Paradigms

The authors first establish that several widely-used paradigms—such as adaptive/robust regularization toward a shared center, global matrix regularization (e.g., group lasso), decomposition-based regularization (e.g., the dirty model), and score-based outlier detection—exhibit a provable failure in this setting. Specifically, for the mean estimation problem, all these approaches incur a worst-case contamination error term of order $\epsilon \sqrt{d/n}$, where $d$ is problem dimension and $n$ per-task sample size.

This $\sqrt{d}$-dependent gap shows these methods cannot attain the minimax lower bound $O(\epsilon/\sqrt{n})$. Thus, without dimension reduction or structural assumptions, regularization-based and score-based detection approaches encounter a fundamental statistical barrier. The arguments are precise and negative: they apply to broad classes of penalties (including lasso, ridge, SCAD, MCP, and more), and extend to generalized regularization and detection algorithms.

Notably, techniques such as geometric median or coordinate-wise median (used for robust aggregation in federated learning [chen2017distributed; yin2018byzantine]) are shown to be similarly affected by this dimensionality curse when used as subroutines for gradient or parameter aggregation. The analysis reveals that increasing flexibility in regularizer choice does not eliminate the core problem.

Minimax Lower Bounds

The formal minimax lower bounds derived in the paper precisely quantify the cost imposed by heterogeneity and contamination, showing for both global ($\theta^*$) and local ($\theta^{(k)}$) parameters: $n$0

$n$1

Here, $n$2 and $n$3 quantify global and local heterogeneity in task gradients, and all terms are explicit, sharp, and model-agnostic.

Importantly, these lower bounds show that, unless $n$4, the information-theoretic optimal robust risk cannot have leading contamination error term scaling as $n$5. This demonstrates that previously proposed regularization and robust aggregation methods cannot close the gap in the high-dimensional regime.

Nearly Minimax-Optimal Algorithms via Gradient Filtering

The core positive contribution is a methodologically simple yet theoretically powerful filtering-based robust multi-task gradient descent (RMT-GD). The estimator combines three ingredients:

• At each iteration, jointly robust mean estimation (JRGE) is applied to task gradients, utilizing a computationally tractable filtering algorithm inspired by modern robust statistics [diakonikolas2019robust; diakonikolas2023algorithmic].
• Gradients are aggregated using an explicit robust mean estimator parameterized by a data-driven covariance estimate; this step is shown to circumvent the high-dimensional breakdown that plagues coordinate-wise and geometric medians.
• Local task updates optionally incorporate a proximal step to encourage task-wise personalization.

A crucial technical contribution is the proposal and analysis of a practical, consistent estimator of the cross-task gradient covariance even under contamination and heterogeneity. The algorithm only requires empirical per-task covariance estimates and a simple robust selection rule, sidestepping expensive convex programming or intractable depth estimators.

The resulting upper bounds for parameter estimation match the minimax lower bounds up to logarithmic factors, removing the extra $n$6 factor previously proven to be inherent to regularization-based and score-based detection algorithms. This means the estimator:

• Is robust in the presence of arbitrary $n$7-fraction contamination;
• Adapts to heterogeneity, retaining information-sharing benefits when tasks are similar;
• Achieves strong personalization for task-specific parameters without suffering from information leakage or breakdowns due to contaminated tasks.

Numerical Results

Extensive synthetic and real-data experiments substantiate the theory. Under controlled linear regression simulation with varying heterogeneity, contamination levels, and network sizes, the proposed method uniformly attains the smallest or near-smallest global and local parameter errors across all regimes—most notably, it consistently outperforms classical and recent robust aggregation baselines once heterogeneity and contamination co-occur.

For example, in high-heterogeneity and contamination settings, global error for the proposed estimator remains stably low ($n$8–$n$9 vs. $\epsilon$0–$\epsilon$1 for competitors), while local estimation accuracy tracks personalization ($\epsilon$2–$\epsilon$3). Application to federated logistic regression on Human Activity Recognition yields best-in-class classification error across clean and contaminated tasks, even as contamination rates increase to $\epsilon$4.

Implications and Future Directions

This work provides a unifying theoretical framework demonstrating the inherent statistical limits of regularization-based and score-based MTL strategies under simultaneous contamination and heterogeneity, identifying a previously unrecognized dimension-dependent barrier. By constructing minimax-optimal bounds and providing a practical algorithmic strategy (robust filtering and gradient aggregation with adaptive covariance estimation), it resets expectations for what is achievable in robust federated and multi-task regimes.

The results suggest that robust gradient filtering techniques, equipped with covariance adaptation, offer a principled and tractable means to achieve both robustness and personalization in decentralized, high-dimensional, heterogeneous environments. These findings are likely to guide algorithmic choices in real federated learning deployments, especially in privacy- and security-critical domains such as healthcare, IoT, and mobile sensing.

From a theoretical standpoint, the work clarifies the statistical price paid for information-sharing and adaptation in contaminated, distributed settings. Practically, it indicates that robust algorithms designed for high-dimensionality—rather than single-coordinate or geometric aggregation schemes—are essential for optimality.

Future research should explore extension of the filtering architecture to complex loss landscapes beyond strongly convex ERM, deeper integration with differential privacy mechanisms, and the adaptation of these results to nonparametric, sequential, or streaming MTL/federated models. Interaction with strong privacy guarantees is of particular interest due to the close mathematical connection between robustness and privacy in mean estimation [hopkins2022robustness].

Conclusion

This paper rigorously characterizes both the limitations and the achievable statistical rates for multi-task learning and federated learning under adversarial contamination and heterogeneity. It demonstrates that classical regularization and most existing robust aggregation schemes are fundamentally limited in high-dimensional contaminated regimes, while robust gradient filtering—coupled with explicit covariance estimation—achieves minimax optimality. These insights precisely delineate feasible robustness-personalization tradeoffs and set new methodological and theoretical standards for federated and MTL in untrusted, heterogeneous environments.