Multi-task Linear Regression without Eigenvalue Lower Bounds: Adaptivity, Robustness and Safety

Abstract: We study the multi-task linear regression problem in the presence of contaminated tasks. We address the setting where the unknown parameters of a majority of tasks are close in the $\ell_2$-norm, while a fraction of tasks are arbitrary outliers. Existing theoretical frameworks for this problem rely heavily on the assumption that the empirical second moment of each task has a minimum eigenvalue bounded away from zero (order $Ω(1)$). Crucially, this assumption fails in many high-dimensional scenarios, rendering prior guarantees vacuous. To overcome this limitation, we propose an estimator based on matrix-weighted norm regularization. We also introduce a relative balancedness condition, quantified by a balancedness constant, that compares each task's second moment with the average inlier geometry and relaxes the need for taskwise second-moment lower bounds. In favorable regimes with moderate balancedness, our prediction MSE bounds match the rate of Duan and Wang (2023) under substantially weaker spectral assumptions; the resulting task-overall MSE is minimax optimal up to logarithmic factors. Furthermore, we demonstrate that our estimator enjoys a safety guarantee: when the relevant balancedness constant is large or infinite, or when tasks are unrelated, the method performs no worse than independent task learning. Consequently, our methodology achieves simultaneous adaptivity to task similarity, robustness to outliers, and safety outside favorable transfer regimes.

• The paper presents a matrix-weighted estimator that bypasses the need for eigenvalue lower bounds in multi-task linear regression.
• It achieves minimax optimality with robust transfer even under high-dimensional, singular designs and outlier task contamination.
• Empirical validations demonstrate improved prediction accuracy and safety, ensuring graceful degradation when tasks are unrelated.

Multi-task Linear Regression under Mild Spectral Assumptions: Adaptivity, Robustness, and Safety

Introduction and Motivation

The problem of multi-task learning (MTL) in linear models arises when one seeks to jointly estimate model parameters for multiple related tasks, exploiting task relatedness to achieve improved statistical efficiency compared to independent-task estimation. Conventional analyses and algorithms in MTL, particularly in transfer-robust settings where a minority of tasks may be unrelated outliers, have fundamentally relied on the assumption that the empirical second-moment matrix for each task has a uniformly lower-bounded spectrum—i.e., every task’s design is well-conditioned with eigenvalues $\gtrsim 1$. However, this lower-boundedness condition is often violated in realistic high-dimensional regimes where design matrices may be rank-deficient or exhibit rapidly decaying spectra. The work "Multi-task Linear Regression without Eigenvalue Lower Bounds: Adaptivity, Robustness and Safety" (2605.17126) systematically addresses this limitation by developing an estimator and theoretical framework that achieves robust, adaptive multi-task linear regression guarantees without any eigenvalue lower bounds.

Problem Formulation and Technical Innovations

Contaminated Task Model

The considered setup involves $m$ regression tasks with shared or similar parameter vectors for a majority "inlier" fraction, while the remaining "outliers" can be arbitrarily unrelated. The inlier parameters are assumed to lie within an $\ell_2$-ball of radius $\delta$, and the outlier fraction $\varepsilon$ is unknown. The objective is to minimize per-task mean squared prediction error, either in-sample (empirical risk) or population (prediction risk), without prior knowledge of which tasks are inliers or the degree of relatedness.

Evasion of Lower-Boundedness of Covariate Spectrum

Prior robust multi-task learning methods, such as ARMUL [duan2023adaptive], rely on strong convexity (i.e., minimum eigenvalue lower bounds) to obtain high-probability MSE bounds that improve over independent-task learning (ITL) whenever $\delta$ is small. The present work provides a structural relaxation: instead of requiring every $_j$ to be well-conditioned, it introduces a relative balancedness assumption—quantified by a balancedness constant $B$—which posits only that each task’s empirical covariance is upper-bounded compared to the averaged inlier geometry. Notably, $B$ remains finite even when individual $_j$ are singular or have rapidly decaying spectra, provided their principal directions have sufficient overlap.

Matrix-Weighted Regularization

A matrix-weighted norm penalty replaces the standard isotropic $m$0 regularization from prior work. The estimator solves:

$m$1

where $m$2 is the empirical loss for task $m$3, and the penalty geometry is tailored to $m$4. This construction aligns regularization with the observed design, avoiding over-penalization in weakly observed directions and obviating the need for a uniform lower eigenvalue bound.

Theoretical Guarantees

Adaptivity, Robustness, and Safety

Central contributions are captured by a simultaneous set of properties:

• Adaptivity: The estimator exploits inlier task similarity whenever moderate balancedness holds ($m$5 is not too large), achieving minimax-optimal task-average prediction risk (up to logarithmic and $m$6-dependent factors), with no prior knowledge of $m$7 or $m$8 required.
• Robustness: Statistical efficiency is retained even in the presence of an unknown fraction $m$9 of arbitrary outlier tasks.
• Safety: When $\ell_2$0 is large or tasks are unrelated, the estimator provably degrades gracefully, matching the ITL rate with no negative transfer.

Finite-Sample Risk Bounds

The main risk bound (in-sample MSE for inlier task $\ell_2$1) is:

$\ell_2$2

with no explicit dependence on the minimum eigenvalue of $\ell_2$3. The estimator’s worst-case prediction error across all tasks satisfies

$\ell_2$4

matching the independent-task rate universally.

Population Risk and Sample Complexity

The transfer from empirical to population MSE is governed by a task-specific comparability constant $\ell_2$5 (ratio of empirical to population covariance). When $\ell_2$6 and the design is sufficiently concentrated, $\ell_2$7 and the minimax transfer regime persists for population risk. High-probability analyses under general sub-exponential designs are included, with explicit sample complexity characterization.

Empirical Validation

Synthetic experiments verify that the estimator matches or outperforms baselines (ITL, DP, and ARMUL) in diverse regimes, including:

• Varying Inlier Radius $\ell_2$8: When inliers are tightly clustered, substantial gains are obtained on both all-task and related-task splits.

  Figure 1: Population MSE across all, related, and outlier tasks as inlier radius $\ell_2$9 is varied; matrix-weighted method consistently dominates on all-task and related-task MSE.

• Increasing Outlier Fraction $\delta$0: The method retains sharp error increases only gracefully as contamination rises, preserving robustness.

  Figure 2: All-task, related-task, and outlier-task MSE as $\delta$1 is swept; the estimator preserves performance on related tasks even under high contamination.

• Eigendecay $\delta$2 Stress: For models with rapidly decaying spectra, the matrix-weighted estimator preserves strong transfer, unlike methods requiring lower bounds.

  Figure 3: Effects of increasing eigendecay exponent $\delta$3; performance gains persist for matrix-weighted method even in near-singular regimes.

• Large-Balancedness ($\delta$4) Stress Test: When transfer geometry is unfavorable ($\delta$5 large), the estimator tracks ITL risk without negative transfer.

  Figure 4: All-task, related-task, and outlier-task MSE as population balancedness $\delta$6 increases; method never suffers catastrophic negative transfer, matching ITL in high-$\delta$7 regimes.

On the UCI Human Activity Recognition dataset, the estimator achieves a mean classification error of 1.27%, outperforming ITL (4.67%), DP (7.61%), and ARMUL (5.24%).

Implications and Future Directions

Theoretical Impact

This work demonstrates that lower eigenvalue assumptions are not inherent to robust transfer in multi-task linear regression. The relative balancedness condition is both necessary and sufficient for nontrivial transfer, and the safety property removes risk of negative transfer altogether. The analysis accommodates both degenerate (singular) and nearly singular designs, broadening practical applicability and aligning theoretical guarantees with empirical realities in high dimensions.

Practical Applicability

The matrix-weighted approach is implementable using standard convex optimization routines, requiring only empirical covariances—no parametric knowledge of $\delta$8 or task similarity parameters is needed. The empirical balancedness constant $\delta$9 additionally serves as a diagnostic for the potential utility of transfer on new datasets.

Extensions and Open Problems

Future research directions include sharpening the dependence on $\varepsilon$0 and extending these geometric insights to infinite-dimensional models: RKHS regression, kernel and neural tangent regimes, and overparameterized neural networks. These developments are likely to influence adaptive meta-learning and multi-task reinforcement learning, where spectrum degeneracy and outlier contamination arise naturally.

Conclusion

"Multi-task Linear Regression without Eigenvalue Lower Bounds: Adaptivity, Robustness and Safety" (2605.17126) provides a theoretically complete and practically implementable solution to robust multi-task estimation under minimal spectral assumptions. It delivers sharp adaptivity and minimax optimality under mild balancedness, robust outlier handling, and provable safety in all settings, fundamentally altering the requirements for information transfer in high-dimensional multi-task regression.