Multi-task Linear Regression

• Multi-task linear regression is a framework that jointly models related tasks by sharing common structures such as sparsity or clusters.
• It utilizes convex clustering, group sparsity, and hierarchical penalties to enforce similarity and improve estimation accuracy.
• Empirical studies demonstrate its superiority over independent regressions in applications like educational analytics and neuroimaging.

Multi-task linear regression refers to a class of models and algorithms that simultaneously solve multiple related regression tasks, leveraging shared information to improve estimation and prediction. In contrast to fitting independent linear regressions for each task, multi-task approaches regularize or couple task-specific models to capture structural similarities—such as clustered parameter organization, shared sparsity, hierarchical relations, or graph-based proximity—yielding improved performance, especially in high-dimensional or low-sample regimes. This article surveys key frameworks, theoretical results, and algorithmic developments, with emphasis on recent convex clustering approaches and their empirical implications.

1. Convex Clustering Formulation and its Extensions

One influential paradigm for multi-task regression is convex clustering-based multi-task learning (MTLCVX) (Okazaki et al., 2023). Given $T$ tasks, each associated with input matrix $X_m\in\mathbb R^{n_m\times p}$ and response vector $\bm y_m\in\mathbb R^{n_m}$, the method introduces both per-task regression coefficients $\bm w_m\in\mathbb R^p$ and centroids $\bm u_m\in\mathbb R^p$. The regularized convex objective is

$$\min_{\{\bm w_m,\,\bm u_m\}_{m=1}^T} \sum_{m=1}^T \Bigl[\frac{1}{2n_m}\|\bm y_m-X_m\bm w_m\|_2^2 + \frac{\lambda_1}{2}\|\bm w_m-\bm u_m\|_2^2 \Bigr] + \lambda_2\sum_{(m,\ell)\in\mathcal E} r_{m\ell}\|\bm u_m-\bm u_\ell\|_2$$

where $\mathcal E$ is a weighted graph connecting tasks, $r_{m\ell}$ are nonnegative affinities based typically on $k$-NN proximity of single-task estimates, and $\lambda_1,\lambda_2$ are tunable regularization weights. The "regression–centroid" coupling term enforces similarity of each task's coefficients to a latent centroid, while convex clustering on the centroids encourages tasks to fuse into clusters in parameter space. For generality, the per-task loss can be a convex function such as logistic loss.

This decoupling of regression and clustering parameters yields a strictly convex objective solvable by block coordinate descent: alternating between updating centroids (convex clustering with ADMM/proximal-gradient (Nesterov-accelerated) solvers) and per-task coefficients (ridge-type updates for squared loss, Newton–Raphson for logistic loss). Model selection involves grid search over $X_m\in\mathbb R^{n_m\times p}$0 and $X_m\in\mathbb R^{n_m\times p}$1 with affinity graphs constructed via initial $X_m\in\mathbb R^{n_m\times p}$2-NN on OLS, ridge, or lasso fits. Notable properties include robustness to noisy affinities and automatic cluster recovery when tasks are organized into well-separated groups.

Empirical studies (Okazaki et al., 2023) demonstrate that, on synthetic tasks with ground-truth clusters and on practical benchmarks such as school performance and landmine detection, this framework outperforms both network-lasso baselines and independent-task regressions, especially under affinity noise or cluster heterogeneity.

2. Structured Regularization and Hierarchical/Graphical Approaches

Beyond convex clustering, multi-task linear regression incorporates application-specific structural regularizations:

• Feature-level priors and group sparsity: Models such as (Zhang et al., 2023) combine entrywise convex penalties for shared feature selection (via $X_m\in\mathbb R^{n_m\times p}$3 norm promoting row sparsity in $X_m\in\mathbb R^{n_m\times p}$4) with smoothness across tasks (e.g., fused differences $X_m\in\mathbb R^{n_m\times p}$5) and generalized quadratic priors connecting features (encoded via a matrix $X_m\in\mathbb R^{n_m\times p}$6 such that $X_m\in\mathbb R^{n_m\times p}$7 enforces similarity between coefficients for features known to be related). Proximal-gradient methods (ISTA/FISTA, linear-convergence variants) efficiently solve the resulting objective.
• Fusion/hierarchical penalties: Task clustering via pairwise $X_m\in\mathbb R^{n_m\times p}$8 fusion penalties (e.g., $X_m\in\mathbb R^{n_m\times p}$9) (Yu et al., 2017) can induce a complete task hierarchy (latent dendrogram) reflecting parameter similarity, even with unknown a priori groupings. Such approaches allow tasks to share both support (via $\bm y_m\in\mathbb R^{n_m}$0) and identical coefficients within clusters, with global optima found via proximal decomposition.
• Geometric Wasserstein regularization: To exploit known geometry of regressor variables, unbalanced optimal transport penalties are applied across tasks to encourage spatially coherent, but not necessarily identical, support (Janati et al., 2018). Here, convex coupling is achieved via transport cost on the coefficient vectors, allowing flexible, geometry-aware sharing without the rigid overlap constraints of $\bm y_m\in\mathbb R^{n_m}$1 group norms.
• Tensor-structured and low-rank decompositions: Multimodal or multi-indexed task relations can be encoded using tensor CP decompositions in the weights, as in tensorized least-squares SVM approaches (Liu et al., 2023), unifying low-rank subspace, single-task, and structured multimodal settings in a joint alternating-optimization framework.

3. Theoretical Guarantees and Statistical Recovery

The theoretical literature establishes sample complexity and recovery guarantees for multi-task linear regression models under broad regimes:

• Support union recovery: For high-dimensional sparse settings, block $\bm y_m\in\mathbb R^{n_m}$2-regularized multi-task lasso achieves a sharp threshold on sample size necessary for exact recovery of the support union across tasks (Wang et al., 2013). The threshold function depends on the sparsity, covariance structure, and inter-task overlap. Significant gains in per-task sample complexity arise when supports are shared, as opposed to disjoint, among tasks.
• Generalization error and meta-learning: Mixtures of linear regressions (clustered tasks) can be learned via spectral and clustering-based meta-learning, with provable rates depending on the regime of "light" vs. "heavy" task data allocations (Kong et al., 2020). Recent robust meta-learning variants extend these guarantees to adversarial contamination and batch-size heterogeneity (Kong et al., 2020).
• Adaptivity and safety without covariance eigenvalue lower bounds: Newer frameworks relax the often unrealistic requirement of minimal empirical second-moment eigenvalues for each task (Kim, 16 May 2026). By introducing a matrix-weighted norm penalty—$\bm y_m\in\mathbb R^{n_m}$3 with a global centroid $\bm y_m\in\mathbb R^{n_m}$4 and per-task adaptivity—the resulting estimator achieves minimax-optimal rates under mild balancedness conditions, is robust to arbitrary (even adversarial) outlier tasks, and enjoys safety (falling back to independent learning) in the absence of transfer structure.
• High-dimensional risk, hyperparameter selection, and random matrix theory: Tools from random matrix theory yield precise, consistent estimates of training and test risk in multi-task regression under high-dimensional growth and non-Gaussianity, supporting optimal hyperparameter calibration and explaining signal/noise tradeoffs between shared/global-task and task-adaptive regularization (Ilbert et al., 2024).

4. Algorithms and Computational Considerations

Multi-task linear regression employs several recurring algorithmic primitives:

Framework	Regularization/Structure	Optimization Method
Convex clustering (MTLCVX)	Centroid-fused clustering	Block coordinate descent, ADMM, PGD
Graph/Network Lasso	Task-graph Laplacian coupling	Ridge updates, OSLSSVR, WRLS
Feature-sharing sparsity	$\bm y_m\in\mathbb R^{n_m}$5 row-sparsity, prior D	Proximal-gradient (ISTA/FISTA, linear)
Wasserstein geometry	OT-based support transport	Alternating Sinkhorn + coordinate dec.
Task hierarchy/fusion	Pairwise $\bm y_m\in\mathbb R^{n_m}$6 fusion	Proximal decomposition
Tensorized subspace	CP-low-rank tensors	Alternating linear systems

Most are convex and admit global optima. Sparse affinity matrices, precomputation (e.g., Cholesky/SVD factorizations), and block-structured updates are crucial for scalability. For streaming or online settings, recursive least squares and kernel approaches have been developed to give exact (MT-WRLS) or sparse approximate (MT-OSLSSVR) recursions with quadratic per-instance cost, outperforming OGD or cubic ADMM online schemes (Lencione et al., 2023).

5. Practical Applications and Empirical Results

Multi-task linear regression has demonstrated significant empirical advantages in diverse domains:

• Educational and remote sensing analytics: School examination data and large-scale plant trait modeling (Okazaki et al., 2023, Yu et al., 2017) show that task-sharing significantly improves prediction, with convex clustering and fusion penalties yielding both interpretability (dendrogram/task groupings) and robust performance across settings with variable task homogeneity.
• Neuroimaging and scientific data: OT-based geometric regularization produces more spatially coherent and statistically powerful predictors in cortical signal recovery compared to rigid group-sparse approaches (Janati et al., 2018).
• Time series and forecasting: Multivariate time-series forecasting via MTL convex regularization outperforms standard independent-channel architectures across real-world benchmarks, with consistent risk estimators facilitating hyperparameter tuning (Ilbert et al., 2024).
• Classification and outlier regimes: Multi-task methods, when correctly tuned (e.g., in landmine detection), recover meaningful task clusters and deliver strong AUC improvement over single-task and naive multitask methods (Okazaki et al., 2023).

6. Extensions: Function-valued Tasks and Distributed Learning

Function-valued (infinite-dimensional) multi-task regression generalizes linear relations to curves and functions, with splines or reduced-rank representations regularized by manifold constraints and composite quadratic penalties (He et al., 2022). Theoretical analysis connects norm-induced geometry, manifold curvature, and empirical process complexity—yielding unified error bounds, sharp phase transitions, and optimal rates, including for graph-Laplacian and reduced-rank models.

On the systems side, distributed and asynchronous algorithms operating on networked subgraphs of tasks combine local consensus penalties with global Mahalanobis regularization, delivering finite-time convergence guarantees and practical guidance for selecting step sizes and penalties (Hong et al., 2024).

7. Statistical Inference and Covariance Estimation

Inter-task noise correlations, crucial in simultaneous inference and calibration, can be accurately estimated via bias-corrected residual covariance methods built on multi-task lasso/elastic-net estimators (Tan et al., 2022) or via minimal-penalty techniques yielding operator-norm-consistent noise covariance estimates and supporting optimal oracle-inequality generalization error bounds (Solnon et al., 2011).

These statistical tools enable the construction of valid confidence intervals and joint inference regions, accounting for complex noise structures that would otherwise compromise downstream inference and regularization tuning.

---

In summary, multi-task linear regression unifies a suite of convex, interpretable, and theoretically grounded frameworks for leveraging task similarity, prior structure, and inter-task relations. The field has advanced from early group-lasso and fusion penalties to sophisticated clustering, geometric, and tensorized models, offering both practical numerical schemes and strong theoretical guarantees across classic, high-dimensional, outlier-robust, and functional data regimes (Okazaki et al., 2023, Wang et al., 2013, Kim, 16 May 2026, Zhang et al., 2023, Yu et al., 2017, Janati et al., 2018, He et al., 2022, Ilbert et al., 2024).