Multi-Task Linear Regression Insights

• Multi-task linear regression is a framework that fits multiple related linear models using shared regularization to leverage inter-task relationships.
• It employs diverse regularizers—including graph-based, block-sparsity, and optimal transport—to enforce similarity, select features, and enable clear structural discoveries.
• Applications span time series forecasting, neuroimaging, and distributed networks, demonstrating improved prediction accuracy, sample efficiency, and scalability.

Multi-task linear regression is a paradigm that simultaneously fits multiple related linear predictive models by coupling their parameter estimation through shared regularization or structural constraints. The goal is to leverage relationships among tasks—be they known a priori or inferred from data—to improve generalization, sample efficiency, and interpretability relative to single-task approaches. The multi-task objective is typically formulated as a sum of task-specific prediction losses augmented by penalties or constraints that encode inter-task similarity, feature selection, shared support, or other structural information. Recent advances formalize these couplings via graph-based regularizers, convex clustering, block-sparsity, geometric transport, tensor decompositions, and manifold constraints, enabling broad applicability across regression, classification, online learning, time series forecasting, federated/distributed networks, and functional data analysis.

1. Core Formulations and Regularization Strategies

Canonical multi-task linear regression models parameterize each task $t$ with a vector $w_t$ and design matrix $X_t$, seeking to minimize

$$J(w) = \sum_{t=1}^T \|X_t w_t - y_t\|_2^2 + \lambda R(w)$$

where $R(w)$ couples the $w_t$ according to prior knowledge or learned relationships.

Graph-based regularization. In "Online Multi-Task Learning with Recursive Least Squares and Recursive Kernel Methods" (Lencione et al., 2023), $R(w)$ is a quadratic penalty

$$R(w) = \sum_{t=1}^T \sum_{j \in E_t} \| w_t \cdot \mathrm{sim}(t,j) - w_j \cdot \mathrm{sim}(j,t) \|_2^2 + \gamma \sum_t \|w_t\|_2^2$$

which is equivalently written as $w^\top(A \otimes I_d)w$ for a graph Laplacian $A$, enforcing smoothness of parameter vectors over the task-relationship graph.

Task-feature prior and group sparsity. The formulation in "Multi-Task Learning with Prior Information" (Zhang et al., 2023) unifies data fit, group-sparse feature selection via $\ell_{2,1}$ norm, prior-encoded feature constraints $D W$, and temporal or spatial task-smoothness:

$$\min_W \frac{1}{2} \sum_t \|X^{(t)} w^{(t)} - y^{(t)}\|_2^2 + \lambda \|W\|_{2,1} + \frac{\theta}{2} \|D W\|_F^2 + \frac{\epsilon}{2} \sum_{t=1}^{T-1} \| w^{(t)} - w^{(t+1)} \|_2^2$$

yielding a convex but nonsmooth program.

Block-$\ell_{1}/\ell_{2}$ regularization. Multi-task Lasso (multi-response regression) enforces joint support recovery using

$$\min_B \frac{1}{2n} \sum_{k=1}^K \| y^{(k)} - X^{(k)} \beta^{(k)} \|_2^2 + \lambda \sum_{j=1}^p \| B_{j, \cdot} \|_2$$

which promotes row-sparsity in $B=[\beta^{(1)},\ldots,\beta^{(K)}]$—i.e., shared selection of features across tasks (Wang et al., 2013).

Cross-learning and pairwise constraints. Cerviño et al. (Cervino et al., 2020) introduce

$$J(W) = \sum_{t=1}^T \|y_t - X_t w_t\|^2 + \lambda \sum_{t < s} \|w_t - w_s\|^2$$

where the pairwise penalty corresponds to a complete-graph Laplacian, admitting closed-form solutions and efficient projected SGD/dual projection algorithms.

2. Structural Learning: Task Relationships, Clustering, and Precision Graphs

Sparse precision estimation. The Multi-task Sparse Structure Learning (MSSL) model (Goncalves et al., 2014) jointly estimates the parameter matrix $W$ and the task precision matrix $\Theta$:

$$\min_{W,~\Theta \succ 0} \frac{1}{2} \sum_t \| y_t - X_t w_t \|_2^2 + \mathrm{Tr}(W \Theta W^\top) - \frac{T}{2} \log \det \Theta + \lambda \|\Theta\|_1 + \gamma \|W\|_1$$

recovering both task clusters/structure and parameter sparsity via alternating minimization (proximal gradient or ADMM for $W$, graphical lasso for $\Theta$).

Convex hierarchical clustering. In (Yu et al., 2017) and (Okazaki et al., 2023), convex clustering penalties such as $\sum_{(t,t')} r_{tt'} \| u^{(t)} - u^{(t')} \|_2$ (where $u^{(t)}$ are cluster-centroid parameters) induce hierarchical partitions of tasks. Solutions track cluster fusion as the penalty parameter increases, achieving interpretable grouping and flexible sharing.

3. Advanced Geometric and Tensor Regularization

Geometry-aware regularization. "Wasserstein regularization for sparse multi-task regression" (Janati et al., 2018) implements coupling via optimal transport between the (absolute values of) regression coefficients, allowing supports to align according to arbitrary feature geometry (distance matrix $M$), without requiring overlap:

$$\Omega(\beta^{(1)},\ldots,\beta^{(T)}) = \sum_{t < t'} W_\epsilon^\tau ( |\beta^{(t)}|, |\beta^{(t')}| )$$

where $W_\epsilon^\tau$ is the unbalanced entropic optimal transport divergence.

Tensorized multi-modal regression. The tLSSVM-MTL framework (Liu et al., 2023) leverages a CP decomposition over a high-order weight tensor, coupling via shared and task-specific latent factors across indexing modes. Each block update solves a linear system enforcing multilinear relationships, accommodating general multimodal data.

4. Online, Distributed, and Asynchronous Multi-Task Regression

Online recursive multi-task RLS. The MT-WRLS algorithm (Lencione et al., 2023) achieves exact and immediate updates for

$$w^* = [ X^\top X + \lambda (A \otimes I_d) ]^{-1} X^\top y$$

with per-instance complexity $\mathcal{O}(d^2 T^2)$, outperforming cubic-cost ADMM or suboptimal OGD schemes.

Distributed networked multi-task learning. In (Hong et al., 2024), nodes estimate local linear models $w_i$, subject to in-group consensus penalties and cross-group precision coupling:

$$L(W) = \sum_i \ell_i(w_i) + \lambda_1 \sum_i \rho_i(w) + \lambda_2 \sum_\ell \rho_r(W^{(\ell)})$$

Convergence holds under bounded noise and strong convexity; asynchronous two-timescale updates minimize communication overhead and scale to heterogeneous, federated settings.

5. Statistical Guarantees, Sample Complexity, and Hyperparameter Selection

Sharp threshold for support recovery. (Wang et al., 2013) establishes that, under block-$\ell_1/\ell_2$ multi-task Lasso, the necessary and sufficient sample size for recovery of the support union $S$ is

$$n^*(s,p,K) \asymp \psi(B^*, \Sigma^{(1:K)}) \log(p-s)$$

with $\psi$ quantifying the joint sparsity and design structure. When support is shared, sample complexity per task drops by a factor $\sim 1/K$ compared to single-task Lasso.

Consistent regularization parameter estimation. Random matrix theory analysis (Ilbert et al., 2024) gives closed-form expressions for asymptotic training and test errors. The optimal regularization parameter $\lambda^*$ depends on signal-to-noise ratio and can be rigorously estimated from training data under high-dimensional limits.

6. Algorithmic Solutions and Convergence Properties

Optimization schemes include:

• Recursive updates (MT-WRLS, block-coordinate descent).
• Proximal-gradient and accelerated ISTA/FISTA variants for nonsmooth convex programs (Zhang et al., 2023).
• Alternating minimization for joint estimation of parameter and precision/task-structure matrices (Goncalves et al., 2014).
• Block-coordinate descent with ADMM/proximal subsolvers for convex clustering (Okazaki et al., 2023).
• Efficient Sinkhorn iterations for OT-based couplings (Janati et al., 2018).
• Primal-dual projection for large-task constrained SGD (Cervino et al., 2020). Convergence guarantees depend on convexity, regularity, and strong convexity of the underlying loss and penalty structure. Empirical and theoretical analyses confirm monotonic decrease of objectives, recovery of sparsity, and, in most cases, stationarity or global optimality when the problem is jointly convex.

7. Applications and Empirical Performance

Examined use cases include:

• Wind speed time series forecasting—MT-WRLS and MT-OSLSSVR achieve ≈16% reduction in relative RMSE compared to single-task RLS and outperform cubic-cost batch solvers (Lencione et al., 2023).
• School exam scores and Sarcos datasets—prior-encoded and smoothness-augmented models significantly improve explained variance and normalized MSE (Zhang et al., 2023).
• Climate model combination—task-structure learning yields block-diagonal dependency graphs and better RMSE than Laplacian smoothing (Goncalves et al., 2014).
• Remote-sensing-based trait prediction and GWAS—tree-structured MTL outperforms pre-grouped and no-group baselines, recovers meaningful hierarchy, and links genomic loci to structured outputs (Yu et al., 2017).
• High-dimensional neuroimaging—Wasserstein regularization adapts to anatomical geometry, yielding interpretable, spatially-coherent supports (Janati et al., 2018).
• Distributed student performance modeling—DAMTL converges efficiently on real district data, robust to asynchronous updates (Hong et al., 2024).
• In-context multi-task learning with transformers—multi-head architectures implement debiased gradient descent in superposition, generalizing empirically to longer contexts and overlapping task supports (He et al., 17 Mar 2025).

---

In summary, multi-task linear regression constitutes a mature and rapidly evolving class of models capable of exploiting diverse forms of inter-task dependency: explicit graphs, latent clusters, block-sparsity, feature geometry, tensor factorization, and network topology. These approaches offer provable statistical and computational efficiency, interpretable structural discovery, and flexibility in online, distributed, and high-dimensional regimes.