Transfer Learning for Moderate-Dimensional Ridge-Regularized Robust Linear Regression

Abstract: This paper studies transfer learning for ridge-regularized robust linear regression in the moderate-dimensional regime, where the number of predictors is of the same order as the sample size and the regression coefficients are not assumed to be sparse. We propose Trans-RR, which combines a robust ridge estimator from a source study with a robust ridge correction based on the target study. Under mild assumptions, we characterize the asymptotic estimation error of the proposed estimator and show that leveraging source data can substantially improve estimation accuracy relative to the traditional single-study ridge-regularized robust estimator. Simulation results and a real-data analysis support the theory and illustrate both positive and negative transfer as the discrepancy between the source and target studies varies.

• The paper proposes Trans-RR, a two-stage robust ridge-regularized estimator that combines source and target estimates.
• It characterizes the asymptotic ℓ2 error with deterministic risk equations influenced by predictor geometry and regularization.
• Extensive simulations and real-data analyses demonstrate improved performance when source and target models are similar, while cautioning against negative transfer.

Transfer Learning for Moderate-Dimensional Ridge-Regularized Robust Linear Regression

Problem Formulation and Methodological Contributions

This paper addresses transfer learning for linear regression in the moderate-dimensional regime, where the number of predictors $p$ is of the same order as the sample size $n$, and sparsity assumptions on the regression coefficients do not hold. Both the target and source domains may exhibit heavy-tailed errors, outlying observations, and heterogeneous predictor distributions, all of which challenge conventional transfer learning and robust regression approaches. The core methodological advance is the Trans-RR algorithm—a two-stage estimator that first fits a robust, ridge-regularized coefficient vector to the source data, then estimates a robust, ridge-regularized discrepancy vector on the target, and finally sums these components to obtain the target-domain estimator.

Unlike prior work focusing on either high-dimensional sparse settings or classical low-dimensional scenarios, the paper explicitly treats the moderate-dimensional non-sparse case. In this setting, existing lasso- or $\ell_1$-based transfer learning approaches (e.g., [li2022transfer]) are inappropriate since the regression vectors are dense. The estimator replaces quadratic loss with robust losses (e.g., smoothed Huber or Pseudo-Huber), yielding resilience to heavy-tailed noise. All regularization is ridge-type, motivated by the non-sparse nature of $\beta_0$ and $w_0$. Assumption frameworks are nonrestrictive: predictors are allowed to have generalized elliptical distributions (via scaling factors $\lambda_i$), and error distributions may be heavy-tailed, including Cauchy, with only unimodality, symmetry, and mild regularity (not even existence of moments) required.

Asymptotic Risk Analysis

The theoretical centerpiece is a characterization of the Trans-RR estimator’s asymptotic $\ell_2$ error as both $n, p \to \infty$ with $p/n \to \kappa \in (0, \infty)$. The main result (Theorem 1) demonstrates that the conditional estimation error converges in probability to a deterministic value $r_\rho(\kappa)$, whose computation depends on robust loss functions, the regularization parameter, the predictor distribution geometry (via $n$0), and the distance between the target and the estimated source coefficients.

The risk formula is given by a pair of equations involving the proximal operator of the robust loss, the regularization parameter, and the second-moment structure determined by the discrepancy $n$1 (with $n$2 denoting the robust ridge estimator from the source). In the special case of spherically symmetric predictors ($n$3), the equations are further reduced to scalar systems that are numerically tractable.

An important theoretical distinction highlighted by the analysis is the central role of the geometry of the predictor covariance structure (as parameterized by $n$4), in sharp contrast to the classical low-dimensional regime, where only the covariance matrix matters.

Figure 1: Boxplots of $n$5 over 1000 simulations, with red points indicating theoretical $n$6 values from Theorem 1.

The boxplots display empirical estimation error centered at the theoretically predicted values, with tight concentration as $n$7 increases, validating the asymptotic risk characterization.

Behavior of Estimation Error and Transfer Efficacy

The authors provide a detailed investigation of how the potential benefit or hazard of transfer depends on the source-target similarity quantified by $n$8 and the source estimation error. By numerically solving the limiting risk system while varying the discrepancy, they demonstrate that the theoretical risk is a monotonic function of $n$9. In multiple regimes considered (Gaussian, heavy-tailed, and heterogeneous mixtures), transfer is beneficial when the source and target regression vectors are sufficiently close, whereas significant discrepancy leads to negative transfer—i.e., worse performance than a single-study fit.

Figure 2: Theoretical curves of $\ell_1$0 versus $\ell_1$1 for five values of $\ell_1$2 under three structural regimes, as obtained by numerically solving the limiting equations.

These curves rigorously illustrate the thresholding phenomenon for positive/negative transfer and the dependence of estimation error on hyperparameters and model misspecification.

Comparative Empirical Evaluation

Extensive simulations corroborate the theoretical findings. The authors compare Trans-RR with single-study robust ridge, pooled-data robust ridge, single-task and transfer lasso methods under diverse data-generating processes: well-specified (Gaussian), heavy-tailed (Cauchy), and heterogeneous mixtures. Performance is summarized via relative $\ell_1$3 estimation error.

Figure 3: Boxplots (log scale) of relative estimation error for various $\ell_1$4 settings in cases I–III, showing consistent superiority of Trans-RR over Pooled-RR, and adaptive dominance over Single-RR in regimes where the source is informative.

Trans-RR uniformly outperforms naive pooling and lasso-based competitors. Crucially, it dominates single-study ridge when source-target discrepancy is small but yields to it when the discrepancy is large, precisely as the theory predicts. Notably, severe heavy-tailed noise or strong heterogeneity further accentuates these regime distinctions and the necessity for robust procedures.

Real-Data Analysis

The proposed methodology is further validated on a multi-site high-dimensional spectral regression dataset (the NIR IDRC “Shootout” benchmark), with $\ell_1$5 predictors after decorrelation. In both cross-instrument transfer directions, Trans-RR achieves the lowest root mean squared error and lowest variability, outperforming single-study, pooled, and lasso-based alternatives. The results emphasize the method’s effectiveness in real-world, nonsparse, moderate-dimensional applications exhibiting cross-domain heterogeneity.

Implications and Future Directions

This study establishes a new paradigm for robust transfer learning in moderate dimensions, where neither high-dimensional sparse regularization nor assumptions of low-dimensional asymptotics are defensible. Theoretical innovation is centered on non-asymptotic, deterministic risk equations that provide finite-sample performance guarantees. The explicit, interpretable dependence of transfer risk on source-target similarity, regularization level, predictor geometry, and robustness loss enables practitioners to anticipate both improvements and pitfalls when translating information across populations.

The demonstrated possibility of negative transfer underscores the necessity for adaptive diagnostics and data-driven selection mechanisms in practical deployments. Future directions include extension to multi-source transfer, development of optimal robust losses minimizing limiting risk, and generalization to non-identity predictor covariance structures, relaxing the isotropy assumption. The analytical techniques—particularly the use of proximal mappings, robust M-estimation machinery, and precise characterization of moderate-dimensional limits—provide a foundation for further advances in theoretical statistics for complex transfer learning environments.

Conclusion

The paper delivers a comprehensive methodological and theoretical framework for robust, ridge-regularized transfer learning in moderate-dimensional linear regression without sparsity or strong distributional assumptions. The asymptotic analysis, supported by simulation and application studies, reveals clear conditions for the utility of transferring information and quantifies the risk of negative transfer. These results have broad implications for the design, analysis, and theoretical understanding of transfer learning methods in modern statistical practice.