Transfert learning and adaptive LASSO quantile

Abstract: We propose for a quantile regression an estimation method for transferring knowledge using two $L_1$ penalties based on an estimator obtained from a source database. The proposed transfer learning estimator satisfies the properties of consistency and sparsity. Its convergence rate and asymptotic behavior are studied in several scenarios. This knowledge transfer results in a shorter computation time than that of the standard adaptive LASSO estimator. Another advantage of our method is that it can be applied to models with non-Gaussian errors. In addition, in order to implement the computing of the adaptive transfer LASSO quantile estimator, we propose an algorithm. The simulations confirm the theoretical results and demonstrate that the adaptive learning estimator, calculated using the proposed algorithm, is more competitive than the LASSO estimators. Finally, we illustrate the practical utility of the proposed transfer learning estimator and algorithm using a real-data application involving the physicochemical properties of protein tertiary structures.

• The paper introduces an adaptive transfer LASSO quantile estimator that leverages source data for enhanced high-dimensional regression.
• It establishes oracle properties with consistency, sparsity, and controlled asymptotic behavior across varied penalty regimes even for non-Gaussian errors.
• Numerical studies and a protein structure application show the method achieves lower bias, faster computation, and superior variable selection performance.

Transfer Learning and Adaptive LASSO for Quantile Regression

Introduction

This paper introduces an adaptive transfer learning estimator for quantile regression in high-dimensional linear models, with an emphasis on practical settings where the error distribution is non-Gaussian and the number of covariates may be large. The methodology leverages a source dataset to aid estimation on a smaller target dataset, utilizing two $L_1$-type penalties: a standard adaptive LASSO term and a transfer penalty encouraging proximity to the source estimator. The estimator is termed the adaptive transfer LASSO quantile (TransaLASSO) estimator.

Key theoretical contributions include rigorous proofs of consistency, variable selection (sparsity), establishment of convergence rates and asymptotic distributions across several scaling regimes, and characterizations of optimal tuning regimes for estimation. The utility and computational efficiency of the methodology is supported by numerical studies and an application to protein tertiary structure data.

Model Formulation and Estimation Procedure

Consider the classic high-dimensional linear model:

• Source model: $Y_i = X_i^\top \beta^* + \varepsilon_i$, $i=1,\ldots,m$
• Target model: $Y_i = X_i^\top \beta^* + \varepsilon_i$, $i=m+1,\ldots,m+n$

All covariate and response structures are assumed consistent between domains, but extensions for additional covariates in the source are discussed.

The estimation of $\beta^*$ proceeds in two steps:

1. Source Quantile Estimation: Compute the standard $\tau$-quantile estimator $\widetilde\beta_m$ on the source data.
2. Adaptive Transfer LASSO Quantile Estimator: On target data, estimate

$$\widehat\beta_{n} = \arg\min_\beta \left[ \sum_{i=m+1}^{m+n} \rho_\tau\left(Y_i - X_i^\top\beta\right) + \lambda_n \sum_{j=1}^p v_{m,j}|\beta_j| + \eta_n \sum_{j=1}^p \omega_{m,j} |\beta_j - \widetilde\beta_{m,j}| \right],$$

with adaptive weights $v_{m,j} = |\widetilde\beta_{m,j}|^{-\gamma_1}$ and $Y_i = X_i^\top \beta^* + \varepsilon_i$0 for fixed $Y_i = X_i^\top \beta^* + \varepsilon_i$1.

The penalty terms encourage automatic variable selection (sparsity) and proximity to the source estimator (knowledge transfer). Notably, the method is compatible with heavy-tailed and heteroskedastic non-Gaussian errors.

Theoretical Properties

Consistency, Sparsity, and Asymptotic Distribution

The estimator is shown to possess oracle properties under standard quantile regression assumptions:

• Consistency: $Y_i = X_i^\top \beta^* + \varepsilon_i$2 converges to $Y_i = X_i^\top \beta^* + \varepsilon_i$3 under suitable scaling.
• Sparsity: With high probability, the estimator recovers the true support; non-informative coefficients are set to exactly zero.
• Asymptotic Distribution: Depending on the asymptotics of $Y_i = X_i^\top \beta^* + \varepsilon_i$4, there are three regimes:

1. Source-dominated regime ($Y_i = X_i^\top \beta^* + \varepsilon_i$5): The estimator converges at rate $Y_i = X_i^\top \beta^* + \varepsilon_i$6; the limiting distribution for nonzero coefficients is identical to the source quantile estimator.
2. Balanced regime ($Y_i = X_i^\top \beta^* + \varepsilon_i$7): The estimator converges at rate $Y_i = X_i^\top \beta^* + \varepsilon_i$8, with a limiting distribution a function of both penalty sequences and source proximity.
3. Target-dominated/high-penalty regime: Slower rates and potential asymptotic bias may occur; not recommended for optimal inference.

Figure 1: Evolution with $Y_i = X_i^\top \beta^* + \varepsilon_i$9 of the estimations $i=1,\ldots,m$0 under different penalty/scaling regimes as compared to classical adaptive LASSO and adaptive LASSO on target only.

The dominance of the source sample improves estimation, particularly when the target data are scarce or noisy. The estimator is robust with respect to non-Gaussian, even asymmetric, error distributions.

Comparison with Classical Adaptive LASSO

A direct comparison with:

• (i) Adaptive LASSO quantile on pooled (source + target) data,
• (ii) Adaptive LASSO quantile on target only,

shows that TransaLASSO achieves comparable or lower bias—especially for small or near-zero coefficients and with heavy-tailed errors—while providing superior computational scalability in high dimensions.

Figure 2: Estimation error, false zero rate, and false nonzero rate comparisons of TransaLASSO vs. classical adaptive LASSO methods across varying $i=1,\ldots,m$1 and $i=1,\ldots,m$2.

Algorithmic Implementation

The paper provides a block coordinate descent algorithm based on subgradient KKT conditions, allowing efficient numerical solution for the TransaLASSO quantile estimator even in high $i=1,\ldots,m$3 settings. Importantly, empirical convergence is shown to be robust with respect to initialization.

Key algorithmic details:

• Closed-form updates or thresholding rules for each coordinate based on the magnitude of the adaptive penalties.
• Stopping criteria based on $i=1,\ldots,m$4-norm convergence, with typical $i=1,\ldots,m$5.
• Penalized quantile regression steps efficiently computed using standard tools (e.g., quantreg R package for the underlying quantile estimation subproblems).

The algorithm demonstrates rapidly decreasing runtime compared to standard adaptive LASSO quantile solvers, especially as $i=1,\ldots,m$6 increases into the hundreds or thousands.

Numerical Studies

Extensive Monte Carlo simulations are performed to validate theoretical results:

• Scenarios involving varying signal strengths, different error distributions (Gaussian, non-Gaussian, and uniform), and diverse $i=1,\ldots,m$7 sizes.
• Metrics include estimation relative error, false zero/nonzero rates, and computational time.

Results demonstrate that TransaLASSO:

• Retains oracle-like variable selection even when $i=1,\ldots,m$8 and the error distribution is highly non-normal.
• Achieves lower estimation error and better support recovery for weak signals than both pooled adaptive LASSO and target-only adaptive LASSO.

Figure 3: Histograms of estimates $i=1,\ldots,m$9 over replications, highlighting precise normality in the source-dominated regime.

Additionally, runtime improvements become substantial in high-dimensional cases, as confirmed in empirical comparisons.

Figure 4: Evolution with $Y_i = X_i^\top \beta^* + \varepsilon_i$0 of key performance metrics, confirming consistency and rapid convergence of $Y_i = X_i^\top \beta^* + \varepsilon_i$1 in different regimes.

Application: Physicochemical Properties of Protein Structures

The methodology is illustrated using a real dataset predicting residue size from various physicochemical properties of protein tertiary structures. The source dataset is large ($Y_i = X_i^\top \beta^* + \varepsilon_i$2), while the target set is small ($Y_i = X_i^\top \beta^* + \varepsilon_i$3), mirroring scenarios considered in simulation studies.

Key findings:

• Variable selection results are consistent with scientific expectations and results from aggregation-based procedures.
• Prediction accuracy (in terms of MAD) using TransaLASSO is comparable to adaptive LASSO using all data but achieved at dramatically reduced computation and parameter estimation time.

Implications and Future Directions

Practical implications:

• TransaLASSO offers estimation and variable selection in high-dimensional, non-Gaussian settings with computational efficiency tractable for massive datasets.
• Especially relevant for applications with abundant auxiliary (source) data and limited target data, such as federated learning, multi-center biomedical studies, and large scientific databases.

Theoretical extensions include:

• Potential generalization to transfer learning in non-linear models, generalized linear models, and other M-estimation frameworks using similar adaptive penalization and transfer penalties.
• Investigation into further adaptive penalty tuning guidelines (beyond the theoretical regimes established) for model misspecification and heterogeneity between source and target.

Conclusion

The adaptive transfer LASSO quantile approach provides an effective and computationally efficient means of performing transfer learning for high-dimensional quantile regression with non-Gaussian errors. This framework guarantees consistency, oracle variable selection, and rapid convergence rates by leveraging robust information from large source datasets. The associated algorithm is easily implemented and scalable. The approach demonstrates significant practical value across both synthetic studies and real-world scientific applications.

Figure 5: Evolution of estimation performance metrics and sparsity as a function of increasing sample size $Y_i = X_i^\top \beta^* + \varepsilon_i$4, highlighting the method's scalability and robustness.

Figure 6: Summary histograms for estimated coefficients, verifying asymptotic normality as predicted by theory.

Figure 7: Histograms of $Y_i = X_i^\top \beta^* + \varepsilon_i$5 under differing theoretical regimes, illustrating the impact of penalty configuration on bias and variance.