Transfer Learning in Regression with Influential Points

Abstract: Regression prediction plays a crucial role in practical applications and strongly relies on data annotation. However, due to prohibitive annotation costs or domain-specific constraints, labeled data in the target domain is often scarce, making transfer learning a critical solution by leveraging knowledge from resource-rich source domains. In the practical target scenario, although transfer learning has been widely applied, influential points can significantly distort parameter estimation for the target domain model. This issue is further compounded when influential points are also present in source domains, leading to aggravated performance degradation and posing critical robustness challenges for existing transfer learning frameworks. In this study, we innovatively introduce a transfer learning collaborative optimization (Trans-CO) framework for influential point detection and regression model fitting. Extensive simulation experiments demonstrate that the proposed Trans-CO algorithm outperforms competing methods in terms of model fitting performance and influential point identification accuracy. Furthermore, it achieves superior predictive accuracy on real-world datasets, providing a novel solution for transfer learning in regression with influential points

• The paper introduces Trans-CO, a novel framework that robustly integrates transfer learning with influential point detection in regression models.
• It employs a mean-shift model with a Θ-IPOD algorithm and a composite objective function to optimize performance, especially in high-dimensional settings.
• Experiments, including Beijing air quality prediction, demonstrate that Trans-CO achieves lower MSE and higher F1-scores than benchmarks, highlighting its practical utility.

Transfer Learning in Regression with Influential Points

The study investigates transfer learning in regression models affected by influential points, introducing a robust framework called Transfer Learning Collaborative Optimization (Trans-CO). The paper emphasizes the need for effective transfer learning approaches in scenarios where target domain data is limited and contaminated by influential points, presenting a novel method to tackle these challenges.

Methodology

Robust Regression Model for Influential Point Detection

The study addresses influential points by adopting a robust regression approach, where a mean-shift model is used to accommodate any observation as a potential influential point:

$$Y_i = \bm{X}_i\bm{\beta}+\gamma_i+\epsilon_i,\quad i=1,...,n,$$

where $\bm{\beta}$ is the coefficient vector, and $\gamma_i$ identifies influential points. The method incorporates a $\Theta$-IPOD algorithm, leveraging thresholding to handle non-convex penalties and ensure robust estimation.

Transfer Learning Collaborative Optimization Framework

The Trans-CO framework integrates knowledge from $K$ source datasets under linear approximation assumptions, formalized as:

$\bm{\beta} = \bm{B}\bm{w}+\bm{\delta},$

where $\bm{B}$ represents source model coefficients and $\bm{\delta}$ is a sparse vector indicating discrepancies. The method iteratively estimates model parameters by minimizing a composite objective function combining model fitting and detection of influential points.

Figure 1: Comparison of different methods for different sample size of target dataset and source datasets when $s = 25$ in Example \ref{ex1}.

Simulation Experiments

Impact of Sample Size and Dimensionality

Simulations evaluate the efficacy of Trans-CO against benchmarks: Profiled Transfer Learning (PTL) and $\Theta$-IPOD. Even in high-dimensional settings ($n<p$), Trans-CO demonstrates minimal Mean Squared Error (MSE) and superior F1-scores for influential point detection, asserting its robustness and adaptability.

Figure 2: Comparison of different methods for different sample size of target dataset and source datasets when $s = 75$ in Example \ref{ex1}.

Sensitivity to Influential Points

Trans-CO outperforms alternative methods in scenarios with varying influential point proportions, showcasing stability in log(MSE) and consistently achieving higher detection accuracy, proving effective under challenging conditions such as heteroscedasticity and when unique identification assumptions are unmet.

Real Data Application

Beijing Air Quality Prediction

Trans-CO's real-world applicability is demonstrated through Beijing Multi-Site Air Quality data, predicting CO levels using data from multiple sites as source domains. It achieves the lowest Huber Loss and highest R-squared values compared to PTL and $\Theta$-IPOD, underscoring its practical utility.

Figure 3: Comparison of the predicted Log($L_\alpha$) and R-squared on the test set of the target dataset.

Conclusion and Discussion

The study confirms that the Trans-CO framework significantly enhances model fitting by leveraging information from source domains. Despite added computational complexity, its accuracy improvements are substantial. Future work will aim to optimize runtime and explore broader applications.

In conclusion, Trans-CO provides an efficient solution for regression problems riddled with influential points and data scarcity, fostering innovation in fields like finance and healthcare where such challenges are prevalent.