Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Abstract: This paper develops a general transfer learning framework for nonparametric regression with data consisting of multiple groups. Under the assumption that groups share a common structure along with group-specific deviations in additive form, the proposed method employs a two-stage offset learning procedure: the first stage pools data from all groups to estimate an overall mean function, and the second stage estimates offsets for each group, yielding final group-level estimators through additive combination. Upper bounds on the $\mathcal L_2$ error are established for the proposed framework, covering a broad class of nonparametric estimators under mild complexity and noise conditions. When instantiated with deep ReLU networks, explicit convergence rates are derived under hierarchical composition models, demonstrating the ability to overcome the curse of dimensionality. Conditions that enable positive transfer with faster rates are considered, including learning with simpler functions and data augmentation through pooling samples across groups. Various simulations and real-data experiments further validate the effectiveness of the proposed method.

• The paper develops a two-stage offset-based method that pools groups to estimate shared structure, then learns group-specific corrections with deep ReLU networks under an overlap condition.
• The theory achieves rates driven by intrinsic dimension and smoothness rather than ambient dimension, supports growing group counts and vanishing target-group proportions, and identifies when transfer outperforms separate estimation.
• Experiments show consistent gains across simulations and real datasets, including minority-group improvements and lower MSE on Beijing PM2.5 and UTKFace, while highlighting open issues such as negative transfer and non-additive task structures.

Overview and problem setting

This paper develops a two-stage offset-based transfer learning framework for nonparametric regression with grouped data, with deep ReLU networks as the primary estimator class. The data consist of $n$ i.i.d. copies of $(X,Y,Z)$, where $Z \in \{1,\ldots,L\}$ indexes groups, generated from the additive model $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$, where $f_0$ is shared across groups and $f_{0,\ell}$ is a group-specific deviation. The target is the group-conditional mean $g_{0,\ell}(x) = f_0(x) + f_{0,\ell}(x)$.

The method proceeds in two stages. First, all $L$ groups are pooled to estimate the overall mean $\bar f(x) = \mathbb{E}(Y \mid X=x)$ by empirical risk minimization over a generic function class $\mathcal{F}$. Second, for each group $(X,Y,Z)$0, an offset function $(X,Y,Z)$1 is estimated on group-$(X,Y,Z)$2 data alone, regressing residuals against $(X,Y,Z)$3. The final estimator is the additive combination $(X,Y,Z)$4, with truncation at levels $(X,Y,Z)$5 and $(X,Y,Z)$6.

The framework requires only an overlap-type assumption: group membership probabilities bounded away from 0 and 1 given covariates, analogous to overlap conditions in causal inference. Notably, the theory accommodates $(X,Y,Z)$7 growing with $(X,Y,Z)$8 and allows the target-group proportion $(X,Y,Z)$9 to vanish, covering both transfer learning (data-scarce target) and multi-task learning regimes within one analysis.

General theoretical guarantees

The main general result establishes that, under mild metric entropy conditions on the truncated classes $Z \in \{1,\ldots,L\}$0 and $Z \in \{1,\ldots,L\}$1 and a noise condition $Z \in \{1,\ldots,L\}$2 (which permits sub-Gaussian and sub-exponential errors), the squared $Z \in \{1,\ldots,L\}$3 error satisfies

$Z \in \{1,\ldots,L\}$4

where $Z \in \{1,\ldots,L\}$5 is the first-stage rate for estimating $Z \in \{1,\ldots,L\}$6, $Z \in \{1,\ldots,L\}$7 is the approximation error of the offset class, and $Z \in \{1,\ldots,L\}$8 is the localized entropy majorant. The result holds provided $Z \in \{1,\ldots,L\}$9, i.e., the expected group size diverges. The proof controls the second-stage regression against perturbed responses via empirical process arguments over localized entropy balls, combined with binomial concentration on group sizes.

A complementary result under sample splitting (independent stages) replaces one complexity constraint with an additive term $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$0, which can yield tighter rates; in particular it removes the factor $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$1 multiplying the first-stage error in the network instantiation. The authors conjecture that this factor in the non-splitting bound is a proof artifact, since empirically the estimator performs better without splitting — an honest concession worth noting when interpreting the rates.

As corollaries of generality, the paper derives what appears to be the first convergence guarantee for trend filtering in a transfer learning setting, with rate $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$2 under total variation penalties, and recovers up to logarithmic factors the Sobolev-sieve transfer rates of prior work on orthogonal series regression, unifying offset-based transfer learning across estimator classes.

Deep ReLU networks and positive transfer

Instantiating both stages with dense ReLU networks under hierarchical composition models (each component $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$3-smooth with intrinsic dimension $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$4, uniformly bounded), the overall rate becomes, ignoring logarithmic factors,

$y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$5

with network depth and width chosen as functions of $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$6 and $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$7. Two consequences follow directly:

Curse-of-dimensionality mitigation is preserved under transfer. Because the rates depend only on the intrinsic dimensions and smoothness of the compositional structure rather than the ambient dimension $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$8, the benefit of deep networks carries over to the two-stage procedure. An illustrative example with $y_i = f_0(x_i) + f_{0,z_i}(x_i) + \epsilon_i$9 shows the NN-with-TL rate remaining at $f_0$0 even as the group proportion shrinks to $f_0$1, whereas classical TL estimators degrade to $f_0$2; in the extreme regime NN-TL attains $f_0$3 versus $f_0$4 for classical methods.

Conditions for strictly faster rates than separate estimation. Transfer helps whenever $f_0$5 and $f_0$6 are less complex than $f_0$7 itself — e.g., when averaging similar groups cancels extreme components, or when the offset has higher smoothness and lower intrinsic dimension than the full conditional mean. A concrete example gives $f_0$8 with effective dimension 2 and $f_0$9 depending on only 3 variables while each $f_{0,\ell}$0 depends on all 5.

The authors explicitly state they do not claim optimality, noting that minimax lower bounds for hierarchical composition classes are known only in special cases — so these are upper bounds, not established minimax rates.

Experiments

Simulations span four low-dimensional scenarios ($f_{0,\ell}$1, unbalanced groups, SNR ∈ {2,5,10}) and two high-dimensional latent-factor scenarios ($f_{0,\ell}$2, $f_{0,\ell}$3–$f_{0,\ell}$4), comparing the proposed 2-Stage strategy against Pooled, Separate, Pool-w-L (labels as inputs), and Top-FT (output-layer fine-tuning), using both MLPs and random forests, plus ptLasso in high dimensions. Across 50 Monte Carlo replications, 2-Stage (NN) achieves the lowest MSE in essentially every configuration; e.g., in Scenario 1 at $f_{0,\ell}$5, MSE is 0.27×10⁻² versus 1.93×10⁻² for Pooled and 0.49×10⁻² for Separate. In highly unbalanced settings (one group at 2% prevalence), 2-Stage improves the minority group's MSE monotonically with $f_{0,\ell}$6 while Separate plateaus near 0.02, consistent with the theoretical prediction that pooling benefits small-$f_{0,\ell}$7 groups.

Real-data experiments use the Beijing PM2.5 dataset (grouped by wind direction, autoregressive specification) and UTKFace age estimation (grouped by ethnicity, FaceNet features). On PM2.5, 2-Stage attains the lowest overall test MSE for both NN (0.0519 vs. 0.0555 Pooled) and RF (0.0534 vs. 0.0556), supporting the claim that the framework transfers across estimator classes. On UTKFace, 2-Stage achieves the lowest overall MSE (59.6 vs. 61.8 Pooled, 63.1 Separate), winning on White, Black, and Asian subgroups; complementary experiments with ethnicity-plus-gender grouping show uniform improvements over pooled training.

Limitations and open questions

Several limitations are stated plainly. The additive decomposition $f_{0,\ell}$8 is assumed formally, though simulations without exact additivity still perform well; extending to other coupling structures remains open. Negative transfer is theoretically possible when $f_{0,\ell}$9 or $g_{0,\ell}(x) = f_0(x) + f_{0,\ell}(x)$0 is complex relative to $g_{0,\ell}(x) = f_0(x) + f_{0,\ell}(x)$1, and formal detection of negative transfer is left unresolved. The gap between the sample-splitting and non-splitting bounds (the $g_{0,\ell}(x) = f_0(x) + f_{0,\ell}(x)$2 factor) is conjectured but not proven to be a proof artifact. Finally, the theory covers fully connected ReLU MLPs; analogous guarantees for last-layer fine-tuning, convolutional architectures, Transformers, or continuously varying task parameters are not established here.

Conclusion

The paper provides a general, estimator-agnostic convergence theory for two-stage offset transfer learning in nonparametric regression, instantiated with dense ReLU networks under hierarchical composition to obtain curse-of-dimensionality-free rates of the form $g_{0,\ell}(x) = f_0(x) + f_{0,\ell}(x)$3. It identifies concrete conditions — smoother offsets, simpler pooled means, vanishing target-group proportions — under which transfer strictly dominates separate estimation, and its simulations and real-data results consistently support the theory. Its principal open problems concern lower bounds for compositional classes, formal negative-transfer detection, and extensions beyond the additive decomposition.