Consistency of informative source detection via cross-validation for Transfer MNI

Establish the consistency of the K-fold cross-validation–based informative source detection procedure used for the Informative-Weighted Transfer Minimum-ℓ2-Norm Interpolator (WTM) in high-dimensional linear regression with benign overfitting; specifically, prove that with high probability the estimated informative set \widehat{I} = { q ∈ [Q] : \widehat{L}(\hat{β}_{TM}^{(q)}) − \widehat{L}(\hat{β}_{M}^{(0)}) ≤ D^{(0)} } equals the oracle informative set I = { q ∈ [Q] : R(\hat{β}_{TM}^{(q)}) − R(\hat{β}_{M}^{(0)}) < 0 }, where \hat{β}_{TM}^{(q)} is the Transfer MNI (pre-trained on source q and fine-tuned on the target), \hat{β}_{M}^{(0)} is the target-only MNI, \widehat{L}(·) denotes K-fold cross-validation loss on the target, and R(·) denotes excess risk on the target distribution.

Background

The paper proposes a two-step Transfer MNI (pre-train on a source, then fine-tune on the target while staying close to the pre-trained model) and an ensemble procedure, Informative-Weighted Transfer MNI (WTM), which aggregates transfer models from sources detected as informative.

Informative sources are defined oracle-wise as those for which the Transfer MNI yields lower excess risk than the target-only MNI. Practically, the paper detects informative sources using a K-fold cross-validation rule comparing the cross-validated losses of Transfer MNI and target-only MNI with a data-driven threshold.

While cross-validation consistency has been analyzed for related transfer learning settings with explicit regularization (e.g., GLMs), the benign overfitting regime considered here relies on implicit regularization (minimum-ℓ2-norm interpolation). The authors highlight the need for theoretical guarantees that the proposed cross-validation detection recovers the oracle informative set with high probability.