Transfer Learning in High-dimensional Ising Models

Abstract: In high-dimensional Ising model estimation, target sample sizes are often limited, and effectively using auxiliary binary datasets of unknown relevance remains challenging. To address this, we propose Trans-Ising, a transfer learning method that combines a loss-based source screening rule with a two-stage estimation procedure. The method first identifies informative auxiliary sources using held-out target pseudolikelihood to prevent negative transfer. It then computes an initial estimator via pooled nodewise $\ell_1$-regularized logistic regression, followed by a target-only correction step using a folded-concave penalty. Theoretically, we establish fixed-node $\ell_2$ and $\ell_1$ error bounds, exact graph selection consistency, and the conditional consistency of the screening rule. Through extensive simulations and real-data analyses, we demonstrate that Trans-Ising achieves lower estimation errors than both target-only estimation and naive data pooling.

• The paper introduces Trans-Ising, a selective transfer learning method for joint parameter and graph structure estimation in high-dimensional Ising models.
• It employs a two-stage approach combining loss-based source screening with nodewise logistic regression and SCAD-penalized bias correction to reduce estimation errors.
• Empirical evaluations on synthetic, genomics, retail, and recommendation datasets confirm significant improvements over traditional Ising model estimation methods.

Transfer Learning in High-dimensional Ising Models

Overview

The paper "Transfer Learning in High-dimensional Ising Models" (2607.03005) introduces Trans-Ising, a selective transfer-learning framework for joint parameter and graph structure estimation in high-dimensional Ising models. The method addresses the challenge of leveraging auxiliary binary datasets of unknown relevance when the target sample size is limited and naive pooling can cause negative transfer due to cross-domain heterogeneity.

The core contributions include a two-stage estimator combining loss-based source selection with nodewise logistic regression and a folded-concave (SCAD) bias correction step, coupled with formal theoretical guarantees on estimation rates and exact graph recovery. Empirical results on synthetic and real-world cancer genomics, retail, and recommendation datasets corroborate the method's utility, especially in settings where traditional Ising neighborhood selection suffers from high variance and instability.

Problem Formulation and Challenges

High-dimensional Ising models are ubiquitous for inferring conditional dependence structures among binary variables across network domains such as genomics, psychometrics, and collaborative filtering. Neighborhood selection via nodewise $\ell_1$-regularized logistic regression remains the established approach for sparse structural estimation [ravikumar]. However, when the number of variables $p$ is large relative to the sample size $n_0$, even regularized estimators exhibit high error rates.

Transfer learning addresses this by leveraging $S$ auxiliary datasets, each potentially governed by slightly different interaction structures. However, unknown domain shift (quantified by the cross-domain discrepancy $h_j$) means that naive data pooling can be detrimental, amplifying both false positives and negatives. The key is to selectively identify compatible sources and statistically adjust for residual distributional heterogeneity during inference.

The Trans-Ising Algorithm

Trans-Ising consists of two sequential components: (1) a loss-based source screening stage and (2) a two-step estimation/correction procedure. The major technical advances are the rigorous detection of beneficial sources for variance reduction, and the integration of SCAD-penalized bias correction, enabling support recovery under weaker design conditions compared to lasso.

1. Loss-based Screening for Informative Source Detection

Given target data $X^{(0)}$ and $S$ auxiliary datasets $\{X^{(s)}\}_{s=1}^S$, the method computes cross-validated pseudolikelihoods on held-out target folds to empirically quantify, for each candidate source, the change in target validation loss when including that source. Sources are retained only if their inclusion does not increase the loss beyond a tuned statistical threshold. This screening rule is proven to consistently recover the population-optimal informative set under high-probability separation and validation stability (Theorem~\ref{thm:detection_consistency}).

2. Two-stage Estimation: Pooling and Bias Correction

The algorithm then pools the target data with the selected informative auxiliary datasets and solves a nodewise $\ell_1$-penalized logistic regression to produce a preliminary estimate. Due to potential residual bias from source-to-target drift, a second bias-correction is applied using only target data: each nodewise parameter vector is updated via SCAD-penalized optimization over a sparse correction variable, encouraging sparse adjustment while reducing bias on strong edges.

The folded-concave (SCAD) penalty is essential for support recovery, as it vanishes for coefficients above the SCAD threshold $a\lambda$, avoiding the irrepresentable condition inherent to lasso for variable selection. Optimization is conducted via the Local Linear Approximation (LLA), with KKT-constrained local solutions.

Theoretical Guarantees

The main theoretical result (Theorem~\ref{thm:transfer_rewrite}) establishes that, for a fixed node, the overall estimation error $p$0 decomposes into:

• a pooled variance term $p$1, with $p$2 the cumulative size over target and selected auxiliaries,
• heterogeneity terms indexed by the source-to-target deviation $p$3.

Thus, as long as $p$4 is dominated by the variance term and $p$5, significant error reduction is achievable. The second stage SCAD penalty enables neighborhood and support recovery under beta-min and separation, without irrepresentable constraints (Theorem~\ref{thm:selection_rewrite}). All rates are non-asymptotic, implying strong guarantees even at practical finite sample sizes.

Notably, the error rates show that:

• If all sources are homogeneous ($p$6 small), Trans-Ising achieves the oracle pooled rate.
• If sources are heterogeneous (large $p$7), source screening prevents degradation versus target-only learning.

Empirical Evaluation

Simulations and real-data analyses demonstrate Trans-Ising's statistical and predictive efficacy.

Synthetic Study

Figure 1: Averaged PR curves for Naive-LogLasso (green), Oracle Trans-Ising (black), Pooled-Trans-Ising (red), and Trans-Ising (cyan, dashed) across graph structures and perturbations.

In synthetic experiments with varying numbers of informative and non-informative sources, Trans-Ising consistently tracks the performance of the oracle (true source) selection, outperforming both naive lasso and unscreened pooling, especially at low to moderate heterogeneity. Negative transfer is mitigated by the screening rule.

Cancer Genomics: DepMap Mutation Data

Figure 2: Relative misclassification error rates for Pooled-Trans-Ising (red) and Trans-Ising (cyan) across target cancers, relative to the Naive-LogLasso baseline.

Trans-Ising consistently reduces held-out misclassification error across multiple cancer types relative to both target-only and naive pooling baselines. The source selection algorithm adaptively detects compatible cancers for effective transfer, with the number and identity of selected sources strongly variable across target types.

Exploratory Data Analysis

Figure 3: Mutation frequencies for the most mutated genes highlight heterogeneity across cancer types, motivating selective transfer strategies.

Additional Domains: Retail and MovieLens Data

Figure 4: Relative misclassification error rates for Pooled-Trans-Ising and Trans-Ising on the Online Retail dataset, relative to Naive-LogLasso.

Across retail and collaborative filtering applications, Trans-Ising maintains performance improvements or parity with naive pooling while offering statistical protection when cross-domain distribution shifts are non-negligible.

Graphical Structure Visualization

Figure 5: Underlying synthetic network structures for $p$8 used in the simulation studies.

Practical and Theoretical Implications

Trans-Ising provides a statistically principled and computationally tractable framework for transfer learning in high-dimensional discrete graphical models:

• The separation of variance reduction and transfer risk quantified by $p$9 directly extends minimax optimality insights from Gaussian graphical and linear models to non-Gaussian exponential family settings, with practical formulations for Ising models.
• The combination of loss-based screening and folded-concave post-processing represents a robust platform applicable to broader domains where the challenge of negative transfer is acute.
• Support recovery without the irrepresentable condition, made possible by SCAD regularization, suggests future directions for selective variable selection in even more ill-posed settings.
• The screening algorithm is, however, computationally intensive for large $n_0$0; future developments might leverage scalable meta-learning or structure-aware surrogates.
• Extensions to structured regularization (e.g., group lasso) and continuous graphical models under similar transfer paradigms appear promising.

Conclusion

Trans-Ising establishes a selective, theoretically justified framework for transfer learning in high-dimensional Ising models. By addressing both effective auxiliary selection and bias correction, it robustly improves parameter estimation and graph recovery over standard baselines without incurring negative transfer, as substantiated by both theoretical and empirical analysis. The dual-stage construct and screening paradigm have implications for discrete transfer-learning tasks in modern multi-domain applications.