Transfer learning for nonparametric Bayesian networks

Abstract: This paper introduces two transfer learning methodologies for estimating nonparametric Bayesian networks under scarce data. We propose two algorithms, a constraint-based structure learning method, called PC-stable-transfer learning (PCS-TL), and a score-based method, called hill climbing transfer learning (HC-TL). We also define particular metrics to tackle the negative transfer problem in each of them, a situation in which transfer learning has a negative impact on the model's performance. Then, for the parameters, we propose a log-linear pooling approach. For the evaluation, we learn kernel density estimation Bayesian networks, a type of nonparametric Bayesian network, and compare their transfer learning performance with the models alone. To do so, we sample data from small, medium and large-sized synthetic networks and datasets from the UCI Machine Learning repository. Then, we add noise and modifications to these datasets to test their ability to avoid negative transfer. To conclude, we perform a Friedman test with a Bergmann-Hommel post-hoc analysis to show statistical proof of the enhanced experimental behavior of our methods. Thus, PCS-TL and HC-TL demonstrate to be reliable algorithms for improving the learning performance of a nonparametric Bayesian network with scarce data, which in real industrial environments implies a reduction in the required time to deploy the network.

• The paper introduces PCS-TL and HC-TL methods that leverage auxiliary data to overcome sample scarcity in learning complex Bayesian network structures.
• It employs adaptive risk minimization with a target trust factor and Jensen-Shannon divergence to robustly mitigate negative transfer.
• Empirical evaluations on synthetic and UCI datasets demonstrate improved log-likelihood, structural accuracy, and sample efficiency over traditional methods.

Transfer Learning for Nonparametric Bayesian Networks: A Technical Summary

Introduction

The paper "Transfer learning for nonparametric Bayesian networks" (2604.01021) addresses the challenge of structure and parameter learning of nonparametric Bayesian networks (BNs) in regimes of data scarcity by introducing two unsupervised transfer learning algorithms. The first, PCS-TL, is a constraint-based method extending PC-stable, and the second, HC-TL, is a score-based extension of the standard hill-climbing (HC) structure learning algorithm. Both algorithms explicitly tackle negative transfer and enable structure and parameter induction in target domains where limited labeled data are available by leveraging auxiliary data from related domains. The authors' contributions include the design of robustness mechanisms based on Jensen-Shannon divergence, a target trust factor for controlling the influence of sources, and statistically rigorous benchmarking on synthetic and UCI datasets for kernel density estimation Bayesian networks (KDEBNs).

Background: Nonparametric Bayesian Networks

Nonparametric BNs extend the graphical model formalism to accommodate arbitrary multivariate continuous distributions without assuming Gaussianity or discretization, parameterizing conditional probability distributions (CPDs) using kernel density estimators (KDEs). Given a DAG $\mathcal{G}$, the joint PDF is decomposed as

$$f(x) = \prod_{i=1}^n f(x_i \mid x_{\mathrm{Pa}(i)}),$$

where each factor is represented as a CKDE. For structure learning, constraint-based approaches (e.g., PC-stable) rely on conditional independence (CI) tests, while score-based approaches (e.g., hill climbing) optimize a cross-validated log-likelihood. Data scarcity exacerbates structure learning since the sample complexity grows rapidly with the number of variables and parent sets.

Algorithms

PCS-TL: Constraint-Based Transfer

PCS-TL augments the independence test phase of PC-stable by performing a linear pool of RCoT-based CI test $p$-values across the target and $S$ auxiliary source datasets. The combination is regulated by the target trust factor $\eta$ and outlier detection over Jensen-Shannon (JS) divergences between marginal KDEs. Source contributions are ranked by local CI test consistency and SJS-based similarity weights.

Figure 1: Schematic for PCS-TL and HC-TL workflows, emphasizing shared and algorithm-specific transfer routines.

HC-TL: Score-Based Transfer

HC-TL introduces a cross-validated transfer log-likelihood score ($\mathcal{S}_{\mathrm{CVTL}^k}$), linearly pooling target and source parameter contributions, weighted by data-size-based $\eta$ and SJS-based similarity. Additionally, a risk metric $R$ compares the magnitude of the (folded) log-likelihoods, disabling transfer from sources if their contribution would result in negative transfer (i.e., when source ensemble likelihood is less than that of the target alone). The HC-TL procedure is thus more robust to irrelevant sources due to dynamic transfer gating.

Parameter Transfer

Parameter learning likewise exploits a log-linear pooling strategy, combining target and source CPDs as geometric means, with source weights inherited from SJS-based similarity and $\eta$ controlling the degree of reliance on transfer.

Mitigating Negative Transfer

A major technical contribution is the robust exclusion of sources deemed too dissimilar via the SJS metric aggregated over marginal variable distributions:

$$\mathrm{SJS}^s(X) = \sum_{X_i \in X} \mathrm{JS}(f^T_{KDE}(X_i) \| f^s_{KDE}(X_i))$$

and the function $$f(x) = \prod_{i=1}^n f(x_i \mid x_{\mathrm{Pa}(i)}),$$0 detects statistical outliers (using the $$f(x) = \prod_{i=1}^n f(x_i \mid x_{\mathrm{Pa}(i)}),$$1 test) for adaptive gating. This mechanism impacts both structure and parameter learning, sharply curtailing negative transfer from unrelated domains. The adaptive, sample-size-scaled target trust factor $$f(x) = \prod_{i=1}^n f(x_i \mid x_{\mathrm{Pa}(i)}),$$2 ensures graceful annealing from full transfer to single-domain learning as target data accumulates.

Experimental Evaluation

Synthetic Data

Experiments on mixed and Gaussian BNs of increasing size (from SPBNs to large "magic-irri") verified that both PCS-TL and HC-TL outperform their non-transfer counterparts in log-likelihood and structural accuracy (measured by the DHD metric) at low target sample counts. Notably, transfer learning provided substantial improvements for the smallest targets (25 samples) regardless of the level (up to 20%) of source-target arc modification and noise.

Figure 2: Synthetic SPBN graph structures used for controlled evaluation.

Figure 3: Sample efficiency gain—PCS-TL and HC-TL achieve lower DHD and higher test log-likelihood with only 25 target samples, for two auxiliary sources, compared to PC-stable and HC without transfer.

Figure 4: Effect of increased heterogeneity: performance with three sources, including up to 20% arc modifications, showing graceful performance degradation in PCS-TL and sustained robustness in HC-TL.

UCI Data

Across five real-world continuous UCI datasets, PCS-TL and HC-TL retained statistically significant performance advantages in the data-scarce regime, converging with or exceeding the baseline as more target data accrued.

Figure 5: Log-likelihood and DHD over target dataset size for multiple UCI domains, showing transfer advantage at low sample counts.

Statistical Testing

Friedman and Bergmann-Hommel post-hoc analysis established clear statistically significant gains for transfer algorithms at scarcity levels $$f(x) = \prod_{i=1}^n f(x_i \mid x_{\mathrm{Pa}(i)}),$$3. The negative effect of increasing source-target dissimilarity was more pronounced for PCS-TL compared to HC-TL, indicating the latter’s greater robustness due to explicit risk minimization.

Figure 6: Critical difference diagrams for DHD with two auxiliary sources, confirming significant separation of PCS-TL/HC-TL from non-transfer baselines.

Figure 7: Heatmap of pairwise $$f(x) = \prod_{i=1}^n f(x_i \mid x_{\mathrm{Pa}(i)}),$$4-values for DHD, with strong significance for transfer effect.

Implications and Future Directions

This work fills a substantial gap concerning unsupervised transfer learning methodologies for nonparametric Bayesian networks under continuous variables. The demonstrated reduction in sample requirements for structure and parameter identification directly impacts practical deployment in industrial or medical environments where acquisition of labeled instances is costly or slow. Furthermore, the transfer mechanisms are general and can potentially be integrated with other causal inference frameworks where nonparametric structure learning is bottlenecked by sample complexity.

The robustness under covariate shift and increasing source heterogeneity, especially afforded by risk-aware score integration (HC-TL), is a notable property and opens avenues for application in settings with diverse or incrementally changing ancillary datasets.

Conclusion

PCS-TL and HC-TL provide effective, robust unsupervised transfer learning for nonparametric BN structure and parameter learning under severe data scarcity. Empirical analysis on synthetic and real datasets, combined with rigorous statistical testing, confirms their superiority over standard approaches, particularly in the scarce-data regime and in the presence of moderately heterogeneous auxiliary domains. The mechanisms for negative transfer mitigation, especially sample-size adaptive gating and divergence-based similarity scoring, are essential for robust transfer. Future work may include extension to active transfer selection, deeper theoretical analysis of transfer adequacy, and deployment in real industrial process environments.