Product-of-Experts Interpolation

• Product-of-Experts interpolation is a probabilistic framework that aggregates local expert outputs using weighted products to yield scalable and well-calibrated predictions.
• It leverages the log opinion pool to optimally combine Gaussian process outputs by minimizing a weighted sum of KL divergences, resulting in closed-form Gaussian predictions.
• The method supports heterogeneous expert configurations with map–reduce pipelines and explicit diversity promotion, enhancing performance metrics like NLL and MSE.

Product-of-experts (PoE) style interpolation denotes a class of probabilistic model aggregation techniques in which the predictions of multiple local or specialized models (“experts”) are combined via a (weighted) product of their output densities, rather than by a mixture. In the context of Gaussian process (GP) regression and scalable machine learning, PoE aggregation provides a theoretically grounded, computationally tractable, and highly flexible means of constructing global predictions from local models—crucial for scalability, heterogeneity, and uncertainty quantification (Cao et al., 2015, Schürch et al., 2021).

1. Mathematical Foundation: Log Opinion Pool and PoE Aggregation

The theoretical underpinning of PoE-style interpolation is the log opinion pool, which provides the solution to optimal aggregation of expert distributions under a weighted sum of Kullback–Leibler (KL) divergences. Given $K$ experts providing posterior distributions $p_i(f_*|x_*,D_i)$ at test input $x_*$, and non-negative weights $\alpha_i(x_*)$ satisfying $\sum_{i=1}^K \alpha_i = 1$, the optimally pooled distribution is obtained by minimizing

$\sum_{i=1}^K \alpha_i\,\mathrm{KL}(p \| p_i),$

leading to the log opinion pool:

$$\widetilde{p}(f_*|x_*) = \frac{1}{Z(x_*)}\prod_{i=1}^K p_i(f_*|x_*,D_i)^{\alpha_i(x_*)}, \quad Z(x_*) = \int \prod_{i=1}^K p_i(f_*|x_*,D_i)^{\alpha_i(x_*)} df_*.$$

When all $p_i$ are univariate Gaussians, this yields a closed-form Gaussian for the aggregate, where the combined mean and variance are weighted by both the experts' precision and the pooling weights (Cao et al., 2015).

2. Generalized and Correlated Product-of-Experts Models

The generalized product-of-experts Gaussian process (gPoE-GP) model extends PoE by incorporating data-dependent weights via heuristics such as entropy change:

$$\beta_i(x_*) \propto \Delta H_i(x_*),\qquad \Delta H_i(x_*) = \frac{1}{2}\log\frac{k_i(x_*,x_*)}{\sigma_i^2(x_*)}.$$

This heuristic quantifies how much the data in $D_i$ have influenced expert $p_i(f_*|x_*,D_i)$0 at $p_i(f_*|x_*,D_i)$1.

Further, the Correlated Product-of-Experts (CPoE) model interpolates between fully independent experts and a single global GP by explicitly modeling local correlations between groups of experts. Each expert operates not in isolation but within a “region of correlation” defined by a graph structure, and their predictions are aggregated by a PoE with appropriately chosen weights. The approach admits limiting cases:

• $p_i(f_*|x_*,D_i)$2: All experts independent (gPoE).
• $p_i(f_*|x_*,D_i)$3: Fully correlated, recovering global GP or sparse approximations such as FITC. CPoE enables scalable GP inference at linear cost in $p_i(f_*|x_*,D_i)$4 and provides honest uncertainty quantification (well-calibrated confidence intervals) (Schürch et al., 2021).

3. Theoretical Properties and Justification

The log opinion pool aggregation is a strict minimizer of the weighted sum of KL divergences, requiring neither conditional independence nor shared priors. An approximate expansion of the KL divergence to the unknown ground-truth distribution $p_i(f_*|x_*,D_i)$5,

$p_i(f_*|x_*,D_i)$6

with

$p_i(f_*|x_*,D_i)$7

decomposes global error into bias (E) and diversity (C) terms. Optimal weight selection aims to decrease $p_i(f_*|x_*,D_i)$8 while increasing $p_i(f_*|x_*,D_i)$9, the latter promoting expert heterogeneity and boosting calibration (Cao et al., 2015).

In dLOP-GP (diversified log opinion pool), diversity is further promoted by an explicit normalized gradient step on $x_*$0:

$x_*$1

followed by normalization. Empirical results confirm improvements in both NLL and MSE metrics over standard gPoE and robust Bayesian committee machines.

4. Algorithmic Mechanisms and Practical Pipeline

The PoE interpolation workflow for GP regression follows the “map–reduce” paradigm:

• The dataset is partitioned among $x_*$2 experts, each expert trained (optionally on disjoint or overlapping data) and producing local predictive mean and variance.
• Experts' outputs at a test location are combined via weighted products, with weights selected according to entropy-change, learned coefficients, or (in CPoE) using a correlation structure.
• In CPoE, the degree of correlation ($x_*$3), locality ($x_*$4), and sparsity ($x_*$5) are key tunable parameters. Practitioners use KD-tree or clustering for data partitioning and select graph neighborhoods for correlation.
• The final PoE predictive is Gaussian, with mean and (inverse) variance given by:

$x_*$6

Efficient implementation exploits block sparsity for training and testing, allowing essentially linear scaling in data size for fixed local block and correlation sizes.

5. Empirical Behavior, Calibration, and Scalability

Product-of-experts interpolation, specifically via gPoE-GP, dLOP-GP, and CPoE, achieves scalable, accurate, and well-calibrated regression on both synthetic and benchmark datasets. Empirical findings include:

• dLOP-GP delivers up to 10–20% improvement in standardized NLL and MSE over alternatives on tasks such as KIN40K, SARCOS, UK-APT.
• CPoE rapidly converges to full-GP accuracy and log likelihood as correlation $x_*$7 increases, without degradation in confidence interval validity.
• Both frameworks admit heterogeneous expert configurations (distinct data, kernels, and hyperparameters).
• Map–reduce style parallelization leads to high throughput, with dLOP overhead at test time only 5–10%.

The infrastructure is robust to local expert heterogeneity, and explicit diversity encouragement (as measured by the $x_*$8 term) yields further calibration gains. This suggests PoE-style interpolation is particularly effective for large, multimodal, and distributed regression problems (Cao et al., 2015, Schürch et al., 2021).

6. Limiting Cases, Flexibility, and Connections

Product-of-experts style interpolation frameworks interpolate smoothly between extremes of model combination:

• Fully independent PoEs: corresponds to naive locality.
• Fully correlated, dense GP: exact global solution.
• Sparse global GP (e.g., FITC) as intermediate cases.

The approach also encompasses traditional PoE, Bayesian Committee Machine, and robust combinations as specializations of the general log opinion pool framework (Cao et al., 2015, Schürch et al., 2021).

In summary, PoE-style interpolation, grounded in the log opinion pool, offers a scalable, theoretically justified, and empirically robust framework for combining local models in Gaussian process regression and related probabilistic modeling tasks. It provides a principled axis—locality, correlation, sparsity, heterogeneity—along which practitioners may interpolate to meet accuracy, calibration, and scalability requirements.