Correlated Product of Experts (CPoE)

• CPoE is a scalable Gaussian process regression framework that models correlations among local experts to balance computational complexity and posterior accuracy.
• It uses sparse precision structures and tunable parameters (C and γ) to interpolate between independent PoE, global sparse, and full GP models.
• Empirical results on synthetic and real-world data show that CPoE significantly reduces KL divergence and RMSE, offering improved uncertainty calibration.

The Correlated Product of Experts (CPoE) is a scalable framework for Gaussian process (GP) regression, designed to efficiently aggregate predictions from multiple local experts while explicitly modeling their correlations. By constructing a sparse precision structure over local inducing variables, CPoE interpolates between independent product-of-experts (PoE) models, global sparse GPs, and full GPs, allowing practitioners to control the trade-off between computational complexity and posterior accuracy. This framework accommodates general kernel functions and multi-dimensional inputs, and achieves linear scaling in both time and memory with respect to the total data size, offering practical scalability and improved uncertainty calibration for large datasets (Schürch et al., 2021).

1. Motivation and Context

Exact inference in standard GPs requires $\mathcal{O}(N^3)$ time and $\mathcal{O}(N^2)$ memory for $N$ training points, due to the cost of inverting the dense $N\times N$ kernel matrix. Two primary strategies have been adopted to address this bottleneck: global sparse methods, which introduce a set of $M\ll N$ inducing points and achieve $\mathcal{O}(NM^2)$ time and $\mathcal{O}(NM)$ memory, but incur $\mathcal{O}(M^3)$ per-iteration cost; and local PoE approaches, which partition the data into $J$ batches and train $J$ independent local GPs, achieving parallelizable computations with $\mathcal{O}(N^2)$0 time for $\mathcal{O}(N^2)$1 points per expert. However, traditional PoE methods ignore dependencies between local experts, leading to poor posterior uncertainty quantification, particularly in regions between partitions. CPoE addresses this deficiency by incorporating a tunable, sparse-correlation structure among experts, smoothly interpolating between PoE and global GP limits, and maintaining favorable linear scaling (Schürch et al., 2021).

2. Theoretical Formulation

Given $\mathcal{O}(N^2)$2 input-output pairs $\mathcal{O}(N^2)$3, the data are partitioned into $\mathcal{O}(N^2)$4 disjoint batches, each with $\mathcal{O}(N^2)$5 points. For expert $\mathcal{O}(N^2)$6, denote data as $\mathcal{O}(N^2)$7, local inducing inputs $\mathcal{O}(N^2)$8 ($\mathcal{O}(N^2)$9 with sparsity parameter $N$0), and latent function values $N$1, $N$2. The probabilistic model is constructed as follows:

• Expert Prior: $N$3, where $N$4 is block-diagonal over all local inducing sets.
• Conditional: $N$5, where $N$6, $N$7, and $N$8 denotes the local neighborhood of related experts.
• Observation Model: $N$9.

Correlations between local experts are encoded by a directed graphical model over the inducing variables $N\times N$0, specified by a predecessor set $N\times N$1 of at most $N\times N$2 earlier experts. This induces the prior

$N\times N$3

where $N\times N$4 and $N\times N$5. The global prior precision $N\times N$6 is banded with bandwidth approximately $N\times N$7 (Schürch et al., 2021).

3. Predictive Aggregation and Limiting Cases

The posterior on inducing variables is Gaussian:

$N\times N$8

where $N\times N$9, $M\ll N$0. For a new test input $M\ll N$1, each active expert $M\ll N$2 yields a local prediction $M\ll N$3, with $M\ll N$4, $M\ll N$5, and $M\ll N$6. The global prediction fuses experts $M\ll N$7 using a covariance-intersection:

$M\ll N$8

with normalized weights $M\ll N$9, $\mathcal{O}(NM^2)$0.

Three limiting cases are encompassed:

• $\mathcal{O}(NM^2)$1: Recovers block-diagonal independent PoE.
• $\mathcal{O}(NM^2)$2: Recovers global sparse GP (FITC) with $\mathcal{O}(NM^2)$3 inducing points.
• $\mathcal{O}(NM^2)$4: Reduces to the exact full GP (Schürch et al., 2021).

4. Training and Inference Algorithms

CPoE employs a marginal likelihood objective:

$\mathcal{O}(NM^2)$5

Gradients with respect to hyperparameters $\mathcal{O}(NM^2)$6 are calculated via matrix calculus, leveraging the sparsity of $\mathcal{O}(NM^2)$7 and using matrix identities (Woodbury, determinant lemma) to avoid dense matrix operations. Both batch (L-BFGS) and mini-batch (Adam) optimization are compatible. The algorithmic flow includes constructing predecessor sets, forming block-sparse matrices $\mathcal{O}(NM^2)$8 and $\mathcal{O}(NM^2)$9, assembling $\mathcal{O}(NM)$0, and computing posterior quantities via sparse Cholesky decomposition (Schürch et al., 2021).

5. Computational Complexity

With $\mathcal{O}(NM)$1 (batch size), $\mathcal{O}(NM)$2 (number of inducing variables per expert), and $\mathcal{O}(NM)$3, training cost is

• Time: $\mathcal{O}(NM)$4,
• Memory: $\mathcal{O}(NM)$5.

Prediction for $\mathcal{O}(NM)$6 test points costs $\mathcal{O}(NM)$7, reducing per-test cost with precomputed local quantities. For $\mathcal{O}(NM)$8, CPoE matches the complexity of global sparse GP with $\mathcal{O}(NM)$9 inducing variables but can use $\mathcal{O}(M^3)$0 overall (Schürch et al., 2021).

6. Empirical Performance and Interpretability

Empirical evaluation on synthetic and real-world benchmarks demonstrates that CPoE achieves improved calibration of predictive uncertainties and lower Kullback-Leibler (KL) divergences compared to independent PoE, global sparse GP, and other state-of-the-art methods. For example, with $\mathcal{O}(M^3)$1 synthetic data, CPoE$\mathcal{O}(M^3)$2 achieves KL $\mathcal{O}(M^3)$3 GPoE/BCM at comparable wall-clock time and produces well-calibrated $\mathcal{O}(M^3)$4 coverage. On UCI benchmarks ($\mathcal{O}(M^3)$5–$\mathcal{O}(M^3)$6), CPoE$\mathcal{O}(M^3)$7–$\mathcal{O}(M^3)$8 consistently offers lower KL and root mean squared error (RMSE) than alternatives for equal or reduced runtime. For instance, on the "concrete" dataset ($\mathcal{O}(M^3)$9), PoE(4) attains KL $J$0 at $J$1s, CPoE(4) achieves KL $J$2 at $J$3s, and the full GP KL $J$4 at $J$5s. Large-scale tests ($J$6–$J$7) demonstrate that CPoE with stochastic-Adam dominates SGP, GPoE, and GRBCM in CRPS versus time. Empirical results confirm that CPoE provides a tunable continuum between independent local and fully correlated global methods, with interpretable hyperparameters $J$8 (correlation order) and $J$9 (inducing sparsity), enabling effective trade-off control between computational budget and posterior quality (Schürch et al., 2021).