---
title: Correlated Product of Experts (CPoE)
url: https://www.emergentmind.com/topics/correlated-product-of-experts-cpoe
type: topic
---

# Correlated Product of Experts (CPoE)

The Correlated Product of Experts (CPoE) is a scalable framework for Gaussian process (GP) regression, designed to efficiently aggregate predictions from multiple local experts while explicitly modeling their correlations. By constructing a sparse precision structure over local inducing variables, CPoE interpolates between independent product-of-experts (PoE) models, global sparse GPs, and full GPs, allowing practitioners to control the trade-off between computational complexity and posterior accuracy. This framework accommodates general kernel functions and multi-dimensional inputs, and achieves linear scaling in both time and memory with respect to the total data size, offering practical scalability and improved uncertainty calibration for large datasets [2112.09519].

## 1. Motivation and Context

Exact inference in standard GPs requires $\mathcal{O}(N^3)$ time and $\mathcal{O}(N^2)$ memory for $N$ training points, due to the cost of inverting the dense $N\times N$ kernel matrix. Two primary strategies have been adopted to address this bottleneck: global sparse methods, which introduce a set of $M\ll N$ inducing points and achieve $\mathcal{O}(NM^2)$ time and $\mathcal{O}(NM)$ memory, but incur $\mathcal{O}(M^3)$ per-iteration cost; and local PoE approaches, which partition the data into $J$ batches and train $J$ independent local GPs, achieving parallelizable computations with $\mathcal{O}(N B^2)$ time for $B=N/J$ points per expert. However, traditional PoE methods ignore dependencies between local experts, leading to poor posterior uncertainty quantification, particularly in regions between partitions. CPoE addresses this deficiency by incorporating a tunable, sparse-correlation structure among experts, smoothly interpolating between PoE and global GP limits, and maintaining favorable linear scaling [2112.09519].

## 2. Theoretical Formulation

Given $N$ input-output pairs $\{(x_i, y_i)\}$, the data are partitioned into $J$ disjoint batches, each with $B = N/J$ points. For expert $j$, denote data as $X_j\in\mathbb{R}^{B\times D}$, local inducing inputs $A_j\in\mathbb{R}^{L\times D}$ ($L = \lfloor \gamma B\rfloor$ with sparsity parameter $\gamma\in(0,1]$), and latent function values $f_j = f(X_j)$, $a_j = f(A_j)$. The probabilistic model is constructed as follows:

- **Expert Prior**: $p(a|\theta) \propto \mathcal{N}(a|0, K_{AA}(\theta))$, where $K_{AA}$ is block-diagonal over all local inducing sets.
- **Conditional**: $p(f|a, \theta) = \prod_{j=1}^J \mathcal{N}(f_j|H_j a_{\psi(j)}, \bar{V}_j)$, where $H_j = K_{X_j A_{\psi(j)}}K_{A_{\psi(j)}A_{\psi(j)}}^{-1}$, $\bar{V}_j = K_{X_j X_j} - K_{X_jA_{\psi(j)}}K_{A_{\psi(j)}A_{\psi(j)}}^{-1}K_{A_{\psi(j)}X_j}$, and $\psi(j)$ denotes the local neighborhood of related experts.
- **Observation Model**: $p(y|f) = \prod_{j=1}^J \mathcal{N}(y_j|f_j,\sigma_n^2 I)$.

Correlations between local experts are encoded by a directed graphical model over the inducing variables $a_j$, specified by a predecessor set $\pi_C(j)$ of at most $C-1$ earlier experts. This induces the prior
$$
q(a) = \prod_{j=1}^J p(a_j|a_{\pi_C(j)}) = \prod_{j=1}^J \mathcal{N}(a_j|F_j a_{\pi_C(j)}, Q_j),
$$
where $F_j=K_{A_j A_{\pi_C(j)}}K_{A_{\pi_C(j)}A_{\pi_C(j)}}^{-1}$ and $Q_j=K_{A_j A_j} - K_{A_j A_{\pi_C(j)}}K_{A_{\pi_C(j)}A_{\pi_C(j)}}^{-1} K_{A_{\pi_C(j)}A_j}$. The global prior precision $S=F^T Q^{-1} F$ is banded with bandwidth approximately $C$ [2112.09519].

## 3. Predictive Aggregation and Limiting Cases

The posterior on inducing variables is Gaussian:
$$
q(a|y) = \mathcal{N}(a|\mu_a, \Sigma_a),\qquad \Sigma_a^{-1} = S + H^T V^{-1} H,
$$
where $V = \bar V + \sigma_n^2 I$, $\mu_a = \Sigma_a H^T V^{-1} y$. For a new test input $x_*$, each active expert $j$ yields a local prediction $q_j(f_*|y) = \mathcal{N}(m_{*j}, v_{*j})$, with $m_{*j}=h_{*j}^T \mu_{\psi(j)}$, $v_{*j}=h_{*j}^T \Sigma_{\psi(j),\psi(j)} h_{*j}+\cdots$, and $h_{*j}=K_{x_* A_{\psi(j)}}K_{A_{\psi(j)}A_{\psi(j)}}^{-1}$. The global prediction fuses experts $j=C,\ldots, J$ using a covariance-intersection:
$$
q(f_*|y) = \prod_{j=C}^J q_j(f_*|y)^{\beta_{*j}},
$$
with normalized weights $\beta_{*j}\propto \frac{1}{2}\log\left(\frac{v_{*0}}{v_{*j}}\right)$, $v_{*0} = k(x_*, x_*)$.

Three limiting cases are encompassed:
- $C=1, \gamma=1$: Recovers block-diagonal independent PoE.
- $C=J, \gamma<1$: Recovers global sparse GP (FITC) with $M=\gamma N$ inducing points.
- $C=J, \gamma=1$: Reduces to the exact full GP [2112.09519].

## 4. Training and Inference Algorithms

CPoE employs a marginal likelihood objective:
$$
\ell(\theta)=\log q(y;\theta) = -\frac{1}{2}\left[y^T P^{-1} y + \log|P| + N\log 2\pi\right], \qquad P = H S^{-1} H^T + V.
$$
Gradients with respect to hyperparameters $\theta$ are calculated via matrix calculus, leveraging the sparsity of $S$ and using matrix identities (Woodbury, determinant lemma) to avoid dense matrix operations. Both batch (L-BFGS) and mini-batch (Adam) optimization are compatible. The algorithmic flow includes constructing predecessor sets, forming block-sparse matrices $F$ and $Q$, assembling $S$, and computing posterior quantities via sparse Cholesky decomposition [2112.09519].

## 5. Computational Complexity

With $B=N/J$ (batch size), $L=\gamma B$ (number of inducing variables per expert), and $\alpha=C\gamma$, training cost is
- Time: $\mathcal{O}(J(LC)^3)=\mathcal{O}(N B^2 \alpha^3)$,
- Memory: $\mathcal{O}(J(LC)^2)=\mathcal{O}(N B \alpha^2)$.

Prediction for $N_t$ test points costs $\mathcal{O}(N B^2 \alpha^3 + N B \alpha^2 N_t)$, reducing per-test cost with precomputed local quantities. For $\alpha=1$, CPoE matches the complexity of global sparse GP with $M=B$ inducing variables but can use $M=\gamma N$ overall [2112.09519].

## 6. Empirical Performance and Interpretability

Empirical evaluation on synthetic and real-world benchmarks demonstrates that CPoE achieves improved calibration of predictive uncertainties and lower Kullback-Leibler (KL) divergences compared to independent PoE, global sparse GP, and other state-of-the-art methods. For example, with $N=8192$ synthetic data, CPoE$(C=2,\gamma=0.5)$ achieves KL $\ll$ GPoE/BCM at comparable wall-clock time and produces well-calibrated $95\%$ coverage. On UCI benchmarks ($N\approx 10^3$–$10^4$), CPoE$(C=2$–$4)$ consistently offers lower KL and root mean squared error (RMSE) than alternatives for equal or reduced runtime. For instance, on the "concrete" dataset ($N=927$), PoE(4) attains KL $\approx 122$ at $1.5$s, CPoE(4) achieves KL $\approx 80$ at $2.8$s, and the full GP KL $=0$ at $7.3$s. Large-scale tests ($N\approx 10^4$–$4\times 10^4$) demonstrate that CPoE with stochastic-Adam dominates SGP, GPoE, and GRBCM in CRPS versus time. Empirical results confirm that CPoE provides a tunable continuum between independent local and fully correlated global methods, with interpretable hyperparameters $C$ (correlation order) and $\gamma$ (inducing sparsity), enabling effective trade-off control between computational budget and posterior quality [2112.09519].

Source: https://www.emergentmind.com/topics/correlated-product-of-experts-cpoe