DeepEP Mixture-of-Experts Kernels

• The paper introduces a novel architecture that combines deep DNN gating with sparse GP experts using a one-pass CCR algorithm for efficient MAP estimation.
• It leverages FITC and hard clustering to achieve scalable inference, significantly reducing computational costs on high-dimensional datasets.
• Empirical results demonstrate competitive R² scores, tight credible intervals, and decreased run times relative to state-of-the-art methods.

DeepEP Mixture-of-Experts Kernels are a class of models for supervised learning that combine deep neural network (DNN) gating functions with ensembles of sparse Gaussian process (GP) experts. The architecture employs a mixture-of-experts formulation, allowing both the mean function and output distribution to adapt flexibly with respect to the input. This method integrates efficient approximate inference using the one-pass Cluster-Classify-Regress (CCR) algorithm for maximum a posteriori (MAP) estimation. The approach achieves competitive or superior accuracy and uncertainty quantification (UQ), with computational advantages particularly pronounced in higher-dimensional and large-scale data regimes [2006.13309].

1. Model Architecture and Formulation

The DeepEP model specifies the predictive density as a mixture-of-experts:
$$
p(y \mid x) = \sum_{k=1}^{K} \pi_k(x; \theta) \; p_k(y \mid x; \Phi^{(k)}) ,
$$
where $\pi_k(x; \theta)$ is a DNN-based gating function yielding mixture weights ($\sum_k \pi_k = 1$), and $p_k(y \mid x; \Phi^{(k)})$ is the conditional distribution from the $k$-th sparse GP expert with hyperparameters $\Phi^{(k)}$.

The gating network $h(x; \psi) = (h_1(x; \psi), ..., h_K(x; \psi))$ consists of $J$ feed-forward layers with ReLU nonlinearities. Output mixing coefficients are obtained via a softmax:
$$
\pi_k(x; \psi) = \frac{\exp(h_k(x; \psi))}{\sum_{\ell=1}^{K}\exp(h_\ell(x; \psi))}.
$$

Each expert is modeled as a Gaussian process, $f_k(\cdot) \sim \mathrm{GP}(\mu_k, K_{\phi_k}(\cdot, \cdot))$, typically using an ARD squared-exponential kernel:
$$
K_{\phi_k}(x, x') = \sigma_{f,k}^2 \exp\left( -\frac{1}{2} \sum_{d=1}^D \frac{(x_d - x'd)^2}{\ell{k,d}^2} \right),
$$
with input dimension-specific lengthscales $\ell_{k,d}$ and variance $\sigma_{f,k}^2$.

2. Sparse Gaussian Process Experts

For scalability, each expert leverages sparse GP inference via the fully independent training conditional (FITC) approximation. Introducing $M_k \ll N_k$ inducing locations $Z_k = {\tilde{x}{k,m}}$, the GP marginal and posterior computations are replaced by lower-cost approximations. For a data point $x_i$, the FITC predictive mean and variance are:
$$
\widehat{\mu}_k(x_i) = \mu_k + k{Z_k, x_i}^\top K_{ZZ}^{-1}(u_k - \mu_k \mathbf{1}), \
\lambda_k(x_i) = K_{\phi_k}(x_i, x_i) - k_{Z_k,x_i}^\top K_{ZZ}^{-1} k_{Z_k,x_i},
$$
where $u_k$ is the vector of inducing function values and $k_{Z_k, x_i}$ is the covariance between the inducing points and $x_i$. The likelihood for $y_i$ under expert $k$ is:
$$
p(y_i \mid x_i, z_i = k) = \mathcal{N}(y_i \mid \widehat{\mu}k(x_i), \sigma{n,k}^2 + \lambda_k(x_i)).
$$

3. Cluster-Classify-Regress (CCR) Algorithm

CCR is a one-pass MM (max–max) MAP estimation method comprising three sequential steps:

1. Cluster: Data $(y, x)$ is rescaled to emphasize $y$, then clustered using K-means or GMM, producing hard assignments $z_i \in {1, ..., K}$.
2. Classify: The DNN gating network is trained to predict these cluster assignments by maximizing the conditional log-likelihood with respect to $\psi$, $$ \psi = \arg\max_\psi \sum_{i=1}^{N}\sum_{k=1}^{K} \mathbf{1}{z_i = k} \left[ h_k(x_i; \psi) - \log \sum_{\ell=1}^K e^{h_\ell(x_i; \psi)} \right]. $$
3. Regress: For each cluster $k$, FITC-based sparse GP hyperparameters $(\phi_k, \sigma_{n,k}^2, Z_k)$ are optimized by maximizing $$ \log \mathcal{N}(y_k \mid \mu_k \mathbf{1}, \Sigma_k), $$ with $\Sigma_k = K_{Z_k X_k}^\top K_{Z_k Z_k}^{-1} K_{Z_k X_k} + \Lambda_k + \sigma_{n,k}^2 I$, $\Lambda_k = \mathrm{diag}(\lambda_k(x_i))$.

No further alternations between these steps are performed. Empirical results indicate the CCR solution typically approaches a local MAP mode, and additional MM iterations produce negligible improvement.

4. Objective Function and Computational Complexity

The maximum a posteriori (MAP) or type-II maximum likelihood objective, including allocations $z$, is:
$$
\log \pi(\psi, {\Phi_k}, {u_k} \mid X, y, z) \propto \sum_{i=1}^N \log \pi(z_i \mid x_i; \psi) + \sum_{i=1}^N \log p(y_i \mid x_i, z_i; \Phi_{z_i}) + \sum_{k=1}^K \log p(u_k \mid Z_k, \phi_k) + \text{(priors)}.
$$
The FITC structure facilitates blockwise decoupling. CCR executes clustering, DNN gating, and sparse GP regression each once:
- Clustering: $O(N K)$
- DNN training: $O(N p_c)$ (with $p_c$ DNN parameters)
- Sparse GP regression: $\sum_k O(N_k M_k^2)$

Overall CCR computational cost is $O(N P_{\max})$, where $P_{\max} = \max{p_c, M_1^2, ..., M_K^2}$. Full MM or two-pass MM2r variants are substantially more expensive due to repeated block updates.

5. Empirical Evaluation and Comparative Performance

CCR was evaluated on six datasets spanning dimensions $d=1$ to $d=10$ and sample sizes up to $N=150,000$. Baselines included mixture-density networks (MDN), generalized/robust product-of-experts (gPoE/RBCM), FastGP, BART, ORTHNAT, PPGPR, DSPP, Deep GPs, and treed GPs. Performance metrics covered $R^2$ predictive accuracy, 95% credible-interval (CI) average length and empirical coverage (EC), and wall-clock run time.

Selected quantitative results (5-fold CV averages):

Dataset	$N$, $d$	CCR $R^2$	Baseline Best $R^2$	CI Length	EC	CCR Time (s)	Baseline Time (s)
NASA	3,167, 3	97.07%	MDN: 96.80%	0.35	98.4%	10.1	MDN: 188
kin40k	40,000, 8	94.53%	FastGP: 92.94%	0.51	95.1%	85.7	FastGP: 120.8
chi (tokamak)	150,000, 10	95.71%	gPoE: 91.92%	0.63	97.5%	496	RBCM: 1,542

In all cases, CCR matched or exceeded the strongest baselines in both predictive accuracy and uncertainty quantification, typically running 2–3× faster than MM2r and an order of magnitude faster than large product-of-experts (PoE) or fully Bayesian GP mixtures on high-dimensional, large-$N$ problems [2006.13309].

6. Methodological Context and Implications

The integration of deep neural gating with sparse GP experts, and the use of the CCR algorithm, addresses the dual challenges of modeling flexible, input-dependent predictive densities and achieving computational tractability for large or high-dimensional datasets. The insight that single-pass CCR provides a solution nearly as good as iterative MM estimation suggests broader applicability to other mixture-of-experts settings. The use of FITC approximation, blockwise objective factorization, and hard clustering in expert assignment allows the method to maintain robustness, flexibility, and efficiency simultaneously.

A plausible implication is that for mixture-of-expert models where gating and local prediction are both nonlinear and complex, a one-pass MM approach with sparse local models and deep gating may suffice for practical purposes in many supervised learning problems, especially under computational constraints.

7. Connections to Related Methodologies

DeepEP Mixture-of-Experts Kernels generalize over classical mixture-of-experts frameworks by replacing parametric or logistic gating with flexible DNN gating, and by promoting nonparametric local models via sparse GPs. The approach is compared against mixture-density networks (MDNs), generalized product-of-experts (gPoE), robust Bayesian committee machines (RBCM), FastGP, Bayesian additive regression trees (BART), and various scalable and deep GP regression schemes. Its empirical evaluation confirms both its statistical and computational advantages relative to these frameworks, particularly in uncertainty quantification and speed for large-scale learning [2006.13309].