Mixture of Generalized Additive Models

• Mixture of GAMs is a machine learning framework characterized by integrating kernel-level random Fourier features with soft clustering and local additive models to capture nonlinear effects.
• It employs a four-stage pipeline including RFF approximation, PCA for dimensionality reduction, GMM-based clustering, and local spline-based GAM estimation to handle high-dimensional data.
• Empirical benchmarks demonstrate that the approach outperforms global GAMs and mixtures of linear models while maintaining clear per-covariate interpretability.

A mixture of generalized additive models (GAMs) is a machine learning framework that combines kernel-level representation learning via random Fourier features (RFFs), dimensionality reduction, probabilistic soft clustering, and locally adaptive generalized additive modeling. This approach aims to balance the empirical performance characteristic of complex nonparametric models with the interpretability associated with classical additive models. The methodology is constructed as an end-to-end pipeline, integrating RFF-based embeddings, principal component analysis (PCA) for latent structure compression, Gaussian mixture modeling (GMM) for soft clustering, and cluster-specific GAMs built from univariate spline smoothers. The combination enables nuanced, cluster-adaptive regression functions which are interpretable at the level of individual covariate effects while capturing local nonlinearities and heterogeneity in the data (Huang et al., 22 Dec 2025).

1. Model Architecture and Formulation

The mixture-of-GAMs framework begins with the selection of a shift-invariant kernel $\kappa(x - x')$ (e.g., Gaussian RBF), whose Fourier transform provides a spectral density $\rho(\omega)$. RFF approximates the kernel by sampling frequencies $\omega_1, \ldots, \omega_K \sim \rho(\omega)$ and defining the complex feature map: $$\varsigma(x) = \left[ e^{i \omega_1 \cdot x}, \ldots, e^{i \omega_K \cdot x} \right]^T \in \mathbb{C}^K$$ The regression function $m(x) = \mathbb{E}[Y|X=x]$ is approximated as a linear combination: $$\bar m(x) = \beta^T \varsigma(x) = \sum_{k=1}^K \beta_k e^{i \omega_k \cdot x}$$ where $\beta \in \mathbb{C}^K$ is fit by Tikhonov-regularized least squares. The design matrix $\Phi_{ik} = e^{i \omega_k \cdot x_i}$, and the regularized normal equations are: $$(\Phi^\dagger \Phi + \lambda I) \beta = \Phi^\dagger y$$ For clustering, a real "spectral" embedding is formed via the Hadamard product, $s(x) = \operatorname{Re}(\beta \odot \varsigma(x))$, producing $S \in \mathbb{R}^{N \times K}$ for $N$ samples.

Given that $K$ is generally large to capture fine kernel structure, PCA is applied to the centered $S$ to produce a low-dimensional latent representation $Z = \bar S V_d \in \mathbb{R}^{N \times d}$. On $Z$, a Gaussian mixture model with $L$ components is fit, yielding soft assignments $\gamma_\ell(x)$ for each data point through posterior probabilities computed with respect to the mixture densities.

Each cluster $\ell$ receives a local GAM: $$\tilde f^{(\ell)}(x) = \alpha^{(\ell)} + \sum_{j=1}^p g_j^{(\ell)}(x_j)$$ where $g_j^{(\ell)}$ is a univariate spline (B-spline) basis expansion. The overall mixture prediction is given as: $$\hat y(x) = \sum_{\ell=1}^L \gamma_\ell(x) \tilde f^{(\ell)}(x)$$ This structure enables the resulting regression surface to be locally adaptive, nonparametric, and interpretable in terms of per-covariate effects (Huang et al., 22 Dec 2025).

2. Training Pipeline

Optimization of the entire model is executed in a structured, four-stage process:

1. Random Fourier Feature Model Fitting: Solve the Tikhonov-regularized normal equations to obtain $\beta$ for the RFF regressor.
2. Spectral Embedding, PCA, and Clustering: Compute $S$, center it, perform SVD to retain $d$ principal directions, form the latent representation $Z$, and fit a GMM via the EM algorithm to estimate $(\pi_\ell,\,\mu_\ell,\,\Sigma_\ell)$ and soft cluster assignments $\gamma_\ell(x)$.
3. Local GAM Estimation: Each sample is assigned to the cluster of highest $\gamma_\ell$. In each cluster, fit a GAM by minimizing the sum of squared errors plus a quadratic roughness penalty (on B-spline coefficients) via backfitting, ensuring control of smoothness through $\lambda_\text{smooth}$.
4. Inference: At prediction time, compute $s(x)$, project to $h(x)$ (PCA space), evaluate $\gamma_\ell(x)$, and generate the final output as the soft mixture of local GAM predictions.

This staged pipeline is designed for computational tractability, as joint optimization of all parameters is intractable in practice (Huang et al., 22 Dec 2025).

3. Interpretability and Analysis

Each local GAM decomposes its contribution into univariate smooth functions $g_j^{(\ell)}(x_j)$, preserving GAM-style transparency: the effect of each covariate is isolated within each cluster. Because clustering operates on a PCA-compressed RFF embedding, the resulting latent regimes correspond to regions of the input space with similar local structure as revealed by the learned spectral features. The cluster assignments are soft (i.e., probabilistic), allowing for partial association with multiple regimes.

Interpretability is enhanced further by:

• Visualizing each cluster's shape functions to elucidate how marginal effects of each covariate vary across data regimes.
• Applying standard tools such as partial-dependence plots to each local GAM.
• Mapping soft responsibilities $\gamma_\ell(x)$ back to the input space for analysis of geographic or domain-specific structure.

The spectral embedding also offers insight into the most informative input directions, with distributions of learned $\omega_k$ frequencies often revealing dominant variation modes (for example, spatial gradients in housing price data) (Huang et al., 22 Dec 2025).

4. Empirical Performance and Benchmark Results

The mixture-of-GAMs framework demonstrates consistent empirical gains on benchmark regression tasks compared to classical interpretable and mixture-of-linear approaches:

Dataset	Metric	Mixture-of-GAMs	Global GAM	LASSO	MARS	Mixture-of-Lin	RFF/Other
California Housing	RMSE [$10^5$]	0.50	0.57	0.72	0.64	0.57–0.58	-
NASA Airfoil Self-Noise	RMSE [dB]	2.22	4.51	-	-	-	1.08 (RFF)
Bike Sharing	RMSE [rentals/hour]	58.2	88.8	-	-	comparable	-

Data augmentation with perturbed RFF samples further reduces the NASA Airfoil mixture RMSE to 2.02 dB. On all tasks, the proposed method matches or outperforms global additive models and noninterpretable baselines, maintaining full nonlinear interpretability (Huang et al., 22 Dec 2025).

5. Relationship to Existing Methods

The mixture-of-GAMs approach bridges black-box models (kernel machines, DNNs) and transparent statistical models (GAMs, splines, additive models) by blending expressive random Fourier-based representations and explicit regime discovery with classic additive interpretability. Distinct from global GAMs, which impose a uniform functional structure, the present method provides locally adaptive smoothing and effect decomposition. Relative to prior mixture-of-linear models, the use of B-spline-based smoothers in each cluster introduces nonlinear flexibility while retaining clear visualization and effect analysis capabilities (Huang et al., 22 Dec 2025).

6. Limitations and Prospective Extensions

The staged optimization approach is a necessary response to the nonconvexity of joint estimation but implies that fitting is not globally optimal and may depend on choices in early pipeline stages. Model complexity is governed by several parameters: the number of RFFs ($K$), latent dimension ($d$), cluster count ($L$), and spline basis sizes. The interpretability advantages rely on the meaningfulness of latent clusters; poorly separated spectral embeddings may hinder local interpretability. The integration of richer cluster models or alternative embeddings, as well as fully end-to-end training schemes, could be explored to further combine statistical efficiency with transparency (This suggests active research opportunities for the development of more optimal or adaptive pipelines.) (Huang et al., 22 Dec 2025).

7. Conclusion

The mixture-of-GAMs framework constructed from RFF embeddings, dimensionality compression, soft clustering, and additive smoothing demonstrably bridges the gap between predictive accuracy and interpretability. By identifying locally homogeneous regimes in the kernel feature space and fitting explicit additive models within each regime, the approach delivers clear per-covariate effect plots and nuanced, data-adaptive regression surfaces. Extensive empirical benchmarks confirm its capacity to match or exceed traditional GAMs and mixtures of linear models while preserving the hallmark transparency of the additive modeling paradigm (Huang et al., 22 Dec 2025).