Spectral Mixture Generalizations of RFF

• The paper introduces a unified framework that approximates shift-invariant kernels by extending classical RFFs with spectral mixture models.
• It employs advanced techniques including mixtures of α-stable laws, variational inference, and quantum sampling to construct explicit spectral representations.
• The methodology improves kernel approximation accuracy and convergence rates, enhancing applications in SVMs, ridge regression, and Gaussian process models.

Spectral mixture generalizations of Random Fourier Features (RFF) provide a unified framework for approximating a broad class of positive-definite, shift-invariant kernels through explicit spectral representations and associated sampling techniques. Building on Bochner’s theorem, these generalizations move beyond the classical (Gaussian) RFF setting to mixtures of α-stable laws, mixtures of Gaussians with variational inference, data-adaptive spectral samplers (such as restricted Boltzmann machines or quantum devices), and scale mixtures supporting diverse, flexible kernels. The resulting methodology has far-reaching implications for kernel-based machine learning, particularly support vector machines, kernel ridge regression, and Gaussian process modeling.

1. Theoretical Framework for Spectral Mixture Kernels

The cornerstone of kernel generalization through spectral mixtures is Bochner’s theorem: every continuous, shift-invariant, positive-definite kernel $k(x - y)$ on $\mathbb{R}^d$ corresponds to the Fourier transform of a nonnegative, even spectral density $S(\omega)$. For isotropic kernels, the spectral density is radial and admits a representation as a mixture of characteristic functions of symmetric $\alpha$-stable laws. That is, for $k(r) = k(\|x - y\|)$, there exists a nonnegative random variable $R$ and a $\mathbb{R}^d$ symmetric $\alpha$-stable vector $S_\alpha$ such that the kernel admits the form (Langrené et al., 2024): $$k(r) = \mathbb{E}\bigl[\cos(\eta^\top u)\bigr],\quad \eta = (\lambda R)^{1/\alpha}\, S_\alpha,\quad \|u\| = r$$ and

$k(r) = \phi_R(i\lambda r^\alpha)$

where $\phi_R$ denotes the characteristic function (or equivalently Laplace transform) of $R$. This "scale-mixture" representation encompasses many classical and novel kernel families as special cases, such as exponential power, generalized Matérn, Cauchy, Beta, and Tricomi kernels.

2. Generalized Random Fourier Features (RFF) for Mixture Kernels

Random Fourier Features provide an explicit, finite-dimensional feature mapping to approximate $k(x-y)$: $$k(x-y) \approx \frac{1}{M} \sum_{m=1}^M \cos(\omega_m^\top(x-y))$$ For generalized kernels, $\omega_m$ is generated by:

1. Sampling a mixing scale $\tau_m \sim p_R(\cdot)$ (matching the target kernel family distribution).
2. Sampling $\mathbf{S}_m$ from a symmetric $\alpha$-stable distribution in $\mathbb{R}^d$.
3. Setting $$\omega_m = (\lambda\,\tau_m)^{1/\alpha} \mathbf{S}_m$$.

The feature embedding then takes $$z(x)_m = \sqrt{\frac{2}{M}} \cos(\omega_m^\top x + b_m)$$ with a random phase $b_m \sim \mathrm{Uniform}[0, 2\pi]$. For symmetric $\alpha$-stable laws, fast samplers are available (e.g., Devroye–Nolan–Chambers construction), and many kernel hyperparameters become interpretable as parameters of $p_R$(Langrené et al., 2024).

For spectral mixture (SM) kernels, as introduced by Wilson & Adams, the spectral density $S(\omega)$ is modeled as a finite mixture of Gaussians, possibly symmetrized, enabling a highly flexible adaptation to non-trivial stationary covariance structures (Jung et al., 2020). RFFs are extended by drawing spectral points from variational posteriors over SM parameters, constructing corresponding cosine-sine features, and aggregating across mixture components with optimal allocation weights.

3. Variational and Monte Carlo Inference for Spectral Mixtures

For flexible, data-adaptive kernels, aggregating random feature approximations requires effective learning of the underlying spectral measure. This is addressed through Bayesian variational inference over spectral points $S$ (Jung et al., 2020):

• A variational posterior $q(S)$ is optimized, with $S$ including both means and covariances for each SM mixture.
• The evidence lower bound (ELBO) is estimated as

$$\mathcal{L} = \mathbb{E}_{q(S)}[\log p(Y|X, S)] - \operatorname{KL}(q(S)\,||\,p(S))$$

• Gradients are computed via the reparameterization trick over $S$.
• Variance-reducing sampling schemes optimally allocate more random features to mixture components contributing higher kernel variance.
• Approximate natural-gradient updates in the log-parameter space allow for accelerated convergence of variational parameters.

Alternative approaches employ quantum annealers to sample from complex data-adaptive spectral models, typically by parameterizing the spectral density as a restricted Boltzmann machine (RBM), then mapping RBM samples to frequencies via a Gaussian–Bernoulli transformation (Hasegawa et al., 13 Jan 2026). Training occurs by optimizing a leave-one-out Nadaraya–Watson loss using squared-kernel-weight regression.

4. Kernel Mixture Families and Sampling Algorithm

The scale-mixture framework leads to tractable sampling and expansion formulas for a wide variety of kernels. The following table summarizes the kernel-specific correspondences and associated mixing distributions (Langrené et al., 2024):

Kernel Type	Spectral Mixture $p_R(\tau)$	$\alpha$-Stable Law Parameters
Exponential Power	$\delta(\tau-1)$	$\alpha \in (0,2]$, $R \equiv 1$
Generalized Matérn	Inverse Gamma ($\nu$)	$\alpha=2$, $R\sim\text{InvGamma}$
Generalized Cauchy	Gamma ($\beta$)	$\alpha \in (0,2]$, $R\sim\Gamma$
Kummer (Confluent)	Beta ($\beta, \gamma$)	$\alpha \in (0,2]$, $R\sim\mathrm{Beta}$
Beta Kernel	Transformed Beta	$\alpha \in (0,2]$, $R=-\log \mathrm{Beta}$
Tricomi	Fisher ($2\beta, 2\gamma$)	$\alpha \in (0,2]$, $R\sim\mathrm{F}$

For all these families, the RFF approximation proceeds by drawing $(\tau_m, S_m)$, computing $\omega_m$, and using cosine expansions. The computational overhead over classical RFF is minimal: one extra univariate draw per feature and the cost of $\alpha$-stable sampling, which is $O(d)$ per feature.

5. Error Bounds and Convergence Rates

The Monte Carlo error analysis for generalized RFFs follows Rahimi–Recht (2007) exactly: for each offset $u = x-y$, the estimator mean matches $k(\|u\|)$, variance is $\le 1/2$, and uniform deviations for a finite set are bounded via Hoeffding’s or McDiarmid’s inequalities. Symmetry and unbiasedness are conserved for all mixture models. Finite sample convergence is $O(1/\sqrt{M})$ for uniformly finite sets, with optimal $O(1/M)$ rates in expected $\ell^2$ error for large samples (Langrené et al., 2024).

6. Applications and Practical Implications

Generalized RFFs substantially extend the range of kernels available for scalable approximation in kernel SVMs, kernel ridge regression, kernel PCA, and Gaussian process inference. Practitioners may use these techniques to:

• Rapidly instantiate complex or highly-shaped stationary kernels via explicit random features;
• Learn kernel hyperparameters or even mixing distributions $p_R$ from data;
• Deploy quantum annealing–assisted or variationally trained feature samplers for adaptive data-driven kernel construction (Hasegawa et al., 13 Jan 2026, Jung et al., 2020);
• Incorporate model interpretability via the parametric form of $p_R$ and mixture family selection.

The methodology allows interpolation between a wide family of kernels, including the RBF/Gaussian, Laplace, Student–t, Matérn, and many newly introduced types, with no increase in asymptotic computational complexity relative to Gaussian RFFs. The convergence guarantees and unbiasedness extend to these broader cases under the same sampling and error analyses.

7. Recent Developments and Extensions

Recent literature demonstrates the integration of quantum annealing for kernel learning, using RBMs for flexible, learnable spectral densities and Gaussian‒Bernoulli mappings for feature generation, providing further control and adaptation for kernel regression tasks (Hasegawa et al., 13 Jan 2026). Empirical evidence suggests that such pipeline components can yield improved $R^2$ and RMSE performance over fixed-kernel methods, especially as the number of sampled features at inference increases.

Additionally, variational approaches to SM kernel learning using the ELBO, variance-reduced sampling, and natural-gradient steps have been shown to deliver accelerated convergence and resistance to overfitting relative to direct maximum-likelihood or classical RFF approaches (Jung et al., 2020).

---

References:

• "A spectral mixture representation of isotropic kernels to generalize random Fourier features" (Langrené et al., 2024)
• "Approximate Inference for Spectral Mixture Kernel" (Jung et al., 2020)
• "Kernel Learning for Regression via Quantum Annealing Based Spectral Sampling" (Hasegawa et al., 13 Jan 2026)