---
title: Gaussian Kernel Feature Space
url: https://www.emergentmind.com/topics/feature-space-for-gaussian-kernels
type: topic
---

# Gaussian Kernel Feature Space

A Gaussian kernel is a symmetric, positive-definite function of the form
\[
k(x, y) = \exp\left(-\frac{\|x - y\|^2}{2\sigma^2}\right), \quad x, y \in \mathbb{R}^d
\]
or, more generally, defined on a real separable Hilbert space $\mathcal{H}$. The feature space corresponding to a Gaussian kernel is the Hilbert space into which data are mapped such that $k(x, y)$ equals the inner product between the mapped points. This feature space is typically infinite-dimensional, and its structure underlies the expressive power, universality, and practical implementation of kernel methods in statistical learning, Gaussian processes, SVMs, and related fields. Several analytically distinct but isometrically equivalent constructions capture the geometry and analytic regularity of the Gaussian kernel's feature space across finite- and infinite-dimensional domains.

## 1. Reproducing Kernel Hilbert Space of the Gaussian Kernel

Let $\mathcal{H}$ be a real separable Hilbert space. The Gaussian kernel $k(x, y)=\exp(-\|x-y\|^2/(2\sigma^2))$ is positive definite on $\mathcal{H}$. By Aronszajn’s theorem, there exists a unique reproducing kernel Hilbert space (RKHS) $\mathcal{H}_k$ consisting of functions $f:\mathcal{H}\to\mathbb{R}$ with the following properties [2007.14697]:

- For each $y\in\mathcal{H}$, $k(\cdot, y) \in \mathcal{H}_k$
- The reproducing property: $\forall f \in \mathcal{H}_k$, $f(y) = \langle f, k(\cdot, y) \rangle_{\mathcal{H}_k}$
- The span $\{k(\cdot, y) : y \in \mathcal{H}\}$ is dense in $\mathcal{H}_k$

Every $f \in \mathcal{H}_k$ admits a representation $f(\cdot) = \sum_{i=1}^\infty c_i k(\cdot, x_i)$ (converging in $\mathcal{H}_k$), with norm $\|f\|_{\mathcal{H}_k}^2 = \sum_{i,j} c_i c_j k(x_i, x_j)$. The feature map $\Phi: x \mapsto k(\cdot, x)$ embeds $\mathcal{H}$ into $\mathcal{H}_k$ so that $k(x, y) = \langle \Phi(x), \Phi(y) \rangle_{\mathcal{H}_k}$.

## 2. Mercer and Hermite Expansions: Explicit Feature Maps

On $\mathbb{R}^d$, Mercer’s theorem applies to the Gaussian kernel and gives an eigen-decomposition:
\[
k(x,y) = \sum_{n=0}^\infty \lambda_n \varphi_n(x) \varphi_n(y)
\]
The expansion is explicitly realized in terms of Hermite polynomials. For the one-dimensional case, with standard normal weight $\mu_0(dx)$ and orthonormal Hermite functions $\psi_n(x)$, the kernel admits
\[
\exp\left(-\sigma^2(x-y)^2\right) = \sum_{n=0}^\infty \lambda_n \varphi_n(x) \varphi_n(y)
\]
where the $\varphi_n$ are scaled Hermite functions and $\lambda_n$ depends explicitly on $\sigma^2$ [2110.05778]. In higher dimensions, the eigenfunctions and eigenvalues tensorize across input coordinates.

The associated feature map is
\[
\Phi(x) = \left(\sqrt{\lambda_\beta} \Phi_\beta(x) \right)_{\beta \in \mathbb{N}_0^s} \in \ell^2(\mathbb{N}_0^s)
\]
where $\Phi_\beta(x)$ is the tensor product of univariate Hermite functions. For general (possibly infinite) $s$, the maximal domain consists of sequences $(x_j)$ with $\sum_j \sigma_j^2 x_j^2 < \infty$, and the RKHS is the incomplete tensor product of the univariate spaces [2110.05778].

## 3. Fourier (Bochner) Representations and Random Features

For shift-invariant kernels, Bochner’s theorem yields a spectral representation:
\[
k(x,y) = \int_{\mathbb{R}^d} e^{i \omega \cdot (x-y)} \mu(d\omega)
\]
For the Gaussian kernel, $\mu$ is a Gaussian measure with density $\mu(d\omega) = (2\pi\sigma^2)^{-d/2} e^{-\sigma^2 \|\omega\|^2/2} d\omega$ [1707.08492]. The (complex-valued) feature map is $\phi(x,\omega) = e^{i \omega \cdot x}$ and $\phi(x, \cdot)$ belongs to $L^2(\mu)$.

In practical applications, random Fourier features (RFF) approximate the Gaussian kernel by finite-dimensional real-valued feature maps [1711.05615]. Drawing $w_i \sim \mathcal{N}(0,\sigma^{-2} I_d)$ and $b_i \sim \text{Uniform}[0, 2\pi]$,
\[
\phi(x) = \sqrt{\frac{2}{D}} \left( \cos(w_1^\top x + b_1), \ldots, \cos(w_D^\top x + b_D) \right)^T
\]
The inner product of finite feature vectors converges to $k(x,y)$ at rate $O(1/\sqrt{D})$ in spectral norm, as $D\to\infty$ [1711.05615].

## 4. Polynomial and Maclaurin Expansions; Localized Feature Maps

Equivalently, the exponential can be expanded via Maclaurin series,
\[
k(x,y) = \sigma^2 \exp\left(-\frac{\|x\|^2 + \|y\|^2}{2\ell^2} \right) \sum_{n=0}^\infty \frac{(x\cdot y)^n}{n! \ell^{2n}}
\]
yielding the "polynomial sketch" feature map, where monomials are approximated by random projections (polynomial sketches) [2204.05667]. Localization—centering the feature map around the test point—cures the pathologies of naive Maclaurin truncation, yielding accurate, finite-dimensional local approximations, especially effective for high-frequency (short lengthscale) data regimes.

## 5. Finite-Dimensional Explicit Feature Maps for Finite Sets

For a finite dataset $\{x_1, ..., x_N\}$, an explicit, exact $N$-dimensional feature map can be constructed [2410.12635]. Defining the kernel matrix $K \in \mathbb{R}^{N \times N}$, its inverse square root $K^{-1/2}$, and $k_z = (k(x_1, z), ..., k(x_N, z))^T$, one defines
\[
\phi(z) = K^{-1/2} k_z
\]
This map reproduces the kernel exactly between any pair $x, y$ where at least one argument is a training point: $\phi(x)^T \phi(y) = k(x, y)$. The explicit feature representation enables kernel PCA and other linear methods to be directly implemented in the primal space.

## 6. Fock and Segal–Bargmann Space Constructions

The Gaussian feature space can be realized as a Fock (Segal–Bargmann) space of entire functions $F: \mathbb{C}^d \to \mathbb{C}$,
\[
\|F\|_{H_\sigma}^2 = \left(\frac{2}{\pi \sigma^2}\right)^d \int_{\mathbb{C}^d} |F(z)|^2 e^{-2|z|^2 / \sigma^2} dA(z) < \infty
\]
with an orthonormal basis of monomials and explicit reproducing kernel $K_F(z,w) = \exp(a z \cdot w)$ ($a = 2/\sigma^2$) [2210.14167]. The Segal–Bargmann transform provides an isometric isomorphism between $L^2(\mathbb{R}^d)$ and $H_\sigma$. The Gaussian kernel is realized as the inner product in this space,
\[
k_\sigma(x, y) = \langle \Phi_\sigma(x), \Phi_\sigma(y) \rangle_{H_\sigma}
\]
with explicit Hermite-basis expansion.

## 7. Universality, ISPD, and Domain Extensions

The Gaussian kernel is universal, integrally strictly positive definite (ISPD), and $C_0$-universal on Hilbert spaces; $\mathcal{H}_k$ is dense in $C_0(\mathcal{H})$, ensuring that the feature-map embedding is rich enough to approximate all continuous functions vanishing at infinity [2007.14697]. These properties ensure strong consistency and flexibility for statistical learning. Analogous constructions exist for Gaussian-type kernels on non-Euclidean domains, including hyperbolic spaces via Schoenberg’s theorem and generalizations to conditional negative definiteness (CND) kernels.

## Table: Major Constructions of Gaussian Kernel Feature Spaces

| Construction Type               | Space / Domain        | Feature Map / Expansion                         |
|---------------------------------|-----------------------|-------------------------------------------------|
| Hermite/Mercer Expansion        | $\mathbb{R}^d,\mathcal{H}$ | Hermite polynomials via Mercer, $\ell^2$        |
| Bochner/Fourier Representation  | $\mathbb{R}^d$        | $L^2(\mu)$ (Fourier basis, RFF)                 |
| Polynomial/Maclaurin            | $\mathbb{R}^d$        | Monomials/polynomial sketches, localized        |
| Fock/Segal–Bargmann Space       | $\mathbb{C}^d$        | Entire holomorphic functions, Hermite basis     |
| Explicit finite-dimensional     | $N$-point subset      | $N$-dimensional exact construction              |

Each construction realizes the RKHS with different analytical and computational properties: Hermite gives orthonormal bases, Bochner yields spectral sampling, polynomial/Maclaurin facilitates sparse approximations, Fock spaces bring connections to complex analysis and quantum mechanics, while finite-dimensional explicit maps enable primal implementations over finite data sets.

## References

- [2007.14697]: On Gaussian kernels on Hilbert spaces and kernels on Hyperbolic spaces
- [2110.05778]: Countable Tensor Products of Hermite Spaces and Spaces of Gaussian Kernels
- [1707.08492]: Reproducing kernels and choices of associated feature spaces, in the form of $L^{2}$-spaces
- [2204.05667]: Local Random Feature Approximations of the Gaussian Kernel
- [2410.12635]: An Exact Finite-dimensional Explicit Feature Map for Kernel Functions
- [1711.05615]: Spatial Mapping with Gaussian Processes and Nonstationary Fourier Features
- [2210.14167]: An approach to the Gaussian RBF kernels via Fock spaces

Source: https://www.emergentmind.com/topics/feature-space-for-gaussian-kernels