Exponential Kernel Feature Map

• The exponential kernel feature map is an explicit mapping that transforms data into a finite-dimensional space where the exponential (Laplacian) kernel is exactly reproduced.
• It enables traditional kernel methods to be reformulated in the primal space using exact finite-dimensional constructions or random feature approximations for scalability.
• The method involves computational trade-offs like eigendecomposition cost and numerical stability, making it suitable for reformulating algorithms such as kernel PCA and SVM.

The exponential kernel feature map refers to the explicit mapping of data points into a feature space such that the inner product in this space reproduces the value of an exponential-type kernel, such as the Laplacian kernel $k(x, y) = \exp(-\|x-y\|/\sigma)$. This construction allows kernel methods to be expressed in the primal (feature) space, enabling direct application of linear algorithms and bypassing the need for implicit kernel tricks. Exponential kernels admit both exact finite-dimensional feature maps for empirical settings and random feature or polynomial approximations for scalable learning with theoretical and practical trade-offs.

1. Formal Definition and Exact Finite-Dimensional Construction

Consider any positive-definite kernel $k(x, y)$. By Mercer's theorem, there exists a (typically infinite-dimensional) Hilbert space $H$ and feature map $\psi: X \to H$ satisfying $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$. For a finite training set $\{x_1,\ldots,x_N\} \subset X$, define the Gram matrix $K \in \mathbb{R}^{N \times N}$ by $K_{ij} = k(x_i, x_j)$. The explicit finite-dimensional feature map is

$$\phi(z) = K^{-1/2} \left[k(x_1, z),\ldots, k(x_N, z)\right]^T \in \mathbb{R}^N,$$

for any $z \in X$, where $k(x, y)$0 denotes the symmetric matrix square root of the inverse of $k(x, y)$1 (Ghiasi-Shirazi et al., 2024).

This feature map reproduces the kernel precisely: $k(x, y)$2 for all $k(x, y)$3 and any $k(x, y)$4. If $k(x, y)$5 is of rank $k(x, y)$6, zero-eigenvalue directions may be discarded, yielding a minimal feature dimension $k(x, y)$7.

2. Specialization: Exponential (Laplacian) Kernel

For the Laplacian kernel $k(x, y)$8 with training set $k(x, y)$9, compute the Gram matrix $H$0 and its economy-size eigendecomposition $H$1, where $H$2 and $H$3. The exact exponential kernel feature map is

$H$4

where $H$5. This feature map yields

$H$6

provided at least one of $H$7 or $H$8 is among the training points, and is exact for all training-to-training or training-to-test interactions (Ghiasi-Shirazi et al., 2024).

Table: Feature Map Dimensionality and Properties

Kernel	Construction Type	Feature Dimension $H$9
Exponential (Laplacian)	Exact, empirical	$\psi: X \to H$0 (all coordinates) or $\psi: X \to H$1
Gaussian (Taylor)	Truncated polynomial	$\psi: X \to H$2 for degree $\psi: X \to H$3
Exponential, dot-product	Random Gegenbauer	$\psi: X \to H$4 (user-chosen sample size)

The construction is parameter-free (no approximation parameter) and yields an exact map for the empirical kernel (Ghiasi-Shirazi et al., 2024).

3. Implications for Reformulating Kernelized Algorithms

The availability of exact empirical feature maps enables reformulation of traditional kernel methods in their primal form. For example, in kernel PCA (KPCA), the dual problem using kernel evaluations is replaced by ordinary PCA performed on the mapped and mean-centered feature vectors in $\psi: X \to H$5:

• Each training point is mapped to $\psi: X \to H$6.
• The sample covariance $\psi: X \to H$7 is computed in $\psi: X \to H$8.
• Eigenvectors and principal values are obtained from $\psi: X \to H$9.

Analogous reformulations exist for kernel SVM, kernel ridge regression, and other kernel-based algorithms, admitting primal solvers and leveraging explicit linear algebra (Ghiasi-Shirazi et al., 2024).

4. Computational Complexity and Numerical Stability

Given $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$0 training samples and ambient dimension $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$1:

• Constructing $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$2 requires $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$3 operations.
• Eigendecomposition of $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$4 costs $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$5 flops.
• Constructing $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$6 (or $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$7) costs $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$8, reducible to $k(x, y) = \langle \psi(x), \psi(y)\rangle_H$9 with low-rank truncation for $\{x_1,\ldots,x_N\} \subset X$0.
• Mapping a new test point $\{x_1,\ldots,x_N\} \subset X$1 requires $\{x_1,\ldots,x_N\} \subset X$2 to compute $\{x_1,\ldots,x_N\} \subset X$3, followed by $\{x_1,\ldots,x_N\} \subset X$4 for the feature mapping.

Numerical stability is governed by the conditioning of $\{x_1,\ldots,x_N\} \subset X$5. Adding a small ridge ($\{x_1,\ldots,x_N\} \subset X$6) ensures invertibility and numerical robustness. Eigenvalues below a threshold (e.g., $\{x_1,\ldots,x_N\} \subset X$7) can be discarded to improve stability and reduce feature dimensions (Ghiasi-Shirazi et al., 2024).

5. Approximate Feature Maps for Exponential-type Kernels

While exact finite-dimensional feature maps are feasible for kernels evaluated on finite datasets, exponential kernels can also be approximated for scalable learning in high dimensions:

• For dot-product exponential kernels $\{x_1,\ldots,x_N\} \subset X$8, a random feature map based on the Gegenbauer (ultraspherical) expansion is available (Kar et al., 2012):
  • Draw degree $\{x_1,\ldots,x_N\} \subset X$9 according to the distribution $K \in \mathbb{R}^{N \times N}$0 with $K \in \mathbb{R}^{N \times N}$1 computable in terms of $K \in \mathbb{R}^{N \times N}$2.
  • Draw $K \in \mathbb{R}^{N \times N}$3 uniformly from the unit sphere $K \in \mathbb{R}^{N \times N}$4.
  • The feature is $K \in \mathbb{R}^{N \times N}$5, where $K \in \mathbb{R}^{N \times N}$6 is the Gegenbauer polynomial.

With $K \in \mathbb{R}^{N \times N}$7 samples, this yields an unbiased estimator of the kernel, satisfying

$K \in \mathbb{R}^{N \times N}$8

for all $K \in \mathbb{R}^{N \times N}$9 in the unit ball, and with $K_{ij} = k(x_i, x_j)$0 ensures uniform error less than $K_{ij} = k(x_i, x_j)$1 with probability at least $K_{ij} = k(x_i, x_j)$2 (Kar et al., 2012).

• For Gaussian (RBF) kernels, Taylor polynomial feature maps based on the series expansion of the exponential provide explicit approximations, with an error bounded by

$K_{ij} = k(x_i, x_j)$3

and polynomial feature dimension $K_{ij} = k(x_i, x_j)$4, leading to efficient evaluation for sparse data (Cotter et al., 2011).

A plausible implication is that practitioners may select between exact empirical maps, random features, or polynomial expansions according to memory, scalability, and precision requirements.

6. Limitations and Comparison to Other Kernel Feature Maps

The exact finite-dimensional construction is specific to a fixed set of training data; its extension to unseen data requires access to $K_{ij} = k(x_i, x_j)$5 and its spectral decomposition, and the feature map dimension grows linearly (or as the rank) with the size of the training set. By contrast, approximate random features (e.g., Rahimi-Recht Fourier features, random Gegenbauer features for dot-product kernels) can be constructed at arbitrary dimension and optimized for out-of-sample generalization (Cotter et al., 2011, Kar et al., 2012).

Random features offer explicit scalability-accuracy trade-offs, with uniform error decaying as $K_{ij} = k(x_i, x_j)$6, whereas the exact map is parameter-free, non-approximate, but subject to the finite sample size and potential numerical instability for ill-conditioned $K_{ij} = k(x_i, x_j)$7.

7. Applications and Impact

Explicit exponential kernel feature maps enable kernel methods to be executed in primal form, allowing application of standard (non-kernelized) solvers for PCA, ridge regression, SVM, and visualization techniques (e.g., t-SNE directly in feature space) (Ghiasi-Shirazi et al., 2024). For high-dimensional or large-scale settings, approximate or random-feature maps facilitate kernel learning under restricted computational budgets, with super-polynomial error decay in certain regimes for polynomial (Taylor) features and rapid convergence for random Gegenbauer features (Cotter et al., 2011, Kar et al., 2012).

The methodology generalizes to arbitrary positive-definite kernels, significantly broadening the practical reach of kernel-based learning.