Random Feature Maps for Dot Product Kernels (1201.6530v3)

Abstract: Approximating non-linear kernels using feature maps has gained a lot of interest in recent years due to applications in reducing training and testing times of SVM classifiers and other kernel based learning algorithms. We extend this line of work and present low distortion embeddings for dot product kernels into linear Euclidean spaces. We base our results on a classical result in harmonic analysis characterizing all dot product kernels and use it to define randomized feature maps into explicit low dimensional Euclidean spaces in which the native dot product provides an approximation to the dot product kernel with high confidence.

• The paper introduces random feature maps that accurately approximate dot product kernels, mitigating the 'curse of support' in kernel methods.
• The methodology employs random projections based on the Maclaurin expansion and Schoenberg’s Theorem to ensure uniform kernel approximation.
• The approach enables scalable kernel approximations, significantly cutting computational costs in large-scale machine learning applications.

Overview of Random Feature Maps for Dot Product Kernels

Random Feature Maps for Dot Product Kernels by Purushottam Kar and Harish Karnick introduces a method to approximate dot product kernels through the use of random feature maps. The paper provides a novel approach to achieving dimensionality reduction while maintaining the benefits of kernel-based methods, a crucial development given the computational burdens associated with kernel methods in high-dimensional spaces.

Context and Motivation

Kernel methods, especially in machine learning, enable algorithms designed for linear feature spaces to operate in non-linear, potentially infinite-dimensional spaces. The 'kernel trick' allows computations to occur without explicitly mapping data to a higher-dimensional space, relying solely on inner product computations. However, this procedure often incurs significant computational costs, particularly the 'curse of support,' where large dataset sizes lead to prohibitively large support vectors when classifying or processing with Support Vector Machines (SVMs).

The authors build on Rahimi and Recht’s elegant result, which leveraged Bochner’s Theorem to create low-distortion embeddings for translation-invariant kernels, by extending these methods to dot product kernels. This particular extension addresses a broader class of kernels than previously considered and offers a systematic method to counteract the computational constraints of large support vector sizes.

Contribution and Methodology

Kar and Karnick’s essential contribution involves crafting feature maps for positive definite dot product kernels, i.e., kernels of the form $K(x,y) = f(\langle x, y \rangle)$, via randomized embeddings into Euclidean spaces. Employing results from harmonic analysis, particularly Schoenberg's Theorem, the authors derive conditions under which these feature maps provide estimates closely approximating the desired kernel value with high probability.

The feature maps are constructed by creating random projections based on the Maclaurin expansion of the function $f$ that defines the kernel. A crucial step is the introduction of an external probability measure on the indices of the expansion terms, which, ensured by algorithmic choices, maintains an exponential decrease in the probability of large deviations from the expected kernel value across the entire data domain. Unlike prior methods, the proposed mechanism accommodates homogeneous polynomial kernels which were a limitation in the approaches by Vedaldi and Zisserman.

Theoretical Underpinnings and Results

A significant theoretical component is the characterization of functions that yield positive definite dot product kernels. The results mobilize Schoenberg’s characterization in a finite-dimensional setting, asserting that a function admitting a non-negative Maclaurin expansion over a subset of Euclidean space qualifies as a positive definite kernel.

Theoretical proofs guarantee that these random feature maps give accurate kernel approximations uniformly over compact domains. With finite computational resources, they ensure with high probability that the feature maps provide approximations to the kernel, even when extended to compositional kernels involving other types of positive definite kernels.

Implications and Future Developments

The implications of this work are far-reaching, primarily in making kernel-based learning algorithms more scalable and efficient for large-scale applications. Integrating random feature maps with machine learning algorithms reduces dependency on vast computational resources, thus enhancing accessibility and usability in practical settings.

Additionally, this framework sets a precedent for exploring further kernel types and combination scenarios where similar reductions in dimensional complexity can be actualized without compromising predictive performance. Future work may examine extending these results to semi-definite or indefinite kernels where conventional methods still struggle to balance accuracy and computational load.

In conclusion, the paper's contribution represents a significant stride in kernel approximation techniques, promising enhanced efficiency for machine learning practitioners dealing with the "curse of support" in diverse application domains.