Fastfood: Approximate Kernel Expansions in Loglinear Time (1408.3060v1)

Abstract: Despite their successes, what makes kernel methods difficult to use in many large scale problems is the fact that storing and computing the decision function is typically expensive, especially at prediction time. In this paper, we overcome this difficulty by proposing Fastfood, an approximation that accelerates such computation significantly. Key to Fastfood is the observation that Hadamard matrices, when combined with diagonal Gaussian matrices, exhibit properties similar to dense Gaussian random matrices. Yet unlike the latter, Hadamard and diagonal matrices are inexpensive to multiply and store. These two matrices can be used in lieu of Gaussian matrices in Random Kitchen Sinks proposed by Rahimi and Recht (2009) and thereby speeding up the computation for a large range of kernel functions. Specifically, Fastfood requires O(n log d) time and O(n) storage to compute n non-linear basis functions in d dimensions, a significant improvement from O(nd) computation and storage, without sacrificing accuracy. Our method applies to any translation invariant and any dot-product kernel, such as the popular RBF kernels and polynomial kernels. We prove that the approximation is unbiased and has low variance. Experiments show that we achieve similar accuracy to full kernel expansions and Random Kitchen Sinks while being 100x faster and using 1000x less memory. These improvements, especially in terms of memory usage, make kernel methods more practical for applications that have large training sets and/or require real-time prediction.

• The paper introduces Fastfood, a novel method that leverages Hadamard and Gaussian matrices to approximate kernel expansions and reduce computation from O(nd) to O(n log d).
• The paper applies Fastfood to diverse kernels, including RBF and polynomial, ensuring unbiased approximations with low variance across tasks.
• The paper demonstrates that Fastfood achieves up to 100-fold speed improvements and tenfold memory efficiency, making high-dimensional kernel learning practical.

Fastfood: Approximate Kernel Expansions in Loglinear Time

The paper "Fastfood: Approximate Kernel Expansions in Loglinear Time" presents an innovative method termed "Fastfood" for accelerating kernel computations in large-scale machine learning tasks. Kernel methods are well-regarded for their effectiveness in various machine learning applications, particularly in classification, regression, and feature extraction. However, the computational and storage demands of kernel methods often limit their applicability to smaller datasets. This paper addresses these limitations by introducing a method that reduces both the time complexity from $O(nd)$ to $O(n \log d)$ and the space complexity from $O(nd)$ to $O(n)$, where $n$ is the number of nonlinear basis functions and $d$ is the data dimensionality.

Key Insights and Contributions

1. Hadamard and Gaussian Matrices: The essence of Fastfood is based on the observation that combinations of Hadamard matrices and diagonal Gaussian matrices can mimic the behavior of dense Gaussian random matrices. These combinations offer significant computational advantages as Hadamard matrices facilitate fast Fourier transformations, enabling swift multiplications. This allows for efficient kernel approximations without sacrificing accuracy.
2. Broader Kernel Applicability: Fastfood is applicable to a range of kernel functions, including popular choices such as Radial Basis Function (RBF) and polynomial kernels. This broad applicability stems from Fastfood's unbiased nature and low variance, which ensures that it provides robust kernel approximations.
3. Significant Computational Savings: The Fastfood algorithm requires $O(n \log d)$ operations to compute kernel approximations, a significant improvement over standard methods that require $O(nd)$ operations. This advancement renders kernel methods feasible for applications involving high-dimensional data or requiring real-time predictions.
4. Enhanced Memory Efficiency: Fastfood reduces the storage requirement to $O(n)$, making it particularly attractive for real-time systems and memory-constrained applications.
5. Empirical Validation: Extensive experiments demonstrate that Fastfood achieves comparable accuracy with conventional kernel methods and other approximation techniques like Random Kitchen Sinks (RKS) while being orders of magnitude faster and more memory efficient. In particular, results indicated Fastfood was tenfold more memory efficient and 100-fold faster in specific scenarios.

Implications and Future Directions

The development of Fastfood has substantial practical implications for the deployment of kernel methods in resource-constrained environments such as mobile devices. The algorithm's reduced memory consumption allows for the integration of sophisticated kernel-based models into applications with limited computational resources. Theoretical analyses indicate that Fastfood maintains the advantages of traditional kernel expansions while enabling their application to significantly larger datasets than previously feasible.

From a theoretical standpoint, the Fastfood method opens new avenues for approximate kernel computations, particularly for kernels reliant on various symmetry groups. Future research could explore extending Fastfood to other kernel types, strengthening its theoretical guarantees, and potentially discovering novel kernel applications across diverse fields such as signal processing, natural language processing, and beyond.

The introduction of Fastfood marks a significant advancement in the scalability and efficiency of kernel methods, offering the potential to broaden their impact across various machine learning applications. The ability to perform computations rapidly and with minimal memory overhead is invaluable in the era of big data, where the ability to scale efficiently is as critical as the model's accuracy.