Kernel Linearization & Random Features

• Kernel linearization and random features are methods that approximate nonlinear kernels by mapping data into fixed-dimensional spaces using random projections.
• These techniques leverage Bochner’s theorem and Monte Carlo approximations to transform complex kernel evaluations into efficient linear computations.
• Advanced variants like ORF, SimRF, and HRF reduce variance and improve accuracy, proving effective in regression, classification, and structured data tasks.

Kernel linearization and random features are foundational tools for scaling kernel methods to large datasets, enabling the approximation of nonlinear kernel evaluations within fixed-dimensional, computationally efficient linear models. This article reviews the mathematical basis, algorithmic framework, theoretical properties, efficient variants, and selected extensions of kernel linearization via random features, referencing key contributions across regression, classification, and structured domains.

1. Mathematical Basis: Bochner’s Theorem and Random Features

Kernel linearization is fundamentally grounded in Bochner’s theorem, which states that any continuous, shift-invariant, positive-definite kernel $k(x, x') = k(x - x')$ on $\mathbb{R}^d$ admits a spectral representation: $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$ where $p(\omega)$ is the kernel’s spectral (Fourier) density. This allows the kernel to be approximated by the expected product of random features: $$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$ with $$\phi_{\omega, b}(x) = \sqrt{2} \cos(\omega^\top x + b)$$, $\omega \sim p(\omega)$, and $b \sim \text{Uniform}[0, 2\pi]$ (Bouboulis et al., 2016). For $D$ features, the random feature mapping is $$z(x) = [\phi^1(x), \ldots, \phi^D(x)]^\top \in \mathbb{R}^D$$, yielding a Monte Carlo approximation to the kernel inner product.

2. Canonical Algorithms and Computational Properties

Kernel linearization replaces a nonlinear kernel machine with a linear model learned in random feature space. The process consists of:

1. Sampling: Draw $\mathbb{R}^d$0 i.i.d. pairs $\mathbb{R}^d$1 for $\mathbb{R}^d$2 according to $\mathbb{R}^d$3 and $\mathbb{R}^d$4.
2. Feature Mapping: For each input $\mathbb{R}^d$5, compute $\mathbb{R}^d$6.
3. Learning: Fit a linear model with parameters $\mathbb{R}^d$7 using standard solvers (e.g., ridge regression, LMS, or RLS) in the $\mathbb{R}^d$8-dimensional feature space.

For large-scale regression with input $\mathbb{R}^d$9 and targets $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$0, the regularized objective is: $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$1 where $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$2 contains $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$3 as rows. The solution is $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$4 (Gregorová et al., 2018, Bouboulis et al., 2016).

Complexity: Each alternating iteration typically requires $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$5 for building $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$6, $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$7 or $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$8 (CG) for solving for $$k(x, x') = \int_{\mathbb{R}^d} e^{i \omega^\top (x - x')} \, p(\omega) \, d\omega$$9, and $p(\omega)$0 for gradient steps. For $p(\omega)$1, training and prediction are linear in $p(\omega)$2 and $p(\omega)$3.

Approximation Error: For compact domains and error $p(\omega)$4, $p(\omega)$5 suffices for a uniform $p(\omega)$6-approximation with probability at least $p(\omega)$7, via standard concentration bounds (Gregorová et al., 2018, Bouboulis et al., 2016).

3. Advanced Variants: Structured and Adaptive Random Features

Recent developments have focused on reducing variance, improving memory or computational efficiency, and expanding applicability:

• Orthogonal Random Features (ORF) impose joint orthogonality on the random projections, reducing variance compared to i.i.d. sampling, though with a controlled bias towards approximating a Bessel kernel rather than the Gaussian (Demni et al., 2023).
• Simplex Random Features (SimRF) further minimize mean square error (MSE) for unbiased Gaussian/softmax kernel estimation among weight-independent geometrically coupled positive random features (Reid et al., 2023).
• Normalized RFF (NRFF) normalizes the random feature vector, reducing variance particularly for high-similarity pairs (Li, 2016).
• Hybrid Random Features (HRF) combine multiple base estimators (e.g., trigonometric and positive features) using data-driven mixing, providing strictly smaller worst-case relative errors and variance adaptations across the input space (Choromanski et al., 2021).

Table: Comparison of Selected Variants

Variant	Variance Reduction	Bias	Computational Impact
RFF	Baseline	Unbiased	$p(\omega)$8 per point
ORF	Yes	Bessel kernel	$p(\omega)$9, $$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$0 setup
SimRF	Yes (optimal)	Unbiased	$$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$1, possible $$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$2 block cost
NRFF	Up to $$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$3	Unbiased	Minor overhead
HRF	Adaptive	Unbiased	$$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$4

4. Extensions: Non-Euclidean, Structured, and Domain-Specific Random Features

Several methodologies extend random feature linearization beyond classical Euclidean kernels:

• Graph Random Features (GRF) construct low-rank unbiased random feature factorizations for regularized Laplacian kernels on graphs, using random walks to sidestep $$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$5 inversion and enabling distributed computation and quasi-Monte Carlo variance reduction (Choromanski, 2023).
• Manifold Random Features (MRF) learn continuous random feature fields via neural surrogates trained on signatures derived from GRFs on discretized manifolds, providing positive, bounded, unbiased features for general manifolds where no analytic factorization is available (Parashar et al., 3 Feb 2026).
• Mondrian and Rotated Mondrian Kernels approximate the Laplace kernel in either anisotropic ($$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$6) or isotropic ($$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$7) forms by simulating random tessellations (Mondrian partitions) and, in the rotated case, applying Haar-random rotations before partitioning, achieving fast random binning features with uniformly exponential convergence guarantees (Osborne et al., 6 Feb 2025, Balog et al., 2016, Wu et al., 2018).
• Random Binning Features (RB) use sparse block feature embeddings, achieving $$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$8 convergence—faster than standard RFF rates—especially advantageous with $$k(x, x') = \mathbb{E}_{\omega, b} [\phi_{\omega, b}(x) \phi_{\omega, b}(x')]$$9 regularization and parallel sparse block coordinate descent (Wu et al., 2018).

5. Theoretical Guarantees and Empirical Performance

The random feature paradigm provides rigorous approximation error bounds and convergence rates:

• For classical RFFs, $$\phi_{\omega, b}(x) = \sqrt{2} \cos(\omega^\top x + b)$$0, with expectation $$\phi_{\omega, b}(x) = \sqrt{2} \cos(\omega^\top x + b)$$1 (Bouboulis et al., 2016). Structured and orthogonal blocks reduce the MSE constant (Demni et al., 2023, Reid et al., 2023).
• For Mondrian-type and rotated binning features, uniform convergence rates are exponential in the number of features $$\phi_{\omega, b}(x) = \sqrt{2} \cos(\omega^\top x + b)$$2; for the rotated Mondrian kernel, these are proved for isotropic Laplace limits (Osborne et al., 6 Feb 2025).
• Random feature approximations can concentrate more sharply with adaptive (hybrid, simplex, or orthogonal) schemes. For example, SimRF achieves the smallest possible MSE among geometrically coupled positive RFs, and HRF offers worst-case error improvements by blending estimators (Reid et al., 2023, Choromanski et al., 2021).
• Empirical results consistently confirm that advanced random feature variants (ORF, SimRF, HRF, RB, and Mondrian) outperform basic RFF in kernel Gram approximation, downstream regression and classification, and practical memory—time tradeoffs (Demni et al., 2023, Reid et al., 2023, Wu et al., 2018, Osborne et al., 6 Feb 2025).

6. Algorithmic Innovations: Efficiency and Scalability

Efficient kernel linearization at scale relies on practical algorithmic design:

• Alternating Minimization for Variable Selection: Joint optimization of feature weights $$\phi_{\omega, b}(x) = \sqrt{2} \cos(\omega^\top x + b)$$3 and per-coordinate scaling parameters $$\phi_{\omega, b}(x) = \sqrt{2} \cos(\omega^\top x + b)$$4 (on the simplex) enables nonlinear variable selection via sparsity in the spectral scale vector (Gregorová et al., 2018).
• Sublinear and Structured Embeddings: Fastfood, circulant, or Toeplitz transforms recycle Gaussian vectors to generate structured random matrices for kernel approximation with $$\phi_{\omega, b}(x) = \sqrt{2} \cos(\omega^\top x + b)$$5 or $$\phi_{\omega, b}(x) = \sqrt{2} \cos(\omega^\top x + b)$$6 time (Choromanski et al., 2016).
• Online Learning: KLMS and KRLS algorithms benefit from fixed-size approximations via RFFs, avoiding pathologically growing dictionaries intrinsic to kernel trick-based online learning (Bouboulis et al., 2016).
• Hardware Acceleration: Optical Processing Units (OPUs) perform analog random projections for polynomial kernel approximations, achieving competitive accuracy with dramatically reduced time and energy cost (Ohana et al., 2019).

7. Limitations, Open Directions, and Specialized Domains

Despite broad applicability, kernel linearization by random features presents challenges and open questions:

• Dimensionality Dependence: The approximation constants and convergence rates of Mondrian-type and tessellation-based features depend on ambient dimension, potentially limiting their utility in very high-dimensional settings relative to RFF (Osborne et al., 6 Feb 2025).
• Indefinite, Non-Stationary, or Composite Kernels: Generalized orthogonal random features extend kernel linearization to indefinite yet stationary kernels (e.g., with signed spectral densities) and can be further adapted to compositional or non-Euclidean constructs (Luo et al., 2021).
• Learned and Adaptive Random Features: End-to-end frameworks jointly optimize kernel and predictive models (e.g., with generative networks over the Fourier domain), but theoretical generalization guarantees beyond classical RFF theory are not fully resolved (Fang et al., 2020, Băzăvan et al., 2012).
• Manifold and Graph Domains: While new MRF and GRF frameworks provide unbiased, positive, and scalable random features in non-Euclidean domains, approximation quality depends on discretization and the capacity of the surrogate field, and attaining optimal variance is an open challenge (Parashar et al., 3 Feb 2026, Choromanski, 2023).

Kernel linearization via random features continues to evolve, integrating developments from stochastic geometry, structured random matrix theory, distributive optimization, and geometric learning. The method remains central to the design of scalable, nonlinear learning systems in modern statistical and machine learning practice.