Adaptive Random Fourier Features Overview

• Adaptive Random Fourier Features (ARFF) are methods that tailor random Fourier feature maps by adapting spectral measures to the data, enabling task-specific kernel improvements.
• ARFF employs various strategies including spectral-parameter learning, direct frequency optimization, and adaptive sampling to refine kernel representations.
• Empirical studies show that ARFF methods often outperform traditional RFF techniques in speed, accuracy, and interpretability across diverse applications.

Adaptive Random Fourier Features (ARFF) denotes a family of methods that makes random Fourier feature representations data-dependent rather than fixed after an initial spectral draw. In the standard Random Fourier Features (RFF) construction for a continuous, positive semidefinite, shift-invariant kernel, Bochner’s theorem yields

$$\phi(x)=\sqrt{\tfrac{2}{m}}[\cos(\omega_i^\top x+b_i)]_{i=1}^m,\qquad k(x-x')=\mathbb{E}_{\omega,b}[\phi(x)^\top \phi(x')],$$

with frequencies sampled from a prescribed spectral density and then held fixed. ARFF preserves this Fourier viewpoint, but adapts the spectral measure, the sampled frequencies, or the parameterized feature space to the task and data. In the literature this includes automatic relevance determination (ARD) kernels with learned per-dimension scales, end-to-end optimization of spectral points, data-dependent sampling by ridge leverage scores, and adaptive Metropolis or resampling schemes that target an empirically optimal spectral density (Otto et al., 2022, Dózsa et al., 29 Jan 2026).

1. Mathematical basis and scope

The unifying mathematical object behind ARFF is a finite feature map whose basis functions are Fourier atoms or real trigonometric equivalents. For shift-invariant kernels, the classical RFF approximation replaces kernel evaluation by inner products in a randomized feature space. Several later formulations reinterpret this as learning in a parameter-dependent solution space

$$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$

where the feature family itself depends on trainable parameters $\Lambda$. In this viewpoint, standard RFF is recovered when $\Lambda=[\omega_1,\dots,\omega_m]$ is sampled from the Bochner spectral distribution and then fixed, whereas ARFF treats $\Lambda$ as trainable and therefore turns kernel learning into a variable projection problem over both linear coefficients and nonlinear Fourier parameters (Dózsa et al., 29 Jan 2026).

A central distinction in the ARFF literature is that not every method adapts the same object. Some methods adapt kernel hyperparameters that control the spectral density; some adapt the spectral points directly; some keep the features random but change their sampling law; and some directly approximate a target function rather than approximate a kernel. The wind-field reconstruction model, for example, explicitly states that it does not approximate a kernel and instead fits a vector-valued Fourier expansion whose frequencies are adapted to the task-specific spectrum (Kiessling et al., 2021). This suggests that ARFF is better understood as a methodological family organized around adaptive spectral representations, rather than as a single algorithmic template.

2. Principal forms of adaptivity

A common ARFF mechanism is spectral-parameter adaptation. In “Learning Random Kernel Approximations for Object Recognition” (Băzăvan et al., 2012), the sampling density is parameterized as $p_\theta(\omega)$ and kernel parameters are learned by minimizing validation loss, with fixed base randomness transformed through quantile maps so that frequencies change continuously with $\theta$. In “RFFNet: Large-Scale Interpretable Kernel Methods via Random Fourier Features” (Otto et al., 2022), the adaptive parameters are ARD relevances or inverse lengthscales, yielding feature maps of the form

$$\phi(x;\theta)=\sqrt{\tfrac{2}{m}}[\cos(\omega_i^\top(\theta\circ x)+b_i)]_{i=1}^m,$$

with $\omega_i$ drawn once from the isotropic spectral density and $\theta$ learned by first-order optimization. In “Adaptive Random Fourier Features Kernel LMS” (Gao et al., 2022), the feature-generating frequencies and phases are adapted online in a Gaussian kernel LMS setting, effectively changing the operative kernel scale during streaming learning.

A second mechanism is direct learning of spectral points. “Deep Kernel Learning via Random Fourier Features” (Xie et al., 2019) treats the frequencies in each RFF layer as trainable parameters and stacks multiple cosine–sine layers so that each layer learns its own empirical spectral measure. “Adaptive Kernel Methods” (Dózsa et al., 29 Jan 2026) places this in a more general RKHS framework, where frequencies define a fixed-dimensional parameter-dependent solution space and the reduced objective can be optimized by variable projection. “Learning Landmark-Based Ensembles with Random Fourier Features and Gradient Boosting” (Gautheron et al., 2019) keeps sampled frequencies fixed within each boosting stage but learns a posterior distribution over them, together with a landmark, producing a data-adaptive barycenter of Fourier atoms.

A third mechanism is adaptive sampling of frequencies. “Adaptive random Fourier features with Metropolis sampling” (Kammonen et al., 2020) targets the optimal independent-sampling density

$$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$0

through an adaptive Metropolis procedure driven by fitted amplitudes. “Convergence for adaptive resampling of random Fourier features” (Huang et al., 3 Sep 2025) proves convergence for a resampling-based variant and identifies the asymptotically optimal distribution in the periodic setting by minimizing the rate constant in the generalization bound. “Towards A Unified Analysis of Random Fourier Features” (Li et al., 2018) studies a different data-dependent sampling strategy: frequencies drawn proportional to ridge leverage scores, yielding an ARFF variant that is adaptive to the dataset and regularization level rather than to an unknown Fourier transform directly.

The major forms are summarized below.

ARFF mode	Mechanism	Representative papers
Spectral-parameter learning	Learn bandwidths, ARD scales, or kernel hyperparameters inside the feature map	(Băzăvan et al., 2012, Otto et al., 2022, Gao et al., 2022)
Trainable spectral points	Optimize frequencies or parameter-dependent bases directly	(Xie et al., 2019, Dózsa et al., 29 Jan 2026, Gautheron et al., 2019)
Adaptive sampling	Change the sampling law by leverage scores, Metropolis, or resampling	(Li et al., 2018, Kammonen et al., 2020, Huang et al., 3 Sep 2025)

A common misconception is that “adaptive” always means learning a single kernel bandwidth. The literature is broader: ARFF may mean learning per-dimension ARD scales, learning a full set of frequencies, reweighting sampled frequencies by a posterior, or adaptively resampling from a nonparametric spectral law.

3. Objectives and optimization procedures

The simplest ARFF objectives retain a linear predictor on top of an adaptive feature map. RFFNet optimizes

$$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$1

with squared loss for regression and cross-entropy for classification, and in the default setting $$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$2 (Otto et al., 2022). The paper emphasizes that the problem is non-convex because the cosine argument is nonlinear in $$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$3 and the map is oscillatory, but shows that the block-gradients are Lipschitz continuous. Training uses Adam with mini-batches, a proximal step on $$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$4, early stopping on a validation split, and a range-based initialization

$$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$5

applied element-wise (Otto et al., 2022).

When the frequencies themselves are learned, the optimization is often entirely differentiable. The deep RFFNet model computes a linear transform followed by cosine and sine nonlinearities at each layer, and then updates the frequencies by backpropagation through the trigonometric activations. The layerwise derivatives are explicit; for example, for a feature $$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$6,

$$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$7

which makes the spectral points ordinary trainable parameters in an end-to-end model (Xie et al., 2019). In the adaptive-kernel formulation, square-loss ridge problems admit a closed-form inner solution

$$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$8

and the reduced objective $$f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),$$9 is differentiated either by implicit differentiation or by automatic differentiation through the linear solve (Dózsa et al., 29 Jan 2026).

Sampling-based ARFF uses a different optimization logic. In the Metropolis formulation, amplitudes are recomputed by ridge regression for the current frequencies,

$\Lambda$0

and proposed frequencies are accepted with probability

$\Lambda$1

so that the empirical spectral measure moves toward the optimal density (Kammonen et al., 2020). The resampling-stabilized variant introduces effective sample size,

$\Lambda$2

and triggers multinomial resampling of frequencies when degeneracy is detected; when resampling is used every iteration, the Metropolis test can be omitted (Kammonen et al., 2024). The recent convergence analysis replaces exact least-squares solves by conjugate gradient iterations inside a resampling-and-random-walk loop and proves asymptotic optimality under explicit assumptions on Fourier-coefficient decay, cutoff $\Lambda$3, and mixing proportion $\Lambda$4 (Huang et al., 3 Sep 2025).

4. Interpretability, relevance determination, and structured sparsity

One of the most distinctive ARFF directions treats adaptivity as a route to interpretable kernel learning. In RFFNet, the learned parameters $\Lambda$5 are per-dimension relevances or inverse lengthscales: small $\Lambda$6 indicate slow variation and possible irrelevance, whereas large $\Lambda$7 indicate rapid variation and strong relevance. Variable selection is then performed by a TopK hard-thresholding rule on the learned relevance vector, choosing the sparsity level by validation performance (Otto et al., 2022). The paper reports that increasing sample size improves true discovery rate and reduces false discovery rate, and that the learned $\Lambda$8 produce clear peaks on truly active variables in controlled simulations (Otto et al., 2022).

A related but structurally different approach is “ANOVA-boosting for Random Fourier Features” (Potts et al., 2024), which adapts over subsets of variables rather than only over individual coordinates. It constructs subset-wise RFF models, enforces hierarchical orthogonality or centering among ANOVA terms, and uses boosting-style pruning to identify important main effects and low-order interactions. The paper states that the algorithms are able to find an index set of important input variables and variable interactions reliably, and that the learned model is interpretable even for dependent input variables (Potts et al., 2024). This suggests that ARFF can encode sparsity not only in frequency magnitude, but also in the combinatorial structure of interactions.

Data-dependent sampling provides a third notion of interpretability. Ridge-leverage-score ARFF samples frequencies proportional to

$\Lambda$9

where $\Lambda=[\omega_1,\dots,\omega_m]$0 is the ridge leverage score and $\Lambda=[\omega_1,\dots,\omega_m]$1 the effective degrees of freedom (Li et al., 2018). Here the adaptive distribution is not directly a variable-importance measure, but it does quantify which spectral directions are statistically important for the regularized learning problem. In the Metropolis literature, the asymptotic equidistribution of amplitudes plays a similar role: the algorithm attempts to equalize amplitude magnitudes so that the empirical spectral allocation tracks $\Lambda=[\omega_1,\dots,\omega_m]$2 (Kammonen et al., 2020).

5. Empirical behavior and application domains

The empirical literature presents ARFF as a scalable alternative to both exact kernel methods and fixed-RFF baselines. RFFNet reports substantially lower MSE than isotropic RFF baselines, GP-ARD, SRFF, and others on simulated regression benchmarks, while identifying the truly active variables; on real-world regression datasets such as abalone, compact, powerplant, and yearprediction, it matches or outperforms baselines in most datasets (Otto et al., 2022). On real-world classification datasets including a9a, amazon, higgs, and w8a, the method is reported to excel on large-scale datasets and to attribute high relevance to scientifically meaningful features in HIGGS (Otto et al., 2022).

Deep trainable-frequency models show a different empirical profile. On small UCI datasets, the deep RFFNet reports “monks1 (d=6,n=124): SVM-RBF 81.5±0.0%; RFFNet 100.0±0.0%,” “monks2 (d=6,n=169): SVM-RBF 85.8±1.0%; RFFNet 98.0±0.7%,” and “climate (d=20,n=540): RFFNet 94.6±1.5% vs SVM-RBF 94.2±1.6%” (Xie et al., 2019). On larger datasets it reports “covtype (n=581012): RFFNet 96.6±0.2%; MLP 96.3±0.2%; SVM-RBF 80.0±0.1%,” and on image data without augmentation it reports “MNIST: RFFNet 99.1%” and “CIFAR-10: RFFNet 84.6%” (Xie et al., 2019).

Sampling-based ARFF has been evaluated on regression, classification, and scientific computing tasks. The resampling-stabilized shallow spectral network reports, for image regression on 92 DIV2K images, mean PSNR values of 25.49 for ARFF-initialized RFF layers, compared with 21.88 for a Glorot-initialized RFF layer, 22.53 for a three-layer ReLU network without an RFF layer, and 23.41 for a four-layer ReLU network without an RFF layer (Kammonen et al., 2024). The SDE learning application reports that ARFF matches or surpasses Adam-based optimization in both loss minimization and convergence speed across polynomial SDEs, underdamped Langevin dynamics, a stochastic SIR model, and a stochastic wave equation; for example, in Experiment 4a the reported minimum validation losses and times are “ARFF −16.102 in 11.380 s; Adam −14.031 in 58.231 s” (Douglas et al., 21 Jul 2025).

ARFF has also been applied to domain-specific inverse problems. In wind-field reconstruction, the adaptive Fourier model with Sobolev and divergence penalties reports unexplained variance $\Lambda=[\omega_1,\dots,\omega_m]$3, compared with $\Lambda=[\omega_1,\dots,\omega_m]$4 for Universal Kriging and $\Lambda=[\omega_1,\dots,\omega_m]$5 for inverse distance weighting; the paper further reports that averaging ARFF with Random Forest gives $\Lambda=[\omega_1,\dots,\omega_m]$6 (Kiessling et al., 2021). In online adaptive filtering, ARFF-GKLMS is reported to improve convergence rate, steady-state error, and tracking ability over kernel adaptive filters with preset bandwidth (Gao et al., 2022). In object recognition, gradient-based spectral adaptation and RFF-based group Lasso are described as scaling linearly in the number of training instances and enabling training on more than $\Lambda=[\omega_1,\dots,\omega_m]$7 segments in VOC2011 (Băzăvan et al., 2012).

6. Theoretical guarantees, limitations, and unsettled points

The theory of ARFF is heterogeneous because the methods are heterogeneous. Some results concern risk with data-dependent sampling. Ridge-leverage-score ARFF proves that, for kernel ridge regression, sampling according to leverage scores reduces the number of features needed to achieve the minimax rate relative to plain RFF, and provides a practical approximation scheme for the otherwise expensive leverage scores (Li et al., 2018). The adaptive resampling work proves convergence to the asymptotically optimal distribution and quantifies how the generalization constant approaches the optimum under explicit scaling of $\Lambda=[\omega_1,\dots,\omega_m]$8 and $\Lambda=[\omega_1,\dots,\omega_m]$9 (Huang et al., 3 Sep 2025). The Metropolis-based literature proves optimal independent-sampling density results and asymptotic equidistribution statements, but the resampling-stabilized extension explicitly states that no formal convergence proof is provided for the new resampling-stabilized algorithm (Kammonen et al., 2024).

Other results concern approximation and existence rather than sampling optimality. The adaptive-kernel framework proves deterministic kernel-approximation bounds for parameterized bases, existence of optimal adaptive solutions under continuity and compactness assumptions, and a variable-projection formulation that unifies trainable-frequency RFF with other adaptive kernel constructions (Dózsa et al., 29 Jan 2026). By contrast, deep stacked RFF models emphasize expressivity and empirical performance; they note that the paper does not derive a closed-form composite kernel for the full multilayer architecture (Xie et al., 2019).

Several limitations recur across the literature. Non-convexity is pervasive when frequencies, relevances, or landmarks are learned by gradient methods (Otto et al., 2022, Xie et al., 2019). Hyperparameter sensitivity remains substantial in adaptive sampling methods, especially for proposal scales, acceptance exponents, and batch sizes, even though resampling reduces this sensitivity (Kammonen et al., 2020, Kammonen et al., 2024). Exact leverage scores are expensive to compute and motivate approximation schemes (Li et al., 2018). Highly flexible parameterizations can overfit if regularization is insufficient (Dózsa et al., 29 Jan 2026). Bootstrap-based error estimation for RFF allows adaptive selection of the number of features and estimation of downstream task error, but its formal coverage theorem assumes i.i.d. feature columns and therefore does not cover structured or orthogonal random features (Yao et al., 2023).

The main conceptual controversy is therefore definitional rather than adversarial. In some papers ARFF means learning ARD hyperparameters inside a fixed random cosine layer; in others it means direct optimization of spectral points; in others it means adaptive resampling toward $\Lambda$0; and in still others it means data-dependent frequency sampling by ridge leverage scores. A plausible implication is that “ARFF” names a research program—adaptive spectral feature learning under Fourier parameterizations—rather than a unique estimator. Across these variants, the recurring objective is consistent: retain the computational advantages of explicit random features while replacing fixed, task-agnostic spectral sampling by data-adaptive mechanisms that improve approximation quality, interpretability, or sample efficiency.