---
title: Adaptive Random Fourier Features Overview
url: https://www.emergentmind.com/topics/adaptive-random-fourier-features-arff
type: topic
---

# Adaptive Random Fourier Features Overview

Adaptive Random Fourier Features (ARFF) denotes a family of methods that makes random Fourier feature representations data-dependent rather than fixed after an initial spectral draw. In the standard Random Fourier Features (RFF) construction for a continuous, positive semidefinite, shift-invariant kernel, Bochner’s theorem yields
\[
\phi(x)=\sqrt{\tfrac{2}{m}}[\cos(\omega_i^\top x+b_i)]_{i=1}^m,\qquad
k(x-x')=\mathbb{E}_{\omega,b}[\phi(x)^\top \phi(x')],
\]
with frequencies sampled from a prescribed spectral density and then held fixed. ARFF preserves this Fourier viewpoint, but adapts the spectral measure, the sampled frequencies, or the parameterized feature space to the task and data. In the literature this includes automatic relevance determination (ARD) kernels with learned per-dimension scales, end-to-end optimization of spectral points, data-dependent sampling by ridge leverage scores, and adaptive Metropolis or resampling schemes that target an empirically optimal spectral density [2211.06410], [2601.21707].

## 1. Mathematical basis and scope

The unifying mathematical object behind ARFF is a finite feature map whose basis functions are Fourier atoms or real trigonometric equivalents. For shift-invariant kernels, the classical RFF approximation replaces kernel evaluation by inner products in a randomized feature space. Several later formulations reinterpret this as learning in a parameter-dependent solution space
\[
f^\Lambda(x)=\langle \Phi^\Lambda(x), \bar w\rangle=\sum_{j=0}^{D-1} w_j \phi_j^\Lambda(x),
\]
where the feature family itself depends on trainable parameters $\Lambda$. In this viewpoint, standard RFF is recovered when $\Lambda=[\omega_1,\dots,\omega_m]$ is sampled from the Bochner spectral distribution and then fixed, whereas ARFF treats $\Lambda$ as trainable and therefore turns kernel learning into a variable projection problem over both linear coefficients and nonlinear Fourier parameters [2601.21707].

A central distinction in the ARFF literature is that not every method adapts the same object. Some methods adapt kernel hyperparameters that control the spectral density; some adapt the spectral points directly; some keep the features random but change their sampling law; and some directly approximate a target function rather than approximate a kernel. The wind-field reconstruction model, for example, explicitly states that it does not approximate a kernel and instead fits a vector-valued Fourier expansion whose frequencies are adapted to the task-specific spectrum [2102.02365]. This suggests that ARFF is better understood as a methodological family organized around adaptive spectral representations, rather than as a single algorithmic template.

## 2. Principal forms of adaptivity

A common ARFF mechanism is **spectral-parameter adaptation**. In “Learning Random Kernel Approximations for Object Recognition” [1203.1483], the sampling density is parameterized as $p_\theta(\omega)$ and kernel parameters are learned by minimizing validation loss, with fixed base randomness transformed through quantile maps so that frequencies change continuously with $\theta$. In “RFFNet: Large-Scale Interpretable Kernel Methods via Random Fourier Features” [2211.06410], the adaptive parameters are ARD relevances or inverse lengthscales, yielding feature maps of the form
\[
\phi(x;\theta)=\sqrt{\tfrac{2}{m}}[\cos(\omega_i^\top(\theta\circ x)+b_i)]_{i=1}^m,
\]
with $\omega_i$ drawn once from the isotropic spectral density and $\theta$ learned by first-order optimization. In “Adaptive Random Fourier Features Kernel LMS” [2207.07236], the feature-generating frequencies and phases are adapted online in a Gaussian kernel LMS setting, effectively changing the operative kernel scale during streaming learning.

A second mechanism is **direct learning of spectral points**. “Deep Kernel Learning via Random Fourier Features” [1910.02660] treats the frequencies in each RFF layer as trainable parameters and stacks multiple cosine–sine layers so that each layer learns its own empirical spectral measure. “Adaptive Kernel Methods” [2601.21707] places this in a more general RKHS framework, where frequencies define a fixed-dimensional parameter-dependent solution space and the reduced objective can be optimized by variable projection. “Learning Landmark-Based Ensembles with Random Fourier Features and Gradient Boosting” [1906.06203] keeps sampled frequencies fixed within each boosting stage but learns a posterior distribution over them, together with a landmark, producing a data-adaptive barycenter of Fourier atoms.

A third mechanism is **adaptive sampling of frequencies**. “Adaptive random Fourier features with Metropolis sampling” [2007.10683] targets the optimal independent-sampling density
\[
p_*(\omega)=\frac{|\hat f(\omega)|}{\int_{\mathbb{R}^d}|\hat f(\omega')|\,d\omega'}
\]
through an adaptive Metropolis procedure driven by fitted amplitudes. “Convergence for adaptive resampling of random Fourier features” [2509.03151] proves convergence for a resampling-based variant and identifies the asymptotically optimal distribution in the periodic setting by minimizing the rate constant in the generalization bound. “Towards A Unified Analysis of Random Fourier Features” [1806.09178] studies a different data-dependent sampling strategy: frequencies drawn proportional to ridge leverage scores, yielding an ARFF variant that is adaptive to the dataset and regularization level rather than to an unknown Fourier transform directly.

The major forms are summarized below.

| ARFF mode | Mechanism | Representative papers |
|---|---|---|
| Spectral-parameter learning | Learn bandwidths, ARD scales, or kernel hyperparameters inside the feature map | [1203.1483], [2211.06410], [2207.07236] |
| Trainable spectral points | Optimize frequencies or parameter-dependent bases directly | [1910.02660], [2601.21707], [1906.06203] |
| Adaptive sampling | Change the sampling law by leverage scores, Metropolis, or resampling | [1806.09178], [2007.10683], [2509.03151] |

A common misconception is that “adaptive” always means learning a single kernel bandwidth. The literature is broader: ARFF may mean learning per-dimension ARD scales, learning a full set of frequencies, reweighting sampled frequencies by a posterior, or adaptively resampling from a nonparametric spectral law.

## 3. Objectives and optimization procedures

The simplest ARFF objectives retain a linear predictor on top of an adaptive feature map. RFFNet optimizes
\[
\min_{w,\theta}\;\frac{1}{n}\sum_{i=1}^n L\bigl(y_i,w^\top\phi(x_i;\theta)\bigr)+\lambda\|w\|_2^2+R(\theta),
\]
with squared loss for regression and cross-entropy for classification, and in the default setting $R(\theta)=0$ [2211.06410]. The paper emphasizes that the problem is non-convex because the cosine argument is nonlinear in $\theta$ and the map is oscillatory, but shows that the block-gradients are Lipschitz continuous. Training uses Adam with mini-batches, a proximal step on $w$, early stopping on a validation split, and a range-based initialization
\[
\theta_{\text{init}}=\frac{1}{d}\Bigl(\max_{i\in[n]}x_i-\min_{i\in[n]}x_i\Bigr)
\]
applied element-wise [2211.06410].

When the frequencies themselves are learned, the optimization is often entirely differentiable. The deep RFFNet model computes a linear transform followed by cosine and sine nonlinearities at each layer, and then updates the frequencies by backpropagation through the trigonometric activations. The layerwise derivatives are explicit; for example, for a feature $s_\ell^m=\sqrt{1/D_\ell}\cos((\omega_m^{(\ell)})^\top s_{\ell-1})$,
\[
\frac{\partial s_{\ell}^m}{\partial \omega_m^{(\ell)}}=
-\sqrt{\tfrac{1}{D_\ell}}\,s_{\ell-1}\sin((\omega_m^{(\ell)})^\top s_{\ell-1}),
\]
which makes the spectral points ordinary trainable parameters in an end-to-end model [1910.02660]. In the adaptive-kernel formulation, square-loss ridge problems admit a closed-form inner solution
\[
\alpha^*(\theta)=\bigl(\Phi_\theta^\top\Phi_\theta+\lambda I\bigr)^{-1}\Phi_\theta^\top y,
\]
and the reduced objective $J(\theta)$ is differentiated either by implicit differentiation or by automatic differentiation through the linear solve [2601.21707].

Sampling-based ARFF uses a different optimization logic. In the Metropolis formulation, amplitudes are recomputed by ridge regression for the current frequencies,
\[
\hat\beta=(\Phi^*\Phi+\lambda N I)^{-1}\Phi^*y,
\]
and proposed frequencies are accepted with probability
\[
a_k=\min\{1,(|\hat\beta'_k|/|\hat\beta_k|)^\gamma\},
\]
so that the empirical spectral measure moves toward the optimal density [2007.10683]. The resampling-stabilized variant introduces effective sample size,
\[
K_{\mathrm{ESS}}=1/\sum_{k=1}^K \check p_k^2,\qquad
\check p_k=\frac{|a_k|}{\sum_{j=1}^K |a_j|},
\]
and triggers multinomial resampling of frequencies when degeneracy is detected; when resampling is used every iteration, the Metropolis test can be omitted [2410.06399]. The recent convergence analysis replaces exact least-squares solves by conjugate gradient iterations inside a resampling-and-random-walk loop and proves asymptotic optimality under explicit assumptions on Fourier-coefficient decay, cutoff $\epsilon$, and mixing proportion $q_\epsilon$ [2509.03151].

## 4. Interpretability, relevance determination, and structured sparsity

One of the most distinctive ARFF directions treats adaptivity as a route to interpretable kernel learning. In RFFNet, the learned parameters $\theta_j$ are per-dimension relevances or inverse lengthscales: small $\theta_j$ indicate slow variation and possible irrelevance, whereas large $\theta_j$ indicate rapid variation and strong relevance. Variable selection is then performed by a TopK hard-thresholding rule on the learned relevance vector, choosing the sparsity level by validation performance [2211.06410]. The paper reports that increasing sample size improves true discovery rate and reduces false discovery rate, and that the learned $\theta$ produce clear peaks on truly active variables in controlled simulations [2211.06410].

A related but structurally different approach is “ANOVA-boosting for Random Fourier Features” [2404.03050], which adapts over subsets of variables rather than only over individual coordinates. It constructs subset-wise RFF models, enforces hierarchical orthogonality or centering among ANOVA terms, and uses boosting-style pruning to identify important main effects and low-order interactions. The paper states that the algorithms are able to find an index set of important input variables and variable interactions reliably, and that the learned model is interpretable even for dependent input variables [2404.03050]. This suggests that ARFF can encode sparsity not only in frequency magnitude, but also in the combinatorial structure of interactions.

Data-dependent sampling provides a third notion of interpretability. Ridge-leverage-score ARFF samples frequencies proportional to
\[
q^*(v)=\frac{l_\lambda(v)}{d_{\mathbf{K}}^\lambda},
\]
where $l_\lambda(v)$ is the ridge leverage score and $d_{\mathbf{K}}^\lambda$ the effective degrees of freedom [1806.09178]. Here the adaptive distribution is not directly a variable-importance measure, but it does quantify which spectral directions are statistically important for the regularized learning problem. In the Metropolis literature, the asymptotic equidistribution of amplitudes plays a similar role: the algorithm attempts to equalize amplitude magnitudes so that the empirical spectral allocation tracks $p_*(\omega)\propto |\hat f(\omega)|$ [2007.10683].

## 5. Empirical behavior and application domains

The empirical literature presents ARFF as a scalable alternative to both exact kernel methods and fixed-RFF baselines. RFFNet reports substantially lower MSE than isotropic RFF baselines, GP-ARD, SRFF, and others on simulated regression benchmarks, while identifying the truly active variables; on real-world regression datasets such as abalone, compact, powerplant, and yearprediction, it matches or outperforms baselines in most datasets [2211.06410]. On real-world classification datasets including a9a, amazon, higgs, and w8a, the method is reported to excel on large-scale datasets and to attribute high relevance to scientifically meaningful features in HIGGS [2211.06410].

Deep trainable-frequency models show a different empirical profile. On small UCI datasets, the deep RFFNet reports “monks1 (d=6,n=124): SVM-RBF 81.5±0.0%; RFFNet 100.0±0.0%,” “monks2 (d=6,n=169): SVM-RBF 85.8±1.0%; RFFNet 98.0±0.7%,” and “climate (d=20,n=540): RFFNet 94.6±1.5% vs SVM-RBF 94.2±1.6%” [1910.02660]. On larger datasets it reports “covtype (n=581012): RFFNet 96.6±0.2%; MLP 96.3±0.2%; SVM-RBF 80.0±0.1%,” and on image data without augmentation it reports “MNIST: RFFNet 99.1%” and “CIFAR-10: RFFNet 84.6%” [1910.02660].

Sampling-based ARFF has been evaluated on regression, classification, and scientific computing tasks. The resampling-stabilized shallow spectral network reports, for image regression on 92 DIV2K images, mean PSNR values of 25.49 for ARFF-initialized RFF layers, compared with 21.88 for a Glorot-initialized RFF layer, 22.53 for a three-layer ReLU network without an RFF layer, and 23.41 for a four-layer ReLU network without an RFF layer [2410.06399]. The SDE learning application reports that ARFF matches or surpasses Adam-based optimization in both loss minimization and convergence speed across polynomial SDEs, underdamped Langevin dynamics, a stochastic SIR model, and a stochastic wave equation; for example, in Experiment 4a the reported minimum validation losses and times are “ARFF −16.102 in 11.380 s; Adam −14.031 in 58.231 s” [2507.15442].

ARFF has also been applied to domain-specific inverse problems. In wind-field reconstruction, the adaptive Fourier model with Sobolev and divergence penalties reports unexplained variance $\tilde E\approx 0.370\pm 0.012$, compared with $0.388\pm 0.013$ for Universal Kriging and $0.407\pm 0.014$ for inverse distance weighting; the paper further reports that averaging ARFF with Random Forest gives $0.357\pm 0.012$ [2102.02365]. In online adaptive filtering, ARFF-GKLMS is reported to improve convergence rate, steady-state error, and tracking ability over kernel adaptive filters with preset bandwidth [2207.07236]. In object recognition, gradient-based spectral adaptation and RFF-based group Lasso are described as scaling linearly in the number of training instances and enabling training on more than $10^5$ segments in VOC2011 [1203.1483].

## 6. Theoretical guarantees, limitations, and unsettled points

The theory of ARFF is heterogeneous because the methods are heterogeneous. Some results concern risk with data-dependent sampling. Ridge-leverage-score ARFF proves that, for kernel ridge regression, sampling according to leverage scores reduces the number of features needed to achieve the minimax rate relative to plain RFF, and provides a practical approximation scheme for the otherwise expensive leverage scores [1806.09178]. The adaptive resampling work proves convergence to the asymptotically optimal distribution and quantifies how the generalization constant approaches the optimum under explicit scaling of $\epsilon$ and $q_\epsilon$ [2509.03151]. The Metropolis-based literature proves optimal independent-sampling density results and asymptotic equidistribution statements, but the resampling-stabilized extension explicitly states that no formal convergence proof is provided for the new resampling-stabilized algorithm [2410.06399].

Other results concern approximation and existence rather than sampling optimality. The adaptive-kernel framework proves deterministic kernel-approximation bounds for parameterized bases, existence of optimal adaptive solutions under continuity and compactness assumptions, and a variable-projection formulation that unifies trainable-frequency RFF with other adaptive kernel constructions [2601.21707]. By contrast, deep stacked RFF models emphasize expressivity and empirical performance; they note that the paper does not derive a closed-form composite kernel for the full multilayer architecture [1910.02660].

Several limitations recur across the literature. Non-convexity is pervasive when frequencies, relevances, or landmarks are learned by gradient methods [2211.06410], [1910.02660]. Hyperparameter sensitivity remains substantial in adaptive sampling methods, especially for proposal scales, acceptance exponents, and batch sizes, even though resampling reduces this sensitivity [2007.10683], [2410.06399]. Exact leverage scores are expensive to compute and motivate approximation schemes [1806.09178]. Highly flexible parameterizations can overfit if regularization is insufficient [2601.21707]. Bootstrap-based error estimation for RFF allows adaptive selection of the number of features and estimation of downstream task error, but its formal coverage theorem assumes i.i.d. feature columns and therefore does not cover structured or orthogonal random features [2302.11174].

The main conceptual controversy is therefore definitional rather than adversarial. In some papers ARFF means learning ARD hyperparameters inside a fixed random cosine layer; in others it means direct optimization of spectral points; in others it means adaptive resampling toward $p_*(\omega)$; and in still others it means data-dependent frequency sampling by ridge leverage scores. A plausible implication is that “ARFF” names a research program—adaptive spectral feature learning under Fourier parameterizations—rather than a unique estimator. Across these variants, the recurring objective is consistent: retain the computational advantages of explicit random features while replacing fixed, task-agnostic spectral sampling by data-adaptive mechanisms that improve approximation quality, interpretability, or sample efficiency.

Source: https://www.emergentmind.com/topics/adaptive-random-fourier-features-arff