Multiscale Random Fourier Feature Bank

• Multiscale Random Fourier Feature Banks are collections of Fourier features tuned to distinct frequency scales, enabling efficient representation of both low- and high-frequency components.
• They employ layered, dyadic, and concatenated architectures with learnable parameters and attention-based weighting to overcome spectral bias and enhance convergence.
• These feature banks drive state-of-the-art performance in tasks such as kernel learning, neural field reconstruction, time-series forecasting, and numerical PDE solving.

A Multiscale Random Fourier Feature Bank is an architectural paradigm that constructs, manipulates, and utilizes collections (“banks”) of random Fourier features with multiple, explicitly separated frequency scales. These multiscale banks provide superior expressiveness and rapid convergence for nonlinear function approximation, regression, and dynamical modeling, with applications spanning kernel learning, neural field reconstructions, time-series forecasting, and numerical PDEs. By stacking or aggregating features that focus on different spectral bands and optionally coupling them to attention or residual processing, such banks can overcome spectral bias and represent both low- and high-frequency components efficiently. This approach underpins several state-of-the-art neural, kernel, and reservoir architectures and is applicable in any regime where the signal exhibits multi-band or fast-slow dynamics.

1. Mathematical Foundations and Random Fourier Features

Random Fourier Features (RFFs) approximate shift-invariant kernels by Bochner’s theorem: any continuous, positive-definite kernel $k(x, y) = k(x-y)$ can be expressed as

$$k(x-y) = \int_{\mathbb{R}^d} p(\omega) e^{j\omega^\top(x-y)}d\omega = \mathbb{E}_{\omega \sim p} [e^{j\omega^\top x}e^{-j\omega^\top y}].$$

Discretizing this yields the empirical feature map

$$\phi(x) = \sqrt{\frac{2}{P}}\left[\cos(\omega_1^\top x + b_1),\ldots, \cos(\omega_P^\top x + b_P)\right],$$

with frequencies $\omega_i \sim p(\omega)$, phases $b_i \sim \mathrm{Uniform}[0, 2\pi]$ (Xie et al., 2019, Laha, 4 Nov 2025, Davis et al., 16 Jul 2024, Feng et al., 21 Dec 2025). Increasing $P$ improves the fidelity of the kernel approximation. When distinct spectral distributions $p(\omega\vert\theta_\ell)$ are used for different “layers” $\ell$, composite feature banks can specialize to specific frequency bands, capturing a multi-band or multiscale representation.

2. Construction of Multiscale Feature Banks

A multiscale random Fourier bank comprises multiple subsets of Fourier features, each subset parameterized to concentrate on a distinct frequency scale. The principal strategies to construct such banks are:

• Layered approach: Stack $L$ feature maps $\phi_\ell$ associated with frequency distributions $p(\omega\vert\theta_\ell)$, often Gaussians with bandwidth $\sigma_\ell$, to span scales from coarse (large $\sigma$) to fine (small $\sigma$).
• Dyadic scaling: Apply geometric scaling to base frequencies, e.g., for dyadic scales $k=0,\dots,K$, set $\tilde\omega_{m, k} = 2^k\omega_m$ where $\omega_m$ is drawn from a base normal.
• Bank concatenation: Concatenate feature vectors for all scales, yielding

$$\Phi(x) = [\phi_{\gamma_1}(x); \ldots ; \phi_{\gamma_L}(x)] \in \mathbb{R}^{D L},$$

approximating a sum of kernels at different bandwidths,

$$K_{\text{multi}}(x, x') \approx \sum_{\ell = 1}^L k_{\gamma_\ell}(x, x').$$

Each scale can be further modulated with learnable amplitude envelopes $a_{m, k}$, such as $a_{m, k} = \exp(-\beta\|\tilde\omega_{m, k}\|_2)$, where $\beta$ is learnable (Feng et al., 21 Dec 2025).

A summary of multiscale bank construction methods appears below:

Method	Frequency Parametrization	Feature Combination
Layered	$p(\omega|\sigma_\ell)$	$h_{\ell} = \phi_\ell(h_{\ell-1})$ (Xie et al., 2019)
Dyadic/Geometric	$\tilde\omega_{m,k} = 2^k\omega_m$	Concatenation or tokenization (Feng et al., 21 Dec 2025)
Empirical/Optimal	$p^*(\omega) \propto |\hat{Q}(\omega)|$	Blockwise residual fitting (Davis et al., 16 Jul 2024)

3. Learning and Adaptivity in Multiscale Banks

Multiscale banks provide explicit adaptivity both through learnable parameters (amplitudes, spectral bandwidths) and compositional training schemes:

• End-to-end optimization: Train multi-layer RFF nets with differentiable spectral parameters using stochastic gradient descent. This enables the network to shift its frequency allocation according to signal content (Xie et al., 2019).
• Residual block refinement: Deep random Fourier neural nets (rFNNs) fit residuals at each block, with each block’s frequencies sampled optimally from the current spectrum of the error (Davis et al., 16 Jul 2024).
• Attention-based weighting: Cross-attention modules reweight the contribution of different frequency scales conditional on input, enabling the network to attend to informative bands as needed and dynamically adapt the effective spectrum (Feng et al., 21 Dec 2025).
• Spectral enrichment: Dominant frequencies extracted from prior approximations, via discrete Fourier analysis, can be injected as additional tokens without altering backbone architecture, further refining the bank to newly emergent spectral content (Feng et al., 21 Dec 2025).

4. Architectures and Applications

Multiscale random Fourier banks are integral to several advanced architectures:

• Neural Fourier Filter Bank (NFFB): Couples multi-resolution spatial grids to per-scale trainable Fourier encodings; each scale injects Fourier-encoded features at specific layers of a sine-activated MLP, mirroring a wavelet filter-bank. Output assembly is a residual sum across all levels, ensuring ordered spectral decomposition from coarse to fine (Wu et al., 2022).
• Deep RFF networks: Stack multiple RFF layers, each with its own bandwidth, forming nonlinear kernel cascades with pointwise spectral product, akin to deep kernel learning (Xie et al., 2019).
• Reservoir Computing: Multi-scale RFF banks serve as static, high-dimensional feature maps within echo state-style reservoirs, providing temporal memory across multiple intrinsic timescales for fast-slow dynamical systems (Laha, 4 Nov 2025).
• Random Fourier Neural Networks (rFNN): Deep residual networks where each block employs MCMC-optimized frequency sampling tailored to the current residual, with rigorous $O(1/(WL))$ error rates and empirically demonstrable avoidance of Gibbs phenomena even on discontinuous targets (Davis et al., 16 Jul 2024).
• Cross-Attention Networks: Feed-forward and residual networks where RFF tokens are reweighted by attention mechanisms, accelerating high-frequency convergence and supporting incremental spectrum enrichment; architectures can further combine low- and high-frequency expert subnetworks for efficient PDE regression (Feng et al., 21 Dec 2025).

5. Empirical Properties and Spectral Bias Mitigation

Experiments across multiple domains validate the effectiveness of multiscale feature banks:

• Convergence acceleration: Multiscale banks decouple the frequency domains; low bands are addressed in early layers/blocks, while high-frequency detail is captured later or in fine-scale subnets, yielding faster empirical convergence and reduced model size for fixed accuracy (Wu et al., 2022, Feng et al., 21 Dec 2025).
• Spectral bias suppression: Global RFF or naive MLP architectures are typically spectrally biased, converging much more rapidly to low-frequency content. Multiscale banks mitigate this with learnable scaling, attention-based reweighting, and explicit high-frequency branches, leading to accelerated capture of oscillatory and discontinuous structure (Feng et al., 21 Dec 2025).
• Interpretability: Amplitude- or MCMC-based frequency selection enables inspection of actual frequency usage and spectral specialization per block; empirical studies reveal diversity and adaptation of the distribution of active frequencies even in small-data regimes (Xie et al., 2019, Davis et al., 16 Jul 2024).
• Stability on discontinuities: Residual or blockwise local adaptation of frequencies, as in rFNNs, empirically eliminates the classical Gibbs phenomena—overshoot at sharp jumps—through local, layerwise approximation rather than global trigonometric bases (Davis et al., 16 Jul 2024).

6. Representative Algorithms, Hyperparameters, and Implementation Guidelines

Canonical instantiations of multiscale RFF banks vary by application but typically involve:

• Number of scales $L$: Chosen to match the number of physically or statistically relevant bands (e.g., two for fast/slow, three for slow/burst/spike dynamics (Laha, 4 Nov 2025)).
• Features per scale $D$: $512$–$1024$ per scale is standard; total feature dimension $M = L D$.
• Bandwidths $\{\gamma_\ell\}$: Select logarithmically spaced values to span the relevant frequency intervals, or optimize adaptively through training.
• Training: Use Adam or similar optimizers (when applicable); special-case blockwise MCMC for amplitude and frequency updates in rFNNs (Davis et al., 16 Jul 2024).
• Regularization: $\ell_2$ penalties on weights and spectral parameters as appropriate.
• Data batching and projection: Concatenate feature vectors before passing to downstream classifiers, regressors, or readouts; in reservoir settings, ridge regression readouts are closed-form (Laha, 4 Nov 2025).
• Attention tokenization: For cross-attention designs, RFFs are grouped into tokens (e.g., by scale) and attention is computed over these tokens (Feng et al., 21 Dec 2025).

7. Comparative Evaluation and Theoretical Guarantees

Quantitative empirical findings consistently demonstrate superiority of multiscale banks over single-scale or global Fourier feature methods:

• NFFB outperforms grid-only and Fourier-only baselines on 2D/3D reconstruction (e.g., higher PSNR/SSIM for images, lower Chamfer error for shape reconstruction) at 10× smaller memory budget (Wu et al., 2022).
• In time-series modeling for fast-slow systems, normalized RMSE decreases by orders of magnitude when employing multi-scale versus single-scale RFF reservoirs (Laha, 4 Nov 2025).
• Theoretical approximation errors achieve $O(1/(W L))$ rates for deep rFNNs with sample-efficient scaling (Davis et al., 16 Jul 2024).
• Cross-attention augmented RFF architectures close the convergence gap for high-frequency targets and enable plug-in spectral enrichment without architectural redesign (Feng et al., 21 Dec 2025).

A plausible implication is that explicit multiscale decomposition qualitatively transforms training dynamics and representational efficiency for both data-driven and physics-informed modeling tasks engaging multi-band structure.

---

References:

• Neural Fourier Filter Bank (NFFB) (Wu et al., 2022)
• Deep Kernel Learning via Random Fourier Features (Xie et al., 2019)
• Reservoir Computing via Multi-Scale Random Fourier Features (Laha, 4 Nov 2025)
• Deep Learning without Global Optimization by Random Fourier Neural Networks (Davis et al., 16 Jul 2024)
• Overcoming Spectral Bias via Cross-Attention (Feng et al., 21 Dec 2025)