Trainable Feedforward Kernel Networks

• Trainable Feedforward Kernel Networks are architectures that replace fixed nonlinear activations with parametrized kernel maps to learn data-adaptive representations.
• They employ variants like linear self-attention, SNNK, and NKNs to overcome the quadratic scaling of traditional kernel methods while ensuring universal approximation.
• TFKNs offer improved computational efficiency and reduced parameter counts, enabling scalable, robust performance on structured data tasks.

A trainable feedforward kernel network (TFKN) is a multi-layer computational architecture in which the central operation at each layer is a parametrized, trainable kernel map, rather than a fixed nonlinear activation or random-feature mapping. TFKNs aim to bridge neural and kernel methods by allowing end-to-end learning of kernel feature maps—or entire kernels—within a feedforward network, eliminating the classic quadratic scaling or rigidity of traditional kernel machines while potentially achieving universal approximation, improved efficiency, and strong performance on structured data. Prominent architectures include those that leverage trainable kernel approximations for linear-time self-attention, scalable neural network kernel (SNNK) replacements for dense and convolutional layers, adaptive kernel learning in the infinite-width limit, and layer-wise greedy training with explicit kernel objectives.

1. Rationale and Theoretical Motivation

Trainable feedforward kernel networks arise from the intersection of two traditions: the mathematical foundation of positive-definite kernels (RKHS theory) and the design desiderata of modern deep neural networks. Compared to fixed-kernel approaches (e.g., random Fourier features, NTK/NNGP), a TFKN maps the learning of kernel representations themselves—either via explicit parameterization (e.g., as a small feedforward network) or through joint optimization within the network—to an end-to-end trainable task. This flexibility accommodates data-adaptive geometric structure, nontrivial compositionality, and reductions in both compute and parameter count relative to naive kernel expansions or dense layers.

The universal approximation theorem applies: any continuous, positive-definite kernel can be captured by a sufficiently well-parameterized feedforward architecture. In practice, even shallow, positively-constrained parametrizations suffice to mimic the geometry of exponential or arc-cosine kernels underlying softmax attention or ReLU networks, respectively (Yorsh et al., 2022, Sehanobish et al., 2023).

2. Model Architectures and Parameterizations

TFKN design admits several forms, united by the principle of replacing classic neural layer operations with parametrized, trainable kernel maps:

• Linear Self-Attention with Trainable φ: Replace the softmax attention’s exponential kernel $\kappa(q, k) = \exp(q^T k)$ with a trainable map $\phi(\cdot)$ such that $\kappa(q, k) \approx \phi(q)^T \phi(k)$. Here, $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^C$ is implemented as a one- or multi-layer feedforward network with strictly positive outputs. Common choices are single-layer Softplus ($\phi(x) = \mathrm{Softplus}(xW)$) or multi-layer GLU-based kernels. All parameters (projection matrices, gates) are trained jointly with the rest of the model (Yorsh et al., 2022).
• Scalable Neural Network Kernels (SNNKs): General feedforward or convolutional layers are decomposed into “input towers” $\phi_f(x)$ and "parameter towers" $\psi_f(w, b)$ so that the output is a vector of dot-product or general kernel evaluations $K_f(x, (w, b)) = \phi_f(x)^T \psi_f(w, b)$, stacking one per output unit. SNNK layers can be used to compactify entire networks. The dimension $m$ is a tunable hyperparameter dictating the compression factor; universal random features (URF) instantiate $\phi$ and $\phi(\cdot)$0 for a range of nonlinearities (Sehanobish et al., 2023).
• Feedforward Kernel Networks for Gaussian Processes (Neural Kernel Networks, NKNs): Kernels themselves are composed via trainable sum-and-product layers, with the weights and base kernels' parameters fully optimized by gradient-based methods. Each layer's operation corresponds to a valid kernel (via closure under sum/product with nonnegative weights), yielding a differentiable architecture implementable end-to-end for compositional kernel learning (Sun et al., 2018).
• Parameter-Varying Control and Kernel Regression: For LPV systems, TFKNs parameterize feedforward control laws as basis expansions with coefficients $\phi(\cdot)$1 modeled in an RKHS. The kernel $\phi(\cdot)$2 is selected and tuned according to application-specific smoothness or periodicity priors; the solution is computed via kernel ridge regression and iteratively refined by ILC-like schemes (Haren et al., 28 Feb 2025, Haren et al., 2023).

3. Training Methodologies

TFKNs are optimized using both standard backpropagation and specific layer-wise or closed-form procedures, depending on the architecture:

• End-to-End Gradient Descent: Networks embedding kernel parametrizations (e.g., self-attention φ, SNNK layers, NKNs) are trained via SGD or Adam, with gradients flowing through φ/ψ parameters, matrix weights, and any auxiliary gates. The objective is typically cross-entropy (for classification) or mean squared error (for regression) (Yorsh et al., 2022, Sehanobish et al., 2023, Sun et al., 2018).
• Closed-Form and Layerwise Updates: In architectures such as SNNKs or KNet, certain regimes (linear loss, fully bundled SNNK, or spectral methods) admit closed-form solutions for all or part of the network parameters, avoiding backpropagation. Layerwise greedy training—with explicit hidden-target construction—enables rapid convergence and interpretability but may not achieve the global optimum for non-convex objectives (Sehanobish et al., 2023, Wu et al., 2020, Duan et al., 2018).
• Hybrid Iterative Algorithms: For kernel-based LPV feedforward, each update step involves solving a regularized linear system with kernel matrices constructed from scheduling signals, with performance monitored and hyperparameters (e.g., bandwidth, regularization) selected via marginal-likelihood maximization or cross-validation (Haren et al., 28 Feb 2025, Haren et al., 2023).

4. Computational and Sample Complexity

The computational efficiency of TFKNs is driven by the structure of the chosen kernel parametrization:

Architecture	Complexity per Layer	Key Memory Scalings
Linear Self-Attention TFKN (Yorsh et al., 2022)	$\phi(\cdot)$3 per layer (linear in sequence length for $\phi(\cdot)$4)	$\phi(\cdot)$5, nearly constant w.r.t. $\phi(\cdot)$6
SNNK (general layer) (Sehanobish et al., 2023)	$\phi(\cdot)$7, $\phi(\cdot)$8	$\phi(\cdot)$9
Kernel ridge regression LPV feedforward (Haren et al., 28 Feb 2025)	$\kappa(q, k) \approx \phi(q)^T \phi(k)$0; reduced via low-rank/Nyström	$\kappa(q, k) \approx \phi(q)^T \phi(k)$1
Greedy spectral/closed-form (KNet) (Wu et al., 2020)	~$\kappa(q, k) \approx \phi(q)^T \phi(k)$2 per layer (dominated by kernel matrix)	$\kappa(q, k) \approx \phi(q)^T \phi(k)$3

Trainable, low-dimensional kernel approximations (e.g., $\kappa(q, k) \approx \phi(q)^T \phi(k)$4 with $\kappa(q, k) \approx \phi(q)^T \phi(k)$5) allow TFKNs to scale to large sequence lengths or input dimensions, circumventing the quadratic bottleneck of classical full-kernel methods. In empirical studies, linear attention TFKNs provide up to an order-of-magnitude throughput increase without significant loss of accuracy (Yorsh et al., 2022). Layerwise or closed-form updates can bring down time-to-convergence by orders of magnitude over backpropagation-based schemes, especially for small to moderate datasets (Wu et al., 2020, Duan et al., 2018).

5. Empirical Results and Expressivity

Experimental studies across diverse benchmarks demonstrate the practical capacity of TFKNs:

• Self-Attention Linearization: Trainable kernel $\kappa(q, k) \approx \phi(q)^T \phi(k)$6 maps (Softplus, GLU, OGLU) consistently match or exceed quadratic attention baselines on Long Range Arena classification/matching tasks, even with $\kappa(q, k) \approx \phi(q)^T \phi(k)$7. Linear variants close the gap to best efficient Transformer models, though tasks requiring inherently non-linear composition (e.g., ListOps) remain challenging (Yorsh et al., 2022).
• SNNKs in Deep Architectures: SNNK replacements for dense and adapter layers in Transformers and vision networks provide $\kappa(q, k) \approx \phi(q)^T \phi(k)$8–$\kappa(q, k) \approx \phi(q)^T \phi(k)$9 parameter reduction with negligible accuracy drop and sometimes improved generalization. Bundling and closed-form optimization confer substantial memory and training gains (Sehanobish et al., 2023).
• Automatic Kernel Discovery: NKNs (compositional kernel feedforward networks) learn interpretable multicomponent structure for time-series, regression, and texture completion, outperforming fixed or random-feature kernels and matching the structure search of systems like the Automatic Statistician at a fraction of the training time (Sun et al., 2018).
• LPV Feedforward Control: Kernel-regularized TFKNs adaptively fit nonparametric feedforward controllers that outperform fixed LTI methods, with significant reductions in RMS and peak tracking errors as well as spectral harmonics in challenging industrial systems (Haren et al., 28 Feb 2025).
• Expressivity: SNNKs can represent strictly more expressive function classes than standard feedforward layers (e.g., the first-order arc-cosine kernel in ReLU-SNNKs cannot be written as a scalar activation on $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^C$0). Layerwise kernel networks, when trained greedily, offer competitive or improved test error versus standard MLPs, SVMs, and multiple kernel learning baselines on UCI and vision tasks (Sehanobish et al., 2023, Duan et al., 2018).

6. Design Considerations and Limitations

Designing an effective TFKN requires careful selection and tuning of kernel family, hyperparameters (projection dimension $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^C$1 or $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^C$2, kernel bandwidth, regularization), and parametrization strategy. Key considerations include:

• Choice and Depth of φ/ψ: Shallow φ (e.g., single Softplus) often suffices for kernels mimicking softmax; deeper/gated layers marginally improve expressivity at a parameter cost, with diminishing returns after two GLU or OGLU layers (Yorsh et al., 2022).
• Positivity and Stability: For attention approximation, all feature maps must output strictly positive values to avoid denominator singularities; activation and orthogonal initialization/regularization are crucial for stability (Yorsh et al., 2022).
• Kernel Approximation Error: In SNNK and random feature schemes, the error per layer decays as $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^C$3, but compounded approximation error may necessitate higher $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^C$4 in deeper (bundled) networks (Sehanobish et al., 2023).
• Closed-form vs. End-to-End: Full closed-form solutions are possible for regression loss, but not for cross-entropy or tasks requiring non-linear separability, where gradient-based optimization remains essential (Sehanobish et al., 2023).
• Memory and Compute Constraints: Kernel matrices scale quadratically, but low-rank, random-feature, and mini-batch approximations—plus block-diagonal structure in control applications—enable tractability for moderate dataset sizes (Haren et al., 28 Feb 2025, Wu et al., 2020).
• Task-specificity: Certain problem structures (e.g., periodicity, smooth scheduling variables, or highly compositional time series) benefit particularly from adaptive or compositional kernel architectures (Haren et al., 28 Feb 2025, Sun et al., 2018). ListOps on LRA demonstrates limitations of linear TFKN approximations for hierarchical task representations (Yorsh et al., 2022).

7. Future Directions

Ongoing research on TFKNs targets both theoretical and practical axes:

• Theoretical Guarantees: Quantifying approximation errors (e.g., $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^C$5) and linking them to downstream generalization remains open. Precise bounds on capacity, expressivity, and error propagation under non-convex and infinite-width regimes are actively investigated (Yorsh et al., 2022, Lauditi et al., 11 Feb 2025).
• Architectural Innovations: Directions include deeper φ architectures, alternative positivity-inducing nonlinearities (ELU+1, learned activation functions), dynamic setting of projection dimension per layer or instance, structured random features, and compositional kernels for graph, convolutional, and attention layers (Yorsh et al., 2022, Sehanobish et al., 2023).
• Hybrid and Greedy Training Schemes: Combining layerwise closed-form or spectral training with end-to-end finetuning, or with explicit supervision of hidden representations, offers a route to improved interpretability and efficiency (Wu et al., 2020, Duan et al., 2018).
• Scalability and Hardware Mapping: Leveraging low-rank/Nyström approximations, parallelization (across samples, features, or kernel evaluations), and GPU/TPU-friendly architectures enhances the viability of TFKNs in large-scale settings (Haren et al., 28 Feb 2025, Sehanobish et al., 2023).
• Broader Application Settings: Application to reinforcement learning, associative memory, graph representation, and adaptive control evidences the broad applicability and versatility of TFKNs, especially where data, task, or control structure can be encoded into kernel parameterization, and kernel adaptation confers tangible accuracy or sample-efficiency advantages (Iatropoulos et al., 2022, Haren et al., 28 Feb 2025).

In summary, trainable feedforward kernel networks generalize and improve upon both classical kernel machines and deep neural networks by integrating data-adaptive kernel representation learning into the architecture, yielding efficient, expressive, and performant models for a wide range of domains (Yorsh et al., 2022, Sehanobish et al., 2023, Sun et al., 2018, Haren et al., 28 Feb 2025).