Learnable Kernel Strategies

• Learnable kernel strategies are methods that learn the kernel function’s parameters or feature map from data, allowing adaptive similarity measures beyond fixed kernels.
• They integrate techniques like multiple kernel learning, continuous parameterization, and sample-dependent corrections to tailor similarity geometry for various tasks.
• These strategies enhance scalability and performance in applications ranging from domain adaptation and few-shot regression to deep attention and operator learning.

Learnable kernel strategies are methods that learn the kernel function, its parameterization, or the feature map inducing it from data rather than fixing the kernel a priori. In the reproducing-kernel Hilbert space (RKHS) viewpoint, this amounts to learning the Hilbert space itself, because a kernel uniquely determines its RKHS. Across the literature, this idea appears in several forms: learning nonnegative combinations of base kernels, learning where different kernels should apply, learning spectral measures for stationary kernels, learning sample-dependent or pairwise adaptive kernel corrections, and embedding learnable kernels into deep architectures, linear attention, and operator learning systems (Lu et al., 2021, Moeller et al., 2016, Bullins et al., 2017, Li et al., 2019).

1. Conceptual foundations

The common starting point is that kernel learning is more general than hyperparameter tuning. Classical RBF practice often learns a single bandwidth, and multiple kernel learning (MKL) often learns coefficients over a fixed dictionary, but both remain constrained by a preset kernel family. Several works explicitly argue that this is the central limitation: one learns either a scalar locality scale or a convex combination of hand-selected kernels, while the geometry of similarity remains largely fixed (Lu et al., 2021, Tomioka et al., 2010).

In standard MKL, a classifier or regressor is learned jointly with kernel weights, typically through a conical combination

$$\bar{k}(x,x')=\sum_{m=1}^M d_m\,k_m(x,x'), \qquad d_m\ge 0,$$

or, equivalently, through component functions in multiple RKHS blocks. A major unifying result shows that direct regularization of kernel weights and block-norm regularization of component RKHS functions are related by a concave conjugate transform. This places $\ell_p$-norm MKL, Elastic-net MKL, structured sparsity, and multi-task learning in a common variational framework (Tomioka et al., 2010).

A second foundational line replaces finite kernel dictionaries by continuously parameterized kernel families. Continuous alignment-based kernel learning optimizes centered alignment with the label kernel over a continuous parameter space $\Sigma$, avoiding the discretization step that can become infeasible in multi-dimensional parameterizations. The learned kernel is constructed stagewise by repeatedly adding a kernel from the continuous family that maximizes a directional alignment criterion (Afkanpour et al., 2011).

Taken together, these works distinguish several meanings of “learning the kernel”: learning weights over kernels, learning continuous kernel parameters, learning sample-dependent corrections, or learning spectral and feature-space representations. This suggests that learnable kernel strategies are best understood as a family of mechanisms for adapting similarity geometry to the training task rather than as a single algorithmic template.

2. Mixtures, localization, and continuous kernel dictionaries

The earliest and most explicit learnable-kernel strategies are global kernel mixtures. In the standard MKL formulation,

$$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$

the weights $\eta_i$ are constants, so every example sees the same kernel combination. This globality is precisely what localized kernel learning (LKL) is designed to relax (Moeller et al., 2016).

LKL introduces gating functions that vary across the input space. The gated kernel has the form

$$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$

and the main framework studies separable gates

$$\gamma_i(x,x')=\eta_i(x)\eta_i(x'), \qquad \sum_{i=1}^m \eta_i(x)=1,\ \eta_i(x)\ge 0.$$

Under this separability, each gated term is positive semidefinite by the Schur product theorem, and the kernel admits a product-RKHS interpretation in which examples are placed into local Hilbert subspaces. This geometric view explains why localized methods can use fewer support points than global MKL: the classifier is fitted on region-specific subspaces rather than a single global geometry (Moeller et al., 2016).

Within this framework, LMKL learns a softmax gating function, C-LMKL uses soft clustering and a convex formulation, SwMKL smooths kernel-wise success signals, and SAMKL uses latent binary variables. LD-MKL, the paper’s new method, constructs the final classifier from kernel-specific classifiers and learned gates, uses $\tanh$ rather than a hard sign for noisy local decisions, and is reported to be much faster than LMKL and C-LMKL while typically being best or statistically indistinguishable from the best (Moeller et al., 2016).

A related but distinct strategy is to learn from a continuous set of base kernels rather than a finite dictionary. Continuous alignment-based kernel learning maximizes centered alignment

$$A_c(K,\tilde K)=\frac{\langle K_c,\tilde K_c\rangle_F}{\|K_c\|_F\,\|\tilde K_c\|_F}$$

over a continuous family such as Gaussian kernels with one-dimensional or coordinate-wise bandwidths. The method is explicitly motivated by the observation that the choice of the base-kernel family matters at least as much as the combination rule: finite dictionaries can fail simply because the correct parameter values are not present (Afkanpour et al., 2011).

The regularization perspective adds an important caution. In experiments on large kernel dictionaries, $\ell_2$-norm MKL and Elastic-net MKL achieved comparable accuracy to uniform kernel combination, and uniform combination can be preferable from simplicity. The importance of learnable kernel strategies in that setting lies not only in raw predictive gains, but in learning the usefulness of the information sources represented as kernels; Elastic-net MKL, in particular, achieves sparsity in the kernel weights (Tomioka et al., 2010).

3. Spectral learning and explicit feature constructions

For translation-invariant kernels, Bochner’s theorem makes kernel learning equivalent to learning a nonnegative spectral measure: $\ell_p$0 Several learnable kernel strategies therefore move into the Fourier domain and treat the spectrum itself as the object to be learned (Bullins et al., 2017, Oliva et al., 2015, Li et al., 2019).

“Not-So-Random Features” learns an invariant kernel directly in the Fourier domain rather than sampling random features from a fixed spectral density. The method iteratively finds informative Fourier atoms, adds their corresponding feature pairs, and updates SVM dual variables, yielding a sparse, data-adaptive finite-support spectral measure. The paper interprets this as online equilibrium-finding in a two-player min-max game between a kernel player choosing spectral atoms and an SVM player choosing dual coefficients, with convergence and generalization guarantees (Bullins et al., 2017).

“A la Carte - Learning Fast Kernels” retains the structured Fastfood approximation and learns grouped spectral parameters rather than every frequency individually. Its central computational claim is that Fastfood expansions require

$\ell_p$1

for $\ell_p$2 basis functions and input dimension $\ell_p$3. The paper proposes FSARD and FSGBARD, which learn diagonal Fastfood factors, and also Gaussian spectral mixtures and piecewise linear radial spectra, all trained by Gaussian-process marginal likelihood (Yang et al., 2014).

BaNK makes the spectral density itself Bayesian nonparametric by placing a Dirichlet process prior over a Gaussian mixture

$\ell_p$4

Because the method works through random Fourier features, inference scales linearly in the number of data points rather than through an $\ell_p$5 Gram matrix, while still learning a broad class of stationary kernels (Oliva et al., 2015).

Implicit Kernel Learning (IKL) replaces an explicit spectral density by an implicit neural sampler

$\ell_p$6

so the kernel is learned by learning how to sample its Fourier frequencies. In supervised learning, IKL uses a two-stage procedure: first learn the kernel by alignment, then resample random Fourier features from the learned generator to train the final classifier. In MMD GANs, it learns the spectrum jointly with the critic embedding (Li et al., 2019).

Explicit feature-space kernel construction also appears outside Fourier methods. MBKL learns a kernel as a weighted sum of binary base kernels induced by randomized binary mappings,

$\ell_p$7

typically using random decision stumps. The result is a Mercer-valid kernel with an explicit sparse feature map and linear-SVM training, designed to reduce the burden of manual kernel selection while remaining efficient in time and memory (Roig et al., 2013).

4. Sample-dependent and nonparametric kernel adaptation

A different line of work learns the kernel directly as a data-dependent matrix or as a sample-specific local geometry. In these methods, the kernel is not merely chosen from a family; it is modified entrywise or pointwise to reflect the training sample distribution (Lu et al., 2021, Liu et al., 2018, He et al., 2023).

SDLK-DAL introduces a sample-dependent and learnable Positive Definite Quadratic Kernel (PDQK),

$\ell_p$8

where $\ell_p$9 is the learnable component and $\Sigma$0 is built from evaluations against unsupervised samples. The kernel is learned by minimizing source-target Maximum Mean Discrepancy, so the RKHS itself is adapted before standard domain-adaptation subspace learning is applied. The optimization is constrained to symmetric positive semidefinite matrices and is handled on the SPD manifold with a trust-region algorithm. Embedded into TCA, SSTCA, and IGLDA, the resulting SDLK variants consistently improved performance on face recognition, object recognition, handwritten digit recognition, and text classification; reported average improvements include $\Sigma$1, $\Sigma$2, and $\Sigma$3 on face recognition for the corresponding SDLK versions, with gains over $\Sigma$4 in some target domains (Lu et al., 2021).

DANK learns a nonparametric multiplicative correction

$\Sigma$5

jointly with SVM or SVR dual variables. The adaptive matrix $\Sigma$6 is regularized to stay close to $\Sigma$7 and to have low rank through a nuclear norm term. A key theoretical property is that, after eliminating $\Sigma$8, the resulting objective is gradient-Lipschitz continuous, which allows Nesterov-accelerated projected gradient optimization. For large-scale data, a decomposition-based approximation partitions the problem into clusters and exploits near block-diagonality of the kernel matrix (Liu et al., 2018).

LAB RBF kernels make bandwidths sample-dependent. The kernel

$\Sigma$9

attaches a positive bandwidth vector $$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$0 to each support point, producing an asymmetric kernel because $$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$1 in general. This motivates an asymmetric kernel ridge regression framework with closed-form stationary solutions and an iterative training procedure that alternates between fitting on a support set, optimizing bandwidths on the remaining data, and dynamically expanding the support set (He et al., 2023).

A related geometric strategy learns a linear deformation of the input space and then applies an RBF-type kernel: $$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$2 This two-layer kernel machine generalizes scalar length-scale tuning to anisotropy plus rotation. The learned kernel is then used inside VKOGA to build a sparse, adaptive basis. On several synthetic and real regression benchmarks, the learned map improved the preasymptotic behavior and, in some datasets, allowed around $$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$3 centers to achieve accuracy comparable to a standard kernel using $$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$4 centers (Wenzel et al., 2023).

5. Task-conditioned kernels and architecture-integrated kernels

Deep kernel learning extends kernel learning from static similarity functions to learned feature maps. Adaptive Deep Kernel Learning (ADKL) pushes this further by learning a family of kernels, one per task: $$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$5 The task embedding $$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$6 is computed from the support set through a permutation-invariant task encoder, so the kernel used at inference time is conditioned on the current task rather than shared globally across all tasks. The meta-objective optimizes expected validation loss across tasks, and the paper reports consistent gains on real-world few-shot regression benchmarks in drug discovery, including Binding with $$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$7 tasks and Antibacterial with $$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$8 tasks, against R2-D2, CNP, and MAML (Tossou et al., 2019).

In subquadratic language modeling, learnable kernels appear as attention replacements. ReBased modifies the Based linear-transformer kernel by introducing normalization and an affine transformation before the quadratic map, yielding the learnable form

$$f(x)=\sum_{i=1}^m \eta_i \langle \mathbf{w}_i,\phi_i(x)\rangle + b,$$9

The motivation is that the original quadratic Taylor kernel cannot drive unwanted similarities sufficiently close to zero. On MQAR, ReBased outperformed all baselines except full attention across sequence lengths and model sizes; on the Pile, the reported perplexities were 16.69 for Based and 16.45 for ReBased on all tokens, with AR perplexity improving from 3.72 to 3.67 and Non-AR perplexity from 57.21 to 56.31 (Aksenov et al., 2024).

Flexformer also makes the attention kernel learnable, but through trainable Fourier frequencies in random-feature linear attention. The stationary version learns the spectral density directly, while the nonstationary version uses a Yaglom-style joint spectral representation and a learnable norm scale $\eta_i$0. The paper emphasizes that the nonstationary variant is strictly more expressive because stationary kernels are a special case, and reports strong long-sequence performance as well as distillation from pretrained Transformers to recover softmax attention (Zhang et al., 26 Jun 2026).

ITNet generalizes the notion of a learnable kernel beyond attention by defining a learnable integral transform

$\eta_i$1

with a matrix-valued kernel produced by a small MLP from joint position-content inputs. Convolution, self-attention, and recurrence appear as special cases under appropriate parameterizations, and the paper states that ITNet is a universal approximator of continuous operators. To make the operator practical, it introduces tiled kernel fusion, importance-weighted Monte Carlo integration, and learned low-rank factorization. A single ITNet backbone is reported to reach about $\eta_i$2 top-1 on ImageNet-1K, around $\eta_i$3 on ModelNet40, about $\eta_i$4 on VQA v2, and around $\eta_i$5 on NLVR2 (Dhor et al., 17 Jun 2026).

Kernel attention is also used in continual 3D anomaly detection. C3D-AD employs KAL, which maps tokens into a unified kernel space through positive random features, and KAA, which replaces ordinary attention memory by a continually learnable advisor matrix $\eta_i$6. The full system, together with reconstruction with parameter perturbation, achieved average AUROC values of $\eta_i$7, $\eta_i$8, and $\eta_i$9 on Real3D-AD, Anomaly-ShapeNet, and MulSen-AD, respectively (Lu et al., 2 Aug 2025).

6. Applications, empirical patterns, and open issues

The application range of learnable kernel strategies is unusually broad. In domain adaptation, learnable kernels are used to reduce MMD between source and target domains before subspace learning (Lu et al., 2021). In few-shot regression, they adapt the covariance function to each support set (Tossou et al., 2019). In mean-variance trading, they parameterize path-dependent strategies as functions in an RKHS and retain closed-form representer-type solutions; the resulting kernel and signature methods are reported to outperform classical Markovian methods whenever temporal dependencies are present (Futter et al., 14 Jul 2025). In finite-rate-of-innovation sampling, the sampling kernel and reconstruction encoder are jointly optimized, and a hardware-friendly kernel family is realized by a unity-gain Sallen–Key analog filter; the reported system uses about $$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$0 less sampling than FRIED-Net under matched conditions, a $$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$1M-parameter encoder versus $$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$2M for FRIED-Net, and a bench-test reconstruction with true delays $$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$3, estimated delays $$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$4, and NMSE around $$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$5 dB (Nitsure et al., 28 Sep 2025).

Several recurring themes govern these methods. First, flexibility is almost always paired with structural constraints. Positive semidefiniteness is preserved by PSD matrix learning, Schur products, nonnegative kernel weights, or spectral nonnegativity; DANK constrains $$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$6 by centering and nuclear norm penalties; LAB imposes positive bandwidths; IKL enforces spectral symmetry and a variance constraint; SDLK-DAL optimizes on the SPD manifold (Lu et al., 2021, Liu et al., 2018, Li et al., 2019, He et al., 2023).

Second, scalability is a central design criterion. Fastfood reduces spectral-kernel computation to $$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$7 time and $$\kappa_\gamma(x,x') = \sum_{i=1}^{m} \gamma_i(x,x') \kappa_i(x,x'),$$8 memory; BaNK and IKL avoid Gram-matrix learning by random features; DANK uses Nesterov acceleration and decomposition; ReBased and Flexformer preserve linear-attention structure; ITNet uses tiled fusion, Monte Carlo integration, and low-rank factorization; VKOGA adds sparse greedy basis selection (Yang et al., 2014, Oliva et al., 2015, Liu et al., 2018, Aksenov et al., 2024, Dhor et al., 17 Jun 2026, Wenzel et al., 2023).

Third, empirical superiority is not universal. Uniform kernel combination can remain competitive in MKL (Tomioka et al., 2010). ReBased narrows but does not close the gap to vanilla attention on AR perplexity and attention-map quality (Aksenov et al., 2024). In kernel trading, the kernel method is more general, but signature methods can be more interpretable, more scalable for online use, and less sensitive to hyperparameters (Futter et al., 14 Jul 2025). In two-layer kernel learning with VKOGA, performance is mixed on some real datasets, and query_agg_count is reported as inferior for the learned metric (Wenzel et al., 2023).

A plausible synthesis is that learnable kernel strategies are most effective when the inductive bias embedded in a fixed kernel is visibly misaligned with the data: heterogeneous task families, domain shift, strong local structure, path dependence, long-range retrieval, or hardware-constrained sensing. Their main technical challenge is to enlarge the admissible similarity geometry without losing PSD validity, numerical stability, or tractable optimization.