Structured Fixed Kernel Layers

• Structured fixed kernel layers are modules based on pre-defined positive-definite kernels that remain fixed during downstream computation, enhancing modularity and interpretability.
• They employ techniques like polynomial transforms, Gegenbauer expansions, and RKHS decompositions to optimize structured prediction through criteria such as HSIC.
• Empirical studies demonstrate improved performance in regression, classification, and graph tasks, although fixed configurations may limit expressivity compared to adaptive methods.

Searching arXiv for recent and foundational papers on structured fixed kernel layers and related kernelized/deep modular architectures. Structured fixed kernel layers are kernel-induced modules in which the transformation is determined by a prescribed positive-definite kernel, a structured family of kernels, or a data-dependent RKHS subspace that is subsequently held fixed during downstream computation. In the cited literature, this includes polynomially transformed input and output kernels for structured regression, unsupervised or sketched RKHS bases for graph and output prediction, fixed or frozen kernel-machine modules inside deep networks, and local kernel ridge readouts whose feature locations, targets, or evaluation points may be fixed or learned (Tonde et al., 2016, Chen et al., 2020, Ahmad et al., 2024, Mercier et al., 4 May 2026).

1. Foundational formulations

A foundational formalization appears in two-layer kernel machines, where the predictor is written as a composition

$$f = f_2 \circ f_1, \qquad f_1 : X \to Z,\; f_2 : Z \to Y.$$

The associated variational problem is posed over RKHSs for $f_1$ and $f_2$, and the representer theorem guarantees optimal finite expansions at both layers: $$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$ The composite map is then equivalent to a single-layer kernel machine with a kernel learned from the data,

$K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$

which establishes an explicit connection between multilayer architectures and kernel learning (Dinuzzo, 2010).

A second foundational viewpoint reinterprets standard deep learning layers as kernel machines by absorbing the trailing nonlinearity of layer $i$ into the beginning of layer $i+1$. For a one-hidden-layer network,

$$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$

the kernel-based view is

$$f(\mathbf{x}) = \langle \mathbf{w}_2, \Phi(F_1(\mathbf{x})) \rangle, \qquad k(F_1(\mathbf{u}), F_1(\mathbf{v})) = \langle \Phi(F_1(\mathbf{u})), \Phi(F_1(\mathbf{v})) \rangle.$$

Under this construction, each module is a linear model in a feature space induced by the preceding nonlinearity, and for common neural networks the feature maps are explicit and finite-dimensional, so evaluation remains linear in sample size rather than incurring the quadratic bottleneck of classical kernel machines. This same framework motivates modular training in which earlier modules are frozen and reused without between-module backpropagation (Duan et al., 2020).

2. Algebraic and operator-theoretic constructions

One prominent construction uses polynomial transformations of kernels for structured prediction. For a positive definite kernel $k(x,x')$, the transformed kernel may be expanded in a monomial basis,

$f_1$0

leading to

$f_1$1

or in a Gegenbaur basis,

$f_1$2

with the recurrence

$f_1$3

The same expansion is applied to input and output kernels,

$f_1$4

thereby creating a structured kernel layer tailored to maximize dependence between input and output RKHS features. Dependence is measured by

$f_1$5

and the optimal coefficients are obtained from the top left and right singular vectors of the non-negative matrix $f_1$6 with entries $f_1$7 (Tonde et al., 2016).

A different line of work constructs a monotone sequence of kernel layers by transporting a nonlinear operator dynamic into RKHSs. The operator recursion

$f_1$8

produces a decreasing sequence of positive operators, together with an exact residual decomposition

$f_1$9

With feature map $f_2$0, this yields a family of kernels

$f_2$1

and a residual kernel decomposition

$f_2$2

The limit kernel

$f_2$3

is the maximal kernel dominated by the original one and annihilating the prescribed nuisance subspace $f_2$4, so the sequence $f_2$5 provides a structured path from soft attenuation to hard invariance (Tian, 4 Dec 2025).

A third construction emphasizes analytically tractable fixed kernels. The Yat kernel

$f_2$6

is PSD for $f_2$7, and for $f_2$8 it dominates a scaled inverse-multiquadric in the Loewner order. The corresponding RKHS is universal, characteristic, and strictly positive definite on every compact domain, while a trained shared-$f_2$9 layer remains a finite learned-center expansion in a fixed RKHS with exact norm

$$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$0

The kernel also admits the explicit diagonal

$$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$1

which enters a Rademacher generalization bound directly (Bouhsine, 5 May 2026).

3. What is fixed, and when it is fixed

The expression “fixed” does not denote a single training protocol. In stacked kernel architectures, the kernel feature map can be fixed while the coefficients or anchor parameters remain learnable. The Stacked Kernel Network and Stacked Kernel Convolutional Network parameterize each hidden unit by an RKHS function and provide three representations: nonparametric,

$$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$2

parametric,

$$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$3

and random Fourier feature form,

$$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$4

The paper explicitly notes that traditional fixed kernel layers correspond to fixed, non-trainable, or data-dependent nonlinear transformations, whereas SKN and SKCN stack such transformations with learnable parameters (Zhang et al., 2017).

Graph convolutional kernel networks provide a different fixation regime. Their node feature maps are built from path kernels and then projected onto a low-dimensional Nystrom subspace spanned by anchor points $$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$5, with path embedding

$$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$6

In unsupervised mode, the anchor points are learned via K-means on randomly sampled paths and then fixed; in supervised mode, they can be fine-tuned end-to-end. Deep Sketched Output Kernel Regression fixes a different object: its last layer predicts in a finite-dimensional subspace of the output RKHS spanned by basis elements

$$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$7

where the basis is obtained from a randomized sketch of the empirical output kernel covariance operator and then kept fixed while the input network is trained (Chen et al., 2020, Ahmad et al., 2024).

A further decomposition is made explicit in differentiable kernel ridge regression layers. Dense and sparse kernel modules expose three parameter sets—feature representations, target values, and evaluation points—and each may be fixed or learned independently. In the sparse case, the predictor at an evaluation point $$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$8 uses only its $$f_1^*(x) = \sum_{i=1}^{\ell} K_1(x, x_i) a_i, \qquad f_2^*(z) = \sum_{i=1}^{\ell} K_2(f_1^*(x_i), z) b_i.$$9-nearest neighbors,

$K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$0

so that, when features, targets, evaluation points, and kernel are all fixed, the module acts as a structured, geometry-aware nonlinear readout layer (Mercier et al., 4 May 2026).

Construction	Fixed component	Adaptive component
Polynomial transformed kernels (Tonde et al., 2016)	transformed kernel form after HSIC/SVD selection	expansion coefficients during kernel learning
GCKN Nystrom layer (Chen et al., 2020)	anchor-induced RKHS subspace in unsupervised mode	anchor points if fine-tuned
DSOKR output layer (Ahmad et al., 2024)	sketched output basis $K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$1	input network $K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$2
Sparse Kernels (Mercier et al., 4 May 2026)	any subset of features, targets, evaluation points, kernel	complementary subsets

4. Architectural realizations

In structured prediction, the most direct realization applies fixed or learned kernel transformations simultaneously to inputs and structured outputs. The transformed pair $K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$3 defines a higher-level kernel layer whose objective is to maximize dependency between the two RKHSs, and the method is typically applied to settings such as Twin Gaussian Processes, where both input and output distributions are modeled as Gaussian processes and prediction uses a matching function based on KL divergence or HSIC (Tonde et al., 2016).

For graph-structured data, kernel layers are built from local substructures. In GCKN, the initial node map is $K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$4, and the first structured kernel compares paths of length $K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$5,

$K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$6

typically with Gaussian path kernel

$K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$7

This is iterated layerwise,

$K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$8

so each layer aggregates richer local graph substructures while remaining interpretable as a projection in an RKHS (Chen et al., 2020).

Kernelized layers have also been inserted directly into convolutional architectures. “Kervolutional” layers replace the standard linear inner product with polynomial or Gaussian kernels, learnable weights pooling introduces kernelized pooling rules such as

$K(x_1, x_2) = K_2(f_1^*(x_1), f_1^*(x_2)),$9

and Kernelized Dense Layers replace standard fully connected maps with kernelized alternatives such as

$i$0

or Gaussian RBF forms. The paper emphasizes that linear kernels may not be sufficiently effective to fit input data distributions, whereas high-order kernels are prone to over-fitting, and therefore a trade-off between complexity and performance is required (Mahmoudi et al., 2023).

A deeper kernel-network formulation appears in Structured Deep Kernel Networks. These models alternate matrix-valued linear kernel layers

$i$1

with single-dimensional diagonal nonlinear kernel layers

$i$2

starting and ending with a linear layer. The deep kernel representer theorem implies that each layer admits a finite expansion over propagated centers, and the paper proves universal approximation in three asymptotic regimes: unbounded number of centers, unbounded width, and unbounded depth (Wenzel et al., 2021).

5. Empirical behavior across application domains

Polynomial kernel transformations for structured regression were reported to produce state-of-the-art results on several datasets. On the S-Shape synthetic regression task, the reported percent gain over the baseline was $i$3 for KL-divergence with monomial transformation and $i$4 for KL-divergence with Gegenbaur transformation. On Poser, gains reached $i$5 with the Gegenbaur transform; on USPS digits, the reported gain was approximately $i$6 for KL-divergence with Gegenbaur of degree $i$7; and on HumanEva-I, some settings reported gains up to $i$8 with Gegenbaur and KL-divergence for some features (Tonde et al., 2016).

In modular pairwise learning with kernels, modular training matched end-to-end performance on MNIST and CIFAR-10 with various backbones. One cited CIFAR-10 result with a ResNet-18 backbone reported $i$9 modular accuracy versus $i+1$0 end-to-end. The same framework also reported $i+1$1 accuracy on CIFAR-10 using only $i+1$2 randomly selected labeled examples, one from each class, for output-module training. The paper further states that its proxy objective precisely described the task space structure of $i+1$3 binary classification tasks from CIFAR-10 at practically no computation overhead (Duan et al., 2020).

Stacked kernel architectures and graph kernel networks report gains in both Euclidean and graph domains. On the Segment dataset, a 3-layer P-SKN with polynomial kernel achieved $i+1$4 versus $i+1$5 for a DNN of the same configuration, while on CIFAR-10 a 3-layer RFF-SKCN with dropout achieved $i+1$6 versus $i+1$7 for the CNN baseline. In graph classification, GCKN-subtree-unsup was reported to reach $i+1$8 accuracy on MUTAG, and the paper states that single-layer and multilayer unsupervised GCKNs achieve competitive or superior accuracy to standard graph kernels and state-of-the-art GNNs, especially when data is limited (Zhang et al., 2017, Chen et al., 2020).

Kernelized CNN components also showed dataset-dependent gains. For learnable weights pooling, RAF-DB improved from $i+1$9 with max pooling to $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$0 with a 3rd-order polynomial kernel, and CIFAR-10 improved from $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$1 with max pooling to $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$2 with a 3rd-order polynomial kernel. For Kernelized Dense Layers, FER2013 improved from $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$3 with a linear dense layer to $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$4 with a 3rd-order polynomial kernel, while RAF-DB improved from $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$5 to $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$6 (Mahmoudi et al., 2023).

Output-kernel and kernel-ridge modules extend these empirical patterns to structured outputs and hybrid deep pipelines. DSOKR is reported to exceed baselines on synthetic regression and supervised graph prediction problems, including QM9 and ChEBI-20, while sparse kernel modules served as training-free nonlinear readouts in transfer settings, nonlinear probes for intermediate representations, and augmentations to Double DQN that learned faster and achieved higher reward than standard DQN (Ahmad et al., 2024, Mercier et al., 4 May 2026).

6. Limitations, trade-offs, and theoretical boundaries

A central limitation is that fixed-kernel discrimination can be strictly weaker than feature-learning discrimination. Using the function classes $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$7 for fixed-kernel discriminators and $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$8 for feature-learning discriminators, separation results on hyperspheres show that there exist pairs of distributions that cannot be discriminated by fixed-kernel IPM and Stein discrepancy in high dimensions but can be discriminated by their feature-learning counterparts. For the constructed pair $$G_1(\mathbf{x}) = \Phi(\mathbf{W}_1^\top \mathbf{x}), \qquad f(\mathbf{x}) = \mathbf{w}_2^\top G_1(\mathbf{x}),$$9, the ratio

$$f(\mathbf{x}) = \langle \mathbf{w}_2, \Phi(F_1(\mathbf{x})) \rangle, \qquad k(F_1(\mathbf{u}), F_1(\mathbf{v})) = \langle \Phi(F_1(\mathbf{u})), \Phi(F_1(\mathbf{v})) \rangle.$$0

increases exponentially with dimension when $$f(\mathbf{x}) = \langle \mathbf{w}_2, \Phi(F_1(\mathbf{x})) \rangle, \qquad k(F_1(\mathbf{u}), F_1(\mathbf{v})) = \langle \Phi(F_1(\mathbf{u})), \Phi(F_1(\mathbf{v})) \rangle.$$1 scales as $$f(\mathbf{x}) = \langle \mathbf{w}_2, \Phi(F_1(\mathbf{x})) \rangle, \qquad k(F_1(\mathbf{u}), F_1(\mathbf{v})) = \langle \Phi(F_1(\mathbf{u})), \Phi(F_1(\mathbf{v})) \rangle.$$2. The paper interprets this as evidence that fixed-kernel discriminators are weaker because their corresponding metrics are weaker (Domingo-Enrich et al., 2021).

A second limitation concerns expressivity-control trade-offs. In CNN layer kernelization, linear kernels were found insufficiently effective for fitting input data distributions, but high-order kernels were found prone to over-fitting; adding more kervolution layers only increased overfitting, and replacing all layers with high-order kernelized variants led to dramatically decreasing performance. In SKN, the nonparametric representation contains the global optimal function in RKHS but its parameter count grows as $$f(\mathbf{x}) = \langle \mathbf{w}_2, \Phi(F_1(\mathbf{x})) \rangle, \qquad k(F_1(\mathbf{u}), F_1(\mathbf{v})) = \langle \Phi(F_1(\mathbf{u})), \Phi(F_1(\mathbf{v})) \rangle.$$3 per unit per layer and is described as unscalable for large $$f(\mathbf{x}) = \langle \mathbf{w}_2, \Phi(F_1(\mathbf{x})) \rangle, \qquad k(F_1(\mathbf{u}), F_1(\mathbf{v})) = \langle \Phi(F_1(\mathbf{u})), \Phi(F_1(\mathbf{v})) \rangle.$$4 (Mahmoudi et al., 2023, Zhang et al., 2017).

The literature responds to these constraints with partially fixed, data-dependent, or modular designs. DSOKR fixes only the output RKHS subspace after sketching the empirical output covariance; sparse kernels allow any subset of features, target values, and evaluation points to be fixed or learned; and shorting dynamics provide a controlled path from soft regularization to exact nuisance invariance. This suggests that current research treats “fixedness” less as an absolute architectural property than as a design choice about where adaptivity should reside and where geometric structure, invariance, or interpretability should be enforced (Ahmad et al., 2024, Mercier et al., 4 May 2026, Tian, 4 Dec 2025).