Deep Kernel Greedy Models

• Deep Kernel Greedy Models are a family of methods that combine multilayer composition, kernelized representation learning, and greedy training to optimize models sequentially.
• They include formulations like KNet and kernelized connectionist models, and extend to applications in PDE collocation, surrogate modeling, and operator-aware tasks.
• These models enable layer-wise, non-end-to-end training that provides theoretical guarantees on convergence and generalization while bypassing traditional backpropagation.

Deep kernel greedy models are best understood as a family of kernel-based constructions in which multilayer composition, kernelized representation learning, and greedy or stagewise optimization are combined rather than treated as separate design choices. In the literature surveyed here, the most direct instances are supervised multilayer kernel architectures trained one layer at a time without backpropagation, while adjacent strands extend greedy kernel selection to sparse interpolation, PDE collocation, surrogate modeling, and posterior approximation in function space. This suggests that the term does not name a single standardized model class; instead, it denotes an intersection of themes: deep composition, kernel machinery, and greedy training or selection (Wu et al., 2020).

1. Canonical formulations of the deep kernel greedy idea

A direct formulation appears in the Kernel Dependence Network, or KNet, proposed for multi-class classification. KNet is trained greedily, one layer at a time; each layer is defined as a composition of linear weights with the feature map of a Gaussian kernel acting as the activation function; and at each layer the linear weights are learned by maximizing the dependence between the layer output and the labels using the Hilbert Schmidt Independence Criterion. By constraining the solution space on the Stiefel Manifold, the network can be solved spectrally while leveraging the eigenvalues to automatically find the width and the depth of the network, and the paper states that it theoretically guarantees the existence of a solution for the global optimum while providing insight into the network’s ability to generalize (Wu et al., 2020).

A second direct formulation is the family of kernel method-based connectionist models that “kernelize” feedforward networks by replacing neurons with kernel machines. In that framework, the central claim is not merely that a neural architecture can be kernelized, but that for certain output-layer architectures and objective functions the hidden layer can be assigned an explicit supervised target that is optimal for minimizing the objective function of the network. That characterization makes possible a purely greedy training scheme that learns one layer at a time, starting from the input layer, and yields a layer-wise training algorithm for an $l$-layer feedforward network for classification, where $l\ge 2$ can be arbitrary (Duan et al., 2018).

A third formulation is post hoc rather than intrinsic. Deep Kernel Networks can be converted into Deep Map Networks by a greedy layer-wise approximation procedure that builds explicit maps approximating a pretrained deep kernel machine. Here the model is already deep and kernelized, and greediness enters through the design principle: maps are found layer-wise, first at input layers and then at intermediate and output layers, followed by an extra fine-tuning step based on unsupervised learning (Jiu et al., 2018).

2. Layer-wise training without backpropagation

KNet places the greedy principle at the center of supervised deep kernel learning. The data are $X\in\mathbb{R}^{n\times d}$ and $Y\in\mathbb{R}^{n\times \nclass}$, with $Y$ the matrix of one-hot encoded labels over $\nclass$ classes. The motivation is to train a deep architecture without backpropagation/SGD by learning the network greedily, one layer at a time, where each layer is chosen to maximize dependence between its representation and the labels. The representation at layer $l$ is written as $R_l=\af(R_{l-1}W_l)$, with $W_1\in\mathbb{R}^{d\times q}$ for the first layer and $W_l\in\mathbb{R}^{m\times q}$ for layer $l\ge 2$0; the activation $l\ge 2$1 is the feature map of a Gaussian kernel; and the paper explicitly says that KNet “fundamentally models a traditional fully connected multilayer perceptron (MLP)” but changes the activation mechanism and training procedure (Wu et al., 2020).

The connectionist kernelization framework gives a broader theoretical justification for this kind of stagewise learning. In the two-layer case, the global optimum is written as

$l\ge 2$2

The key step is to characterize a hidden-layer target so that training $l\ge 2$3 greedily is aligned with minimizing the full objective. In the realizations emphasized in the paper, the hidden target takes the form of a supervised similarity condition: same-class pairs should attain kernel similarity $l\ge 2$4, and different-class pairs should attain $l\ge 2$5. The resulting hidden-layer loss is a supervised representation similarity objective, and the training procedure is recursive across layers rather than end-to-end through backpropagation (Duan et al., 2018).

These two lines differ in mechanism. KNet uses HSIC as the layer objective and a spectral/Stiefel formulation, whereas kernelized connectionist models use explicit hidden-layer targets and supervised similarity constraints. A plausible implication is that “greedy” in this literature has two technically distinct meanings: direct layer optimization by a label-dependent criterion, and layer-wise decomposition made valid by a hidden-target theorem.

3. Greedy kernel approximation as the shallow foundation

The deeper architectures rest on a much older body of work on greedy kernel approximation. The data-independent $l\ge 2$6-greedy algorithm selects the next point where the current power function is maximal,

$l\ge 2$7

and the resulting rate of convergence is shown to be near-optimal in the case of kernels generating Sobolev spaces. For kernels of Sobolev spaces, the selected points are asymptotically uniformly distributed, which makes $l\ge 2$8-greedy a canonical geometry-only baseline for sparse kernel models (Santin et al., 2016).

A substantial refinement is the target data-dependent scale of greedy algorithms parameterized by $l\ge 2$9. For $X\in\mathbb{R}^{n\times d}$0, the selection rule is

$X\in\mathbb{R}^{n\times d}$1

with the limiting case $X\in\mathbb{R}^{n\times d}$2 giving $X\in\mathbb{R}^{n\times d}$3-greedy. This unifies $X\in\mathbb{R}^{n\times d}$4-greedy, $X\in\mathbb{R}^{n\times d}$5-greedy, $X\in\mathbb{R}^{n\times d}$6-greedy, and $X\in\mathbb{R}^{n\times d}$7-greedy. The paper proves new convergence rates where the degree of dependency of the selection criterion on the functional data is taken into account, and shows that target data adaptive interpolation can be faster than the rates given by uniform points without any special assumption on the target function. In the special cases highlighted in the text, the gains are an extra $X\in\mathbb{R}^{n\times d}$8 for $X\in\mathbb{R}^{n\times d}$9-greedy and an extra $Y\in\mathbb{R}^{n\times \nclass}$0 for $Y\in\mathbb{R}^{n\times \nclass}$1-greedy, up to a logarithmic factor, relative to $Y\in\mathbb{R}^{n\times \nclass}$2-greedy (Wenzel et al., 2021).

Several later developments modify this kernel-greedy substrate rather than replacing it. Greedy regularized kernel interpolation extends greedy point selection to the Tikhonov-regularized problem

$Y\in\mathbb{R}^{n\times \nclass}$3

and treats both regularized $Y\in\mathbb{R}^{n\times \nclass}$4-greedy and regularized $Y\in\mathbb{R}^{n\times \nclass}$5-greedy in the setting where $Y\in\mathbb{R}^{n\times \nclass}$6 can make even positive semidefinite kernels usable through $Y\in\mathbb{R}^{n\times \nclass}$7 (Santin et al., 2018). Exchange-based refinement then adds a fixed-budget local search layer on top of greedy subset construction: the Kernel Exchange Algorithm performs one-for-one swaps of selected and unselected centers, and the reported effect is an reduction of the error up to 86.4% (17.2% on average) without increasing the computational complexity of the final kernel model (Wenzel et al., 2024). Stabilized vectorial greedy kernel methods add a geometry-aware restriction

$Y\in\mathbb{R}^{n\times \nclass}$8

to prevent clustering and improve conditioning, thereby producing sparse kernel models that trade off approximation greediness and geometric/stability control through the scalar parameter $Y\in\mathbb{R}^{n\times \nclass}$9 (Haasdonk et al., 2020).

This shallow theory matters because every deep kernel greedy model still needs some notion of stagewise informativeness. In the deep setting, that role may be played by HSIC, supervised similarity, function-space divergence, or a layer-wise approximation residual, but the underlying question remains the same: which atom, point, layer, or component should be added next?

4. Deep kernel architectures and deep kernel approximations

Not all “deep kernel” constructions are greedy in the same sense, and not all greedy constructions are deep. Deep Kernel Networks and Deep Map Networks illustrate one axis of variation. A DKN recursively defines a kernel at layer $Y$0 by

$Y$1

and evaluating deep kernel Gram matrices has complexity $Y$2. The associated DMN replaces each kernel-valued node with an explicit vector-valued map constructed layer-wise from a subset $Y$3, using eigendecomposition-based projection at each layer. The method is greedy because each layer is approximated using the lower-layer maps already fixed, and the paper states that the resulting DMNs are as accurate as the underlying DKNs while being at least an order of magnitude faster on large-scale datasets (Jiu et al., 2018).

A distinct notion of depth appears in Hierarchic Kernel Recursive Least-Squares. There the model does not compose learned feature maps through activation functions; instead, the weights of a kernel model over each dimension are modeled over its adjacent dimension, creating a deep hierarchical structure in coefficient space. For $Y$4, the hierarchy successively models kernel coefficients over $Y$5. This is deep in a recursive coefficient-field sense rather than in the modern deep-kernel-learning sense, and the paper explicitly frames the gain as computational speedup and improved modeling accuracy for evenly distributed multidimensional datasets (Mohamadipanah et al., 2017).

The contrast with low-rank deep kernel decomposition is equally instructive. Deep Basis Kernel parameterizes a kernel directly as

$Y$6

with $Y$7 learned by a neural network. This is a deep adaptive low-rank kernel decomposition, but it is not a greedy method: there is no sequential basis selection, forward stagewise dictionary growth, matching pursuit, orthogonal greedy approximation, active rank expansion, or stepwise inducing point addition. The basis functions are learned jointly by end-to-end optimization of the marginal likelihood or a variational lower bound (Zhu et al., 24 May 2025).

This boundary is important. A plausible implication is that the strongest uses of the term “deep kernel greedy model” should be reserved for constructions in which depth and greediness are both intrinsic to training, not merely present in neighboring senses.

5. Function-space, operator-aware, and scientific-computing extensions

The deep-kernel-greedy theme extends beyond supervised classification when kernels are used to organize selection in function space or operator space. In greedy Bayesian posterior approximation with deep ensembles, the kernel is defined over functions represented by neural networks, and the posterior is approximated by

$Y$8

The set function $Y$9 is non-negative and non-monotone submodular, and the practical training rule for the $\nclass$0-th member becomes

$\nclass$1

Here greediness means sequential posterior fitting in function space, and the paper is explicit that this is not a deep kernel learning paper in the usual Gaussian-process or learned-input-kernel sense (Tiulpin et al., 2021).

Operator-aware greedy kernel methods are especially developed in PDE collocation. PDE-greedy kernel methods define candidate functionals

$\nclass$2

and then select them by PDE-$\nclass$3-greedy criteria that generalize PDE-$\nclass$4-greedy and PDE-$\nclass$5-greedy. The main theoretical conclusion is that target-data dependent algorithms that make use of the right hand side functions of the BVP exhibit faster convergence rates than the target-data independent PDE-$\nclass$6-greedy, and that the convergence rate of the PDE-$\nclass$7-greedy possesses a dimension independent rate (Wenzel et al., 2022). The later extension to parametric elliptic boundary value problems lifts the domain to a joint state-parameter domain via state-parameter product kernels, reports algebraic convergence rates for Sobolev-space kernels and exponential convergence rates for infinitely smooth kernels and solutions, and interprets the construction as an “a priori” model reduction method because no solution snapshots need to be precomputed (Haasdonk et al., 9 Jul 2025).

Related scientific-computing applications keep the greedy kernel mechanism but not the deep architecture. Greedy kernel surrogates for center manifold approximation combine a vector-valued kernel surrogate for a manifold map $\nclass$8, a regularized fitting objective with hard manifold-consistency constraints at the equilibrium, and a $\nclass$9-greedy selection rule to sparsify the surrogate before solving the fitting system (Haasdonk et al., 2018). Greedy kernel methods also accelerate implicit integrators for parametric ODEs by learning a vector-valued sparse kernel interpolant for the numerical one-step map and using it only as an initializer for the nonlinear solve, so the ODE solver still guarantees the required precision (Brünnette et al., 2018).

6. Terminological boundaries, misconceptions, and research directions

A recurrent misconception is to treat every greedy kernel method as deep, or every deep kernel model as greedy. The literature does not support either identification. Classical greedy kernel interpolation, stabilized VKOGA, regularized kernel interpolation, PDE-greedy collocation, center manifold approximation, and ODE acceleration are greedy kernel methods, but they do not introduce multilayer learned representations (Santin et al., 2018). Conversely, low-rank deep kernel decomposition is a deep kernel model, but the basis functions are learned jointly rather than greedily (Zhu et al., 24 May 2025).

A second misconception is to assume that every kernelized deep model belongs to standard deep kernel learning. The deep-ensemble posterior-approximation framework is kernelized in function space, not in input space, and the authors explicitly distinguish it from deep kernel learning in the usual Gaussian-process or learned-input-kernel sense (Tiulpin et al., 2021). Likewise, Hierarchic Kernel Recursive Least-Squares is deep because of a hierarchical coefficient-space reorganization, not because it composes a neural feature extractor with a base kernel (Mohamadipanah et al., 2017).

The most stable core of the field therefore consists of models in which depth, kernelization, and greediness are simultaneous rather than merely adjacent. KNet is deep because it stacks layers, kernel because each layer uses a Gaussian-kernel feature map, and greedy because layers are optimized sequentially rather than jointly (Wu et al., 2020). Kernel method-based connectionist models make the same point in a broader theorem-driven way by explicitly characterizing hidden-layer targets that support purely greedy supervised training (Duan et al., 2018). Deep Map Networks offer a complementary view in which a pretrained deep kernel machine is greedily compiled into an explicit deep representation layer by layer (Jiu et al., 2018).

This suggests a useful taxonomy. One branch studies genuinely deep greedy kernel learners; a second studies greedy sparse approximation in fixed RKHSs; a third extends greedy selection to operator-aware and function-space settings; and a fourth studies deep kernel models that are adjacent but not greedy. Within that taxonomy, the enduring research question is not whether greediness should replace depth or kernelization, but how stagewise selection, multilayer representation, and kernel geometry can be coupled without losing the approximation theory that made greedy kernel methods analytically tractable in the first place.