Deep Restricted Kernel Machines Overview

• Deep Restricted Kernel Machines (DRKMs) are deep architectures that stack unsupervised KPCA layers with a supervised classification layer under unified, mathematically principled optimization.
• They leverage dual and primal space representations with Stiefel manifold constraints to enhance sample efficiency, energy savings, and memory advantages, particularly for high-dimensional or small-sample data.
• Empirical results demonstrate that DRKMs outperform conventional neural networks and achieve robust disentanglement and reproducibility across various tasks such as image classification and unsupervised representation learning.

Deep Restricted Kernel Machines (DRKMs) refer to a class of deep architectures that stack multiple levels of kernel principal component analysis (KPCA) and supervised classification layers—typically Least Squares Support Vector Machines (LSSVM) or multilayer perceptrons (MLP)—using a unified, mathematically principled optimization objective. DRKMs leverage dual and primal representations to enable efficient learning for both high-dimensional inputs and large datasets by projecting the kernel operations onto latent spaces and imposing orthonormality (Stiefel manifold) constraints at every layer. DRKMs have demonstrated state-of-the-art sample efficiency, energy savings, memory advantages, and disentanglement capabilities relative to shallow kernel machines, neural networks, and variational approaches, particularly when dealing with small sample sizes or high-dimensional data (Tonin et al., 2023, Tonin et al., 2020).

1. Architectural Principles

DRKMs are characterized by the hierarchical stacking of functional modules:

• Multiple unsupervised KPCA layers: Each layer operates in the dual, extracting $s_j$ principal components from its input via a (possibly data-dependent) kernel matrix $K^{(j-1)}$.
• A single supervised classification layer: Implemented either as an LSSVM (primal form) or via an MLP (nonlinear map), leveraging the final KPCA representation for supervised learning.
• Visible and hidden units: At the first layer, visible units correspond to raw inputs $x_i \in \mathbb{R}^d$. At subsequent layers, hidden representations $h_i^{(j)} \in \mathbb{R}^{s_j}$ become the input to the next KPCA or the classifier.
• Stiefel manifold constraints: All hidden feature matrices $H^{(j)} \in \mathbb{R}^{N \times s_j}$ are orthonormalized, i.e., $H^{(j)\top}H^{(j)} = I_{s_j}$ for every layer, ensuring the latent features are decorrelated and lie on $\mathrm{St}(s_j,N)$.
• Separation of dual and primal spaces: KPCA levels are computed in the dual, depending only on the kernel matrices of size $N \times N$ (sample size), not $d$. The classification head is always primal, allowing efficient training when the final latent dimensionality $s_n \ll N$ (Tonin et al., 2023).

2. Mathematical Formulation and Optimization Objective

The canonical DRKM optimization objective for classification, coupling unsupervised KPCA and supervised LSSVM layers, is:

$K^{(j-1)}$0

where $K^{(j-1)}$1 and $K^{(j-1)}$2 for deeper layers. The terms correspond to variance maximization in KPCA, least-squares SVM hinge loss in the primal, and ridge regularization, all subject to strict orthonormality of latent codes (Tonin et al., 2023).

An alternative joint objective supports a classification layer implemented as an MLP, replacing the LSSVM loss with cross-entropy (Tonin et al., 2023):

$K^{(j-1)}$3

with the same orthonormality constraints.

Each KPCA level seeks the top $K^{(j-1)}$4 eigenvectors of its kernel matrix, maximizing projected variance under Stiefel constraints; the classifier optimizes either the LSSVM or MLP objective in primal, backpropagating gradients through the deep kernel stack.

3. Inference, Training Algorithms, and Computational Analysis

Training proceeds via alternating steps across layers:

1. Kernel computation: For each KPCA level, construct the corresponding $K^{(j-1)}$5 kernel matrix (initially from raw data, then from latent codes).
2. Gradient computations: For latent codes $K^{(j-1)}$6, compute the gradient of the KPCA variance-maximization term and, at the top layer, include the supervised loss gradient.
3. Manifold retraction: After each update, project each $K^{(j-1)}$7 onto the Stiefel manifold using singular value decomposition (SVD), i.e., $K^{(j-1)}$8 after $K^{(j-1)}$9.
4. Classifier updates: For LSSVM, standard primal SGD or similar optimizers; for MLP, e.g., Adam (Tonin et al., 2023).

Memory and computational complexity are dominated by kernel matrices:

• Each dual KPCA level uses $x_i \in \mathbb{R}^d$0 memory; eigendecomposition is $x_i \in \mathbb{R}^d$1.
• The primal classifier head is $x_i \in \mathbb{R}^d$2 per pass and independent of base input dimension $x_i \in \mathbb{R}^d$3.
• DRKMs are thus efficient for $x_i \in \mathbb{R}^d$4 (since no explicit high-dim embeddings are stored), scalable to large $x_i \in \mathbb{R}^d$5 when leveraging approximate kernels or mini-batching.

For small datasets or where $x_i \in \mathbb{R}^d$6, both memory and compute are substantially reduced compared to conventional convolutional neural networks (CNNs), which must allocate per-pixel or per-feature parameters (Tonin et al., 2023).

4. Constrained DRKM and Disentangled Representation Learning

The Constr-DRKM variant augments standard DRKM by enforcing block-orthogonality between and within latent codes at each layer (Tonin et al., 2020):

• Block-orthonormality constraint: For a two-layer model, latent code matrices $x_i \in \mathbb{R}^d$7 and $x_i \in \mathbb{R}^d$8 satisfy

$x_i \in \mathbb{R}^d$9

enforcing intra- and inter-layer decorrelation.

• Quadratic-penalty relaxation: The constraint is softened via a penalty term in the joint objective, optimized by a continuation (warm start) scheme. The penalty weight $h_i^{(j)} \in \mathbb{R}^{s_j}$0 increases by a factor $h_i^{(j)} \in \mathbb{R}^{s_j}$1 across outer iterations, alternating with inner unconstrained minimizations via Adam or similar first-order solvers.
• Initialization: Deterministic layerwise KPCA initialization (taking top eigenvectors for each layer) yields high reproducibility and zero run-to-run variance in empirical scores for disentanglement, overcoming stochasticity in random starts.
• Representation hierarchy: Lower-layer principal components represent large-scale, low-frequency structures; deeper layers refine or recombine these into higher-level features, paralleling convolutional and deep Boltzmann architectures.

5. Empirical Results and Application Domains

• On small, high-dimensional datasets (e.g., ARCENE $h_i^{(j)} \in \mathbb{R}^{s_j}$2, $h_i^{(j)} \in \mathbb{R}^{s_j}$3), DRKM outperforms MLPs, LSSVMs, and even CNNs on several tabular/classification tasks, while achieving up to two orders of magnitude memory savings (e.g., $h_i^{(j)} \in \mathbb{R}^{s_j}$4 MB for DRKM vs $h_i^{(j)} \in \mathbb{R}^{s_j}$5 GB for CNN) (Tonin et al., 2023).
• On moderate-scale image tasks (e.g., MNIST with $h_i^{(j)} \in \mathbb{R}^{s_j}$6), two KPCA levels improve accuracy (81–85%) over single-level DRKM ($h_i^{(j)} \in \mathbb{R}^{s_j}$7) and reduce performance variance.
• Energy efficiency: DRKM models consume substantially less energy (Wh) than CNNs when matched for performance on the same tasks. Depth (i.e., stacking multiple KPCA layers) further decreases variance and increases robustness.
• In unsupervised representation learning, Constr-DRKM achieves disentanglement metrics (IRS, MIG, SAP) comparable to or better than $h_i^{(j)} \in \mathbb{R}^{s_j}$8-VAE under data-limited regimes ($h_i^{(j)} \in \mathbb{R}^{s_j}$9), with much lower sensitivity to hyperparameter choices and random initialization (Tonin et al., 2020).

Empirical findings are summarized in the following table:

Task/Data Regime	DRKM Performance	Key Comparison
Small, high-dim (ARCENE)	Outperforms MLP, LSSVM; beats CNN on tabular	40MB (DRKM) vs 5GB (CNN)
MNIST, $H^{(j)} \in \mathbb{R}^{N \times s_j}$0	Two levels: 81–85% acc.	One level: ~75% acc.
Energy consumption	$H^{(j)} \in \mathbb{R}^{N \times s_j}$1 CNN for same accuracy	Up to order of magnitude gain
Unsupervised, $H^{(j)} \in \mathbb{R}^{N \times s_j}$2	Disentanglement $H^{(j)} \in \mathbb{R}^{N \times s_j}$3 $H^{(j)} \in \mathbb{R}^{N \times s_j}$4-VAE	Lower variance, stable

6. Hyperparameter Sensitivity and Best Practices

• Constr-DRKM's disentanglement scores are insensitive to $H^{(j)} \in \mathbb{R}^{N \times s_j}$5 across wide ranges (0.01–25), whereas $H^{(j)} \in \mathbb{R}^{N \times s_j}$6-VAE's $H^{(j)} \in \mathbb{R}^{N \times s_j}$7 parameter markedly affects both mean performance and variance.
• DRKM exhibits low run-to-run and hyperparameter variance, especially when using deterministic KPCA initialization in each layer.
• Simple configurations—two layers, $H^{(j)} \in \mathbb{R}^{N \times s_j}$8, $H^{(j)} \in \mathbb{R}^{N \times s_j}$9, $H^{(j)\top}H^{(j)} = I_{s_j}$0, $H^{(j)\top}H^{(j)} = I_{s_j}$1, $H^{(j)\top}H^{(j)} = I_{s_j}$2, Adam optimizer ($H^{(j)\top}H^{(j)} = I_{s_j}$3 learning rate)—are recommended for high reproducibility.
• For large $H^{(j)\top}H^{(j)} = I_{s_j}$4, practical usage favors approximate kernel methods or data splitting to manage $H^{(j)\top}H^{(j)} = I_{s_j}$5 memory for the kernel matrices.

7. Context and Research Significance

DRKMs unify deep learning principles with nonparametric kernel methodology, enabling hierarchical representation learning and efficient classification on both high-dimensional and small-sample datasets, with strong guarantees of decorrelation via geometric constraints. Compared to deep neural networks, DRKMs deploy far fewer parameters for input representation, are more energy efficient, and offer competitive (or superior) statistical performance, particularly under data-scarce or high-dimensional conditions (Tonin et al., 2023, Tonin et al., 2020). In unsupervised settings, constrained DRKMs achieve robust disentanglement, rivaling established deep generative models while providing improved reproducibility and calibration of hyperparameters.

A plausible implication is that DRKM and its constrained variants may serve as a blueprint for future generative and discriminative models that seek to combine explainability, efficiency, and robust performance across a wide variety of high-dimensional statistical domains.