Deep Kernelized Autoencoder (dkAE)

• Deep Kernelized Autoencoder (dkAE) is a neural network that integrates user-specified kernel priors to achieve controllable, similarity-preserving representations.
• The architecture jointly optimizes reconstruction loss and kernel alignment, enabling explicit mappings between input space and kernel-induced feature space.
• Empirical results on datasets like MNIST and CIFAR-10 demonstrate dkAE’s competitive performance in classification, denoising, and visualization compared to traditional methods.

A Deep Kernelized Autoencoder (dkAE) is a neural network architecture that generalizes the conventional autoencoder framework by explicitly incorporating user-specified kernel structure into the learned representation. Through joint optimization of reconstruction loss and a kernel alignment objective on the code (latent) representations, the dkAE enables the explicit approximation of mappings from input space to kernel-induced feature space and vice versa. This structure provides controllable similarity-preserving embeddings and supports efficient out-of-sample extensions, while retaining non-linear reconstruction capabilities (Kampffmeyer et al., 2017, Kampffmeyer et al., 2018).

1. Architecture and Mapping Functions

The dkAE architecture extends the standard stacked autoencoder by integrating an additional kernel-alignment term. Let $x \in \mathbb{R}^d$ denote an input vector. The encoder mapping is given by

$$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$

and the decoder by

$D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$

where $\sigma(\cdot)$ is a nonlinear activation (e.g., sigmoid), $W_E \in \mathbb{R}^{N_c \times d}$ and $W_D \in \mathbb{R}^{d \times N_c}$ are weights, and $N_c$ is the code dimension. In practice, a deep, stacked architecture is used, typically of the form $$d \rightarrow 500 \rightarrow 500 \rightarrow 2000 \rightarrow N_c$$, with tied weights $W_D = W_E^\top$ and unsupervised pretraining.

The encoder $E(x)$ approximates the (often implicit) mapping $$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$0 associated with the user-specified kernel, while the decoder $$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$1 serves as an explicit back-projection from code space back to input space, achieving a primal-dual formulation absent in conventional kernel methods (Kampffmeyer et al., 2017).

2. User-Specified Kernel Priors

The dkAE requires a positive semi-definite (PSD) kernel matrix $$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$2, specified over the training set $$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$3. In the canonical implementation, this kernel is constructed using the Probabilistic Cluster Kernel (PCK):

$$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$4

where $$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$5 is the posterior over mixture components from a Gaussian Mixture Model with $$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$6 components and random initialization $$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$7, and $$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$8 is a normalization constant (Kampffmeyer et al., 2017, Kampffmeyer et al., 2018). Arbitrary kernel matrices can be supplied as the prior $$E(x; W_E) = \sigma(W_E x + b_E) \in \mathbb{R}^{N_c}$$9, allowing users to inject specific notions of similarity into the learned embedding.

3. Loss Functions and Training Objective

The dkAE trains on a composite loss consisting of a reconstruction term and a kernel alignment term. The per-sample reconstruction loss is

$D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$0

The kernel alignment loss is based on aligning the Gram matrix of code representations $D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$1, where $D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$2, to the prior kernel $D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$3:

$D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$4

with the kernel alignment score $D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$5.

The overall training objective is

$D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$6

where $D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$7 controls the trade-off between input reconstruction and kernel alignment. When $D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$8, the model reduces to a standard autoencoder; increasing $D(h; W_D) = \sigma(W_D h + b_D) \in \mathbb{R}^d$9 increases alignment to the kernel prior at the expense of reconstruction. Empirically, $\sigma(\cdot)$0 offers a favorable balance (Kampffmeyer et al., 2017, Kampffmeyer et al., 2018).

4. Optimization and Training Procedure

The dkAE is trained using mini-batch stochastic optimization (batch size $\sigma(\cdot)$1, e.g., $\sigma(\cdot)$2), evaluating the loss on per-batch code Gram matrices $\sigma(\cdot)$3 and corresponding kernel submatrices $\sigma(\cdot)$4:

$\sigma(\cdot)$5

Layer-wise unsupervised pretraining is performed for 30 epochs per layer, with the kernel alignment term already active. Full network fine-tuning is then conducted for 100 epochs using the Adam optimizer, Glorot-uniform initialization, and weight-tying for stability (Kampffmeyer et al., 2017, Kampffmeyer et al., 2018). This mini-batch strategy effectively keeps per-batch memory and computational costs at $\sigma(\cdot)$6, enabling scalability well beyond traditional kernel approaches.

5. Explicit Kernel-Feature Embedding and Inverse Mapping

A principal advantage of dkAE is the provision of explicit forward and reverse mappings between the input space and the kernel-induced feature (RKHS) space. The encoder $\sigma(\cdot)$7 yields representations $\sigma(\cdot)$8 that approximate $\sigma(\cdot)$9, supporting out-of-sample extension and the direct application of linear methods (e.g., PCA, SVM) in the learned code space. The decoder $W_E \in \mathbb{R}^{N_c \times d}$0 approximates a pre-image, allowing inverse transformations from code vectors back to the input domain. This bidirectionality enables nonlinear transformations, denoising, and traversals in input space following manipulations in the kernel-aligned code domain (Kampffmeyer et al., 2017, Kampffmeyer et al., 2018).

6. Empirical Results and Performance Characteristics

Experiments conducted on MNIST, CIFAR-10, Cloud, and Reuters datasets in (Kampffmeyer et al., 2017, Kampffmeyer et al., 2018) demonstrate:

• Sensitivity to Loss Trade-off: As $W_E \in \mathbb{R}^{N_c \times d}$1 increases, kernel alignment loss $W_E \in \mathbb{R}^{N_c \times d}$2 decreases and reconstruction error $W_E \in \mathbb{R}^{N_c \times d}$3 increases, confirming the anticipated trade-off. The method is robust for $W_E \in \mathbb{R}^{N_c \times d}$4.
• Code Size Effect: Increasing $W_E \in \mathbb{R}^{N_c \times d}$5 (code dimension) uniformly reduces both losses, enhancing both reconstruction and kernel alignment.
• Reconstruction and Alignment: For $W_E \in \mathbb{R}^{N_c \times d}$6, $W_E \in \mathbb{R}^{N_c \times d}$7, reconstructions are visually faithful to inputs, and the code Gram matrix $W_E \in \mathbb{R}^{N_c \times d}$8 is well-aligned with the prior kernel $W_E \in \mathbb{R}^{N_c \times d}$9. Alignment to an "ideal" supervised kernel $W_D \in \mathbb{R}^{d \times N_c}$0 is superior for $W_D \in \mathbb{R}^{d \times N_c}$1 compared to $W_D \in \mathbb{R}^{d \times N_c}$2 or the code Gram of a conventional AE.
• Kernel PCA Approximation: For $W_D \in \mathbb{R}^{d \times N_c}$3 (number of classes), the dkAE's code Gram matrix $W_D \in \mathbb{R}^{d \times N_c}$4 approximates the prior better than kernel PCA (kPCA) with $W_D \in \mathbb{R}^{d \times N_c}$5 leading components. As $W_D \in \mathbb{R}^{d \times N_c}$6 approaches $W_D \in \mathbb{R}^{d \times N_c}$7, kPCA recovers $W_D \in \mathbb{R}^{d \times N_c}$8 exactly, but dkAE remains competitive under low-rank constraints.
• Classification: Linear SVMs trained on code space (cSVM) generally outperform both kernel SVMs in input space and standard linear SVMs. For example, on MNIST: cSVM 94.8%, kSVM 93.8%, linear SVM 90.6%. Supervised alignment further increases these results.
• Denoising: On noisy two-class MNIST data, dkAE+PCA+decode achieves mean squared error (MSE) $W_D \in \mathbb{R}^{d \times N_c}$9 compared to kPCA+pre-image $N_c$0, with visually sharper denoised outputs.
• Visualization and Traversals: Code-space representations support effective dimensionality reduction and visualization (e.g., 1-NN in 2D after PCA yields 39.6% on MNIST, superior to ISOMAP and KECA). Linear interpolations in code space decode to smooth input-space transitions, illustrating controllable generation and class morphing (Kampffmeyer et al., 2017, Kampffmeyer et al., 2018).

Method	MNIST cSVM	kSVM	Linear SVM	CIFAR-10 cSVM	1-NN in 2D
dkAE	94.8%	93.8%	90.6%	38.17%	39.6%
KECA	-	29.5%	-	-	29.5%
ISOMAP	-	-	-	-	36.8%
AE+PCA	-	-	-	-	30.5%
DAE+PCA	-	-	-	-	31.2%

Select performance metrics from (Kampffmeyer et al., 2018)

7. Limitations and Practical Considerations

dkAE inherits several practical constraints:

• Batch Memory and Computational Trade-offs: Per-batch $N_c$1 memory usage limits the maximum practical batch size $N_c$2, although this is still considerably more scalable than $N_c$3 storage required by classical kernel methods.
• Dependency on Kernel Selection: The kernel prior $N_c$4 critically determines the geometry of the learned embedding. The final encoding quality and downstream task performance are influenced by this choice; any PSD kernel is admissible.
• Generalization and Hyperparameter Sensitivity: The method is robust to $N_c$5 and $N_c$6 within standard ranges, but extremely large network sizes or loss trade-offs may necessitate validation. More comprehensive evaluations of out-of-sample generalization and performance on deeper unsupervised tasks remain open problems (Kampffmeyer et al., 2017).
• Flexible but not Universally Optimal: While dkAE empirically matches or outperforms kPCA and standard autoencoders on reconstruction, classification, and denoising, explicit denoising autoencoders can yield lower MSE when trained specifically for noise removal (Kampffmeyer et al., 2018).

• (Kampffmeyer et al., 2017) "Deep Kernelized Autoencoders"
• (Kampffmeyer et al., 2018) "The Deep Kernelized Autoencoder"