Tensorized Autoencoders (TAEs)

• Tensorized Autoencoders (TAEs) are advanced models that integrate structured, higher-order latent representations using tensor algebra to capture data heterogeneity and disentangled factors.
• They leverage innovations such as cluster-specific encoders/decoders, deep tensor decompositions, and tensor ring constraints to enforce structured priors and improve feature separation.
• TAEs have demonstrated superior performance in unsupervised clustering, denoising, compressive sensing, and tensor completion, often outperforming traditional autoencoders in accuracy and efficiency.

Tensorized Autoencoders (TAEs) generalize conventional autoencoders by introducing structured, often higher-order latent representations rooted in tensor and multiway algebra. They provide enhanced capacity for modeling data heterogeneity, structured priors, and disentangled factors by leveraging architectural innovations ranging from cluster-specific encoders/decoders to tensor-product or tensor-ring constrained latent spaces. TAEs span a spectrum of realizations, including multi-autoencoder meta-algorithms for cluster-specific subspaces, deep tensor decomposition networks, tensor-variate latent VAEs, and models using toroidal and multilinear latent representations. Their principal applications include unsupervised clustering, denoising, compressive sensing, tensor completion, structured disentanglement, and improved generative modeling.

1. Mathematical Formulation and Model Classes

TAEs encompass a family of architectures unified by their use of tensor-valued or multi-branched latent representations:

• Cluster-specific TAEs: Given data $X\in\mathbb{R}^{d\times n}$ and $k$ clusters, the model maintains $k$ sets of encoder-decoder pairs $\{g_{\Psi_j},f_{\Phi_j}\}_{j=1}^k$, a soft assignment matrix $S\in[0,1]^{k\times n}$ with $\sum_{j=1}^k S_{j,i}=1$, and cluster centers $C_j$. The TAE loss, combining reconstruction and latent norm regularization, is

$$\mathcal{L} = \sum_{i=1}^n \sum_{j=1}^k S_{j,i}\Big\|\big(X_i - C_j\big) - f_{\Phi_j}\big(g_{\Psi_j}(X_i - C_j)\big)\Big\|^2 + \lambda\sum_{i,j} S_{j,i}\|g_{\Psi_j}(X_i-C_j)\|^2\text{.}$$

In the linear case with $U_j$, $V_j$ as projection and reconstruction matrices, optimization can be conducted over matrix-tensors $\mathcal{U}, \mathcal{V}$ (Esser et al., 2022).

• Tensor Decomposition via Deep Variational Autoencoders: Each entry of a data tensor $\mathcal{X}(i_1, ..., i_D)$ is generated by first selecting $D$ latent mode vectors $U^{d}_{i_d:}$, which are concatenated and passed through a neural network decoder to output mean and variance per entry. The generative model replaces multilinear CP decomposition with a neural parameterization:

$$\mathcal{X}(i_1, ..., i_D) \sim \mathcal{N}\big(\mu(U^{1}_{i_1:}, ..., U^{D}_{i_D:}), \sigma^2(U^{1}_{i_1:}, ..., U^{D}_{i_D:})\big)\text{.}$$

The evidence lower bound (ELBO) is optimized to fit both decoder and variational posteriors (Liu et al., 2016).

• Tensor-variate Gaussian Process Prior VAEs: The latent $Z$ is a tensor with a tensor-normal Gaussian process prior:

$$p(Z) = \mathcal{TN}(Z; 0, K^{(1)}, ..., K^{(M')})\text{.}$$

The variational posterior and the prior both share separable covariance across modes. The encoder and decoder employ convolutional architectures suitable for the data’s spatiotemporal correlation structure (Campbell et al., 2020).

• Tensor Ring-constrained Latent TAEs: In compressive sensing or structure-exploiting settings, latent codes from all dataset examples are arranged in a K-mode tensor and constrained by a tensor ring factorization, explicitly encoding structured variation across known attributes. The learning objective synchronizes free encoder output, tensor ring codes, and measurement consistency through a multi-term loss (Hyder et al., 2023).
• Tensor Product on Torus (Tⁿ-VAE): The latent code is constructed from D unit vectors $m_a \in S^1$ (i.e., points on the torus), and the full code is their outer product, vectorized together with an orienting term:

$$V(m_1, ..., m_D) = [\text{vec}(m_1 \otimes ... \otimes m_D); m_1^0, ..., m_D^0] \in \mathbb{R}^{2^D+D}\text{.}$$

Training employs an ELBO objective with KL regularization of the pre-normalized latent coordinates to encourage a uniform prior (Rotman et al., 2022).

2. Meta-Algorithm and Training Dynamics

Clustered TAEs use a block-coordinate descent meta-algorithm:

1. Cluster assignment and center initialization with k-means++.
2. Alternating updates:
   • Encoder and decoder parameters for each cluster are updated via gradient steps on the global loss.
   • Assignments $S$ updated via projected gradient steps or discrete assignments minimizing local reconstruction+regularization cost.
   • Cluster centers updated as soft-assignment-weighted means.

For tensor-decomposition VAEs, stochastic variational inference is employed, often with the reparameterization trick and the Adam optimizer for both model and variational parameters. In the tensor ring bottleneck context, both the network weights and TR-cores are optimized, using Adam and SGD, respectively. For toroidal latents, the coordinate-wise Gaussian parameters are regularized to $\mathcal{N}(0,I)$, normalized to the unit circle, and backpropagation-based optimization is used (Esser et al., 2022, Hyder et al., 2023, Rotman et al., 2022).

3. Theoretical Results and Inductive Bias

The clustered TAE can provably recover the correct principal components for each data cluster. For linear encoders and decoders ($g_{\Psi_j}(X) = U_j X,\ f_{\Phi_j}(Z)=V_j Z$), if $U_j$ is orthonormal and $\lambda>0$, then the global minimum corresponds to:

• Hard cluster assignments ($S_{j,i}\in\{0,1\}$),
• Cluster centers at the mean of their assigned points,
• $U_j$’s rows spanning the top $h$ eigenspace of the within-cluster covariance $\hat\Sigma_j$.

This directly generalizes classical k-means + PCA, but with deep or convolutional mappings and soft/hard assignment learning (Esser et al., 2022).

Tensor-product and tensor-variate VAEs imbue the model with explicit factorizations or correlation structures: e.g., toroidal codes encourage disentanglement by ensuring independent, periodic structure in each latent factor; Kronecker-structured GPs encode spatial or temporal smoothness in the latent. Tensor ring bottlenecks limit the expressivity of the latent space, enforcing representation “axes” corresponding to known dataset attributes (Campbell et al., 2020, Rotman et al., 2022, Hyder et al., 2023).

4. Architectural Variants and Relation to Classical Models

TAEs generalize standard autoencoders by:

Variant	Latent Structure	Typical Application/Benefit
Clustered TAEs	k encoder/decoder pairs	Clustering, denoising, modeling heterogeneity
VAE-based tensor decomposition	Mode-wise latent vectors; NN decoder	Nonlinear tensor completion, multilinear data
Tensor-GP prior VAEs	Tensor-valued latent; GP priors	Structured, spatial-temporal generative models
TR-constrained latent TAEs	K-mode tensor ring in bottleneck	Compressive sensing, attribute disentanglement
Tⁿ-VAE (torus/tensor product)	Outer products of unit circles (tori)	Disentanglement, interpretable representations

TAEs recover and extend classical methods: in the limit of identical clusters, the clustered TAE reduces to the standard AE; for linear, isotropic covariances, they converge to k-means plus clusterwise PCA. Nonlinear TAEs using neural network decoders generalize CP and Tucker decompositions to arbitrary interactions (Esser et al., 2022, Liu et al., 2016).

5. Empirical Performance and Experimental Evaluation

TAEs yield improvements across clustering, denoising, disentanglement, and compressive sensing tasks.

• Clustering (Palmer Penguins, Iris, MNIST): The TAE consistently outperforms vanilla AEs and AE+k-means. E.g., on Palmer Penguins ($k=3$), k-means ARI ≈ 0.45, AE+k-means ≈ 0.60, TAE ≈ 0.85; on Iris, TAE ARI ≈ 0.75 vs AE+KM ≈ 0.70 (Esser et al., 2022).
• Denoising: With significant Gaussian noise, TAE reduces MSE by up to an order of magnitude relative to a standalone AE; e.g., on MNIST with CNNs, AE MSE ≈ 0.10, TAE ≈ 0.03 (Esser et al., 2022).
• Compressive sensing / Recovery: Tensor ring TAEs dominate classical and generative prior-based self-supervised methods by 1–5 dB in PSNR across datasets/articulations (Hyder et al., 2023).
• Tensor completion: VAE-based TAEs achieve 10–30% lower test RMSE than CP/Tucker models on chemometrics data (Liu et al., 2016).
• Disentanglement: T⁸-VAE outperforms β-VAE, DIP-VAE-II, Factor-VAE in DC-score across all reported benchmarks, e.g., Teapots: T⁸-VAE DC=0.623 vs Factor-VAE=0.389; 2D-Shapes: 0.702 vs 0.517 (Rotman et al., 2022).
• Structured generative models: Tensor-GP VAEs produce better or equivalent negative log-likelihood than vanilla VAE, with strong gains for data possessing explicit spatial/temporal structure (Campbell et al., 2020).

6. Limitations and Considerations

TAE variants entail increased model complexity and introduce challenges:

• Parameter Efficiency: Clustered TAEs require $k$ separate autoencoders, introducing scaling issues for large $k$.
• Hyperparameter Tuning: Choice of number of clusters, tensor ranks, TR dimensions, and GP kernel hyperparameters must be empirically tuned and can be data-dependent (Esser et al., 2022, Hyder et al., 2023, Campbell et al., 2020).
• Supervision Requirements: TR bottleneck methods require known labels for dataset “axes of variation”; models may not extend directly to fully unsupervised settings unless prior structure is accessible (Hyder et al., 2023).
• Inference Cost: Tensor-normal and GP-latent VAEs require the computation (and inversion) of large covariance matrices; scalable parameterizations (e.g., Markovian Cholesky factors) are essential (Campbell et al., 2020).
• Disentanglement Sensitivity: Too small a torus dimension in Tⁿ-VAE limits disentanglement, while overly strict KL regularization may couple latent circles (Rotman et al., 2022).

7. Future Directions and Research Outlook

Ongoing work explores more expressive but parameter-efficient versions of TAEs, integration with attention mechanisms, and hybridization with discrete latent variables. There is continuing investigation into scalable inference for high-order tensor-valued latents, unsupervised learning of dataset axes, and theoretical properties of tensor-induced regularization in deep generative architectures. The adaptability of TAEs for spatiotemporal, multi-modal, and structured medical datasets is under active development.

---

References:

• "Improved Representation Learning Through Tensorized Autoencoders" (Esser et al., 2022)
• "Tensor Decomposition via Variational Auto-Encoder" (Liu et al., 2016)
• "tvGP-VAE: Tensor-variate Gaussian Process Prior Variational Autoencoder" (Campbell et al., 2020)
• "Compressive Sensing with Tensorized Autoencoder" (Hyder et al., 2023)
• "Unsupervised Disentanglement with Tensor Product Representations on the Torus" (Rotman et al., 2022)