Tensorized GANs: Efficient Multilinear Models

• Tensorized GANs are generative models that employ tensor decompositions and multilinear representations to efficiently capture complex latent distributions.
• They leverage tensor-based latent priors like TRIP and direct tensorization of network layers to enhance multimodal generation and achieve significant parameter compression.
• These architectures improve interpretability, sampling efficiency, and high-resolution image synthesis while addressing issues such as mode collapse and hyperparameter sensitivity.

Tensorized Generative Adversarial Networks (GANs) define a family of generative models that leverage explicit tensor representations and multilinear algebra within the adversarial framework. These architectures apply tensor decomposition, multilinear parameterization, or tensor-based latent priors to enhance sample efficiency, induce structural regularity, enable model compression, and support richer multimodal generation or semantics in the latent space. The tensorization paradigm encompasses both architectural tensorization of neural network layers and tensor-based priors for the GAN latent space, yielding models with reduced parameter counts, improved expressivity, and increased interpretability relative to conventional vectorized approaches.

1. Tensor-Based Latent Priors in GANs

A significant tensorization strategy involves replacing the standard unimodal Gaussian latent prior in GANs with priors induced by tensor networks, such as the Tensor Ring Induced Prior (TRIP). Given a $d$-dimensional latent vector $z = (z_1,\ldots,z_d)$, TRIP models each $z_k$ as a mixture of $N_k$ Gaussians, indexed by discrete variables $s_k \in \{0,\ldots,N_k-1\}$, resulting in a exponentially large mixture model,

$$p(z) = \sum_{s_1=0}^{N_1-1} \cdots \sum_{s_d=0}^{N_d-1} p(s) \; p(z|s)$$

Here mixture weights $p(s)$ form a $d$-way tensor $$\hat{P} \in \mathbb{R}^{N_1 \times \cdots \times N_d}$$. To circumvent the exponential parameter growth, the mixture tensor is approximated via a nonnegative tensor ring decomposition with core tensors $Q_k \in \mathbb{R}^{N_k \times m_k \times m_{k+1}}$, yielding normalized lattice weights through

$$\tilde{P}[s] = \mathrm{Tr}\left( Q_1[s_1] Q_2[s_2] \cdots Q_d[s_d] \right), \quad p(s) = \tilde{P}[s] / Z, \; Z = \sum_s \tilde{P}[s]$$

Sampling $z$ for the generator thus draws from a highly multimodal $p_{\psi}(z)$ parameterized by the tensor ring cores and mixture parameters. This architecture, termed GAN-TRIP, keeps the generator and discriminator networks unchanged but replaces the latent prior. Optimization involves joint training of the generator, discriminator, and TRIP parameters, employing REINFORCE estimators for gradients through the non-reparameterizable discrete latent structure. Empirically, TRIP-parameterized GANs achieve improved FID scores (e.g., on CelebA: WGAN-GP baseline FID $54.71$, WGAN-GP-TRIP $52.86$; CIFAR-10: WGAN-GP FID $29.3$, WGAN-GP-TRIP $16.72$), reduce mode collapse, and support efficient sampling and storage for large $d$ (Kuznetsov et al., 2019).

2. Tensorization of Neural Network Layers

A complementary tensorization approach is direct replacement of dense fully connected layers in both the generator and discriminator by multilinear tensor layers. In "Tensorizing Generative Adversarial Nets" (Cao et al., 2017), the input to a tensor layer is an $N$-way tensor, transformed by $N$ factor matrices via successive mode-$n$ products,

$$Y = \sigma\left( X \times_1 U_1 \times_2 U_2 \cdots \times_N U_N + B \right)$$

where $U_n \in \mathbb{R}^{J_n \times I_n}$ are learned factors and $B$ is a bias tensor. This formulation generalizes the Tucker decomposition, embedding multi-way interactions while drastically reducing parameter counts. For example, the traditional parameter budget for two dense layers with $J$, $I$, $K$ as flattened input/output sizes is $J(I+K)$, versus $\sum_{n=1}^N (I_nJ_n + J_nK_n)$ for the tensor version; as $N$ grows, the reduction is exponential. On MNIST, tensorized GANs deliver a $35\times$ compression, reducing parameters from $429{,}000$ to $12{,}000$ with visual sample quality remaining competitive with larger baselines. The design preserves computational efficiency and supports backpropagation via mode-specific gradients, though optimal tensor ranks for expressivity and compression require tailored selection (Cao et al., 2017).

3. High-Resolution Image Generation via Tensor Super-Resolution

Deep tensor adversarial generative nets (TGAN) (Ding et al., 2019) integrate tensor representations both at the image level and within network operations to enable large-scale high-quality image synthesis. TGAN employs a cascade of:

1. DCGAN-like generator and discriminator on low-resolution images, where image tensors are constructed by stacking pixel-shifted copies, and local blocks ("folded" cubes) are operated on as third-order tensors.
2. A tensor super-resolution module, decomposed into tensor dictionary learning and sparse tensor coefficient learning via ISTA/FISTA. High/low-resolution patches are mapped between domains using tensor product ("t-product") operations, with dictionary update in the frequency domain.
3. Reconstruction of the output image via "unfolding" high-resolution tensors.

Empirically, TGAN achieves $8.5\times$ greater output resolution (e.g., $374\times374$ on PASCAL2) than conventional vectorized GANs and improves perceptual quality and inception scores versus adversarial autoencoders. The tensor architecture efficiently preserves spatial proximity and multiscale pattern structure during both adversarial and dictionary-learning phases (Ding et al., 2019).

4. High-Order Polynomial Generators and Tensor Parameterizations

PolyGAN (Chrysos et al., 2019) models the GAN generator as a high-order multivariate polynomial in the latent code $z$, parameterized by high-order tensors. The direct expansion,

$$x_i = G(z)_i = \beta_i + \sum_{n=1}^N W^{[n]}_{i, k_1, \ldots, k_n} z_{k_1} \cdots z_{k_n}$$

imposes an infeasible parameter cost, addressed by introducing low-rank tensor factorization—such as coupled CP or hierarchical CP decompositions—over the polynomial coefficient tensors $\{ W^{[n]} \}$. These decompositions render the generator as a stack of linear and elementwise blocks, omitting traditional nonlinear activations and achieving universal function approximation. In practice, PolyGANs inserted into DCGAN, SNGAN, or SAGAN architectures yield improved inception and FID scores on CIFAR-10 and ImageNet for minimal parameter overhead. The approach demonstrates that explicit tensorization can replace both nonlinearities and overparametrization while preserving or enhancing generative quality (Chrysos et al., 2019).

5. Tensor Methods for Interpretable Latent Space Analysis

Tensor component analysis and multilinear decomposition enable explicit semantic disentanglement in GAN latent spaces. Given the generator’s intermediate feature map tensor $\mathcal{Z} \in \mathbb{R}^{C \times H \times W}$, a multilinear Tucker-PCA decomposes these activations as

$$\mathcal{Z} \approx \mathcal{G} \times_1 U^{(C)} \times_2 U^{(H)} \times_3 U^{(W)},$$

where $U^{(C)}$ and $U^{(H)}, U^{(W)}$ separately capture style and geometric variation, respectively. A tensor-based regression maps variations in $\mathcal{Z}$ to corresponding latent offsets, enabling both linear and high-order (multiplicative) edits. This facilitates independently controllable edits to, e.g., hair color (style) and pose (geometry), extending the class of achievable image transformations. Quantitatively, this reduces attribute entanglement compared to baseline linear methods such as GANSpace or SeFa (Oldfield et al., 2021).

Similarly, τGAN applies higher-order SVD (HOSVD) to a tensor of latent codes extracted from StyleGAN, modeling semantic subspaces (person, expression, rotation) and enabling attribute transfer using alternating least-squares parameter estimation. Both vectorized and style-separated tensor models show benefits in reconstruction fidelity and semantic trajectory geometry—e.g., recovering star-shaped emotion manifolds and outperforming GANSpace and InterFaceGAN in transfer tasks (Haas et al., 2021).

6. Advantages, Practical Considerations, and Limitations

Tensorized GANs offer several advantages:

• Parameter Efficiency: Exponential reduction in parameters for both priors and layers, enabling model deployment on platforms with constrained resources (TGAN, PolyGAN).
• Expressive Structure: Ability to model multimodal priors (TRIP), high-order nonlinearities (PolyGAN), and preserve spatial and semantic relationships (TGAN, τGAN, (Kuznetsov et al., 2019, Chrysos et al., 2019, Ding et al., 2019, Haas et al., 2021)).
• Enhanced Sampling and Editing: Improved mode coverage, reduction in mode collapse due to rich lattice priors, interpretable editing in latent space via multilinear structure.

Trade-offs and limitations include:

• Hyperparameter Sensitivity: Rank selection in tensor decompositions and regularization weights require empirical tuning for optimal expressivity versus compression.
• Computational Overhead: While parameter count may decrease, some tensor operations (e.g., ring traces, t-products) can introduce nontrivial computation per forward pass.
• Specificity to Data Structure: The approach may require structural assumptions (e.g., availability of aligned, labeled data for latent subspace analysis as in τGAN).

Further research avenues involve extension to convolutional/attention layers, automated rank selection, and integration of more advanced tensor network structures.

7. Summary Table of Representative Tensorized GAN Variants

Model Framework	Tensorization Approach	Principal Benefit
GAN-TRIP (Kuznetsov et al., 2019)	Tensor ring prior over latent codes	Multimodal priors, mitigates mode collapse
TGAN (Cao et al., 2017)	Tensor layers via mode-$n$ multilinear maps	Parameter compression
TGAN (Ding et al., 2019)	Tensorized images and super-resolution	High-res generation, spatial structure
PolyGAN (Chrysos et al., 2019)	High-order polynomial generator, CP factor	Nonlinear expressivity, no activations
τGAN (Haas et al., 2021)	HOSVD factorization of embedded latent codes	Semantic attribute disentanglement

Tensorized GAN architectures represent a unifying, algebraically principled framework for enhancing both the efficiency and capability of generative adversarial models across latent distribution modeling, network parameterization, and post-hoc interpretability.