Toroidal AutoEncoder Models

• Toroidal AutoEncoder is a generative model that uses a torus-shaped latent space where each dimension represents a periodic variable, ideal for angular or rotational data.
• It utilizes both deterministic and VAE-style architectures with polar coordinate transformations, tensor-product codes, and specialized spring loss regularizations to enforce uniform latent distributions.
• The model enables precise disentanglement and supports multiple interpolation paths, offering practical benefits in tasks like morphing and learning periodic generative factors.

A Toroidal AutoEncoder is a class of generative autoencoders in which the latent space is constructed as a torus, typically $\mathbb{T}^d = (S^1)^d$, with each latent component constrained to represent a periodic variable. This topology is enforced using specific regularization mechanisms that ensure the distribution of latent variables is uniform on the torus, providing an inductive bias tailored for periodic or angular generative factors. The resulting models offer exact, topology-aware disentanglement properties and support novel interpolation behaviors unavailable in conventional Euclidean latent spaces (Mikulski et al., 2019, Rotman et al., 2022).

1. Mathematical Structure of Torus Latent Spaces

The fundamental object is the $d$-dimensional torus, defined as the Cartesian product of $d$ copies of the unit circle:

$T^d = S^1 \times S^1 \times \cdots \times S^1.$

A coordinate system for $T^d$ is realized by taking $d$ angles $\varphi = (\varphi_1,\ldots,\varphi_d)$, each in $[-\pi,\pi]$, with the identification

$$(\varphi_1, \ldots, \varphi_i = -\pi, \ldots, \varphi_d) \sim (\varphi_1, \ldots, \varphi_i = +\pi, \ldots, \varphi_d)$$

for each $i$. Arithmetic in the latent space is performed modulo $d$0 coordinate-wise, implementing the quotient structure $d$1 (Mikulski et al., 2019). This endows the latent with global periodicity, ensuring that the representation can encode factors such as rotation or hue naturally.

2. Model Architectures and Encoding Mechanisms

Two principal architectures have been described for toroidal autoencoders:

a) Deterministic Toroidal AutoEncoder

The encoder is a feedforward network with final output of dimension $d$2, interpreted as $d$3 pairs $d$4 representing points in $d$5. Each pair is converted to polar coordinates: $d$6 yielding radii and angle tuples. The decoder consumes $d$7 and reconstructs the input. All toroidal regularization is imposed via the angular coordinates $d$8 (Mikulski et al., 2019).

b) Stochastic (VAE-style) Toroidal AutoEncoder with Tensor Products

The encoder emits for each circle $d$9 a pair of Gaussian parameter vectors $d$0. The reparameterization trick is applied to sample $d$1, which is normalized onto $d$2: $d$3 yielding $d$4 independent circle elements. To enforce disentanglement, the model constructs the tensor (outer) product $d$5 along with linear orientation features, yielding a latent code of size $d$6. The decoder receives this representation and reconstructs the data (Rotman et al., 2022).

3. Training Objectives and Regularization

The objectives combine reconstruction fidelity with explicit regularization to enforce uniformity on the torus:

• Circular Spring Loss: For each angle coordinate, minibatch samples are imagined as points on a circle linked by springs. Ordering the minibatch angles $d$7 and connecting adjacent points and the wrap-around, the energy is

$d$8

Summing over $d$9 yields the toroidal spring loss $T^d = S^1 \times S^1 \times \cdots \times S^1.$0 (Mikulski et al., 2019).

• Total Objective: The combined loss is

$T^d = S^1 \times S^1 \times \cdots \times S^1.$1

where $T^d = S^1 \times S^1 \times \cdots \times S^1.$2 balances reconstruction with angular uniformity. As $T^d = S^1 \times S^1 \times \cdots \times S^1.$3, the model behaves as a vanilla autoencoder; as $T^d = S^1 \times S^1 \times \cdots \times S^1.$4, the latent angles fill the torus uniformly (at the expense of higher reconstruction error) (Mikulski et al., 2019).

• VAE-style KL regularization: In the stochastic, tensor-product variant, a KL term is added:

$T^d = S^1 \times S^1 \times \cdots \times S^1.$5

with $T^d = S^1 \times S^1 \times \cdots \times S^1.$6 controlling the trade-off as in $T^d = S^1 \times S^1 \times \cdots \times S^1.$7-VAE (Rotman et al., 2022).

4. Geometry, Interpolations, and Multiple Path Morphing

On $T^d = S^1 \times S^1 \times \cdots \times S^1.$8, the geodesic (minimal-distance) between two points $T^d = S^1 \times S^1 \times \cdots \times S^1.$9 is defined coordinate-wise: $T^d$0 The toroidal distance is

$T^d$1

Because of the periodic identification, there exist $T^d$2 distinct piecewise-linear geodesic paths connecting two points, depending on the integer wrap $T^d$3 in each coordinate.

Multiple-path morphing leverages this: for two encoded points, interpolations can wrap around the torus along any combination of coordinates: $T^d$4 for $T^d$5 or similar. Intermediate latent vectors are decoded, producing morphing sequences with distinct semantic transitions (e.g. traversing through different “edges” of the torus yields dramatically different intermediates) (Mikulski et al., 2019).

5. Disentanglement and Representation Metrics

Toroidal AutoEncoders are designed to separate generative factors. In the tensor-product construction, disentanglement is enforced by the fact that, by forming the full tensor product, each generative factor corresponds precisely to manipulation of a single $T^d$6 coordinate. The representation is a product state: $T^d$7 so that variation in $T^d$8 affects only a specific factor in the expanded code. This ensures vanishing “entanglement entropy” and guarantees exact factorization (Rotman et al., 2022).

Evaluation is performed via DCI metrics (Disentanglement, Completeness, Informativeness):

• Disentanglement ($T^d$9): Measures how each code dimension controls only one generative factor.
• Completeness ($d$0): Measures how each generative factor is captured primarily by a single code.
• Informativeness ($d$1): Measures prediction error of ground-truth factors from code.

Formally, for a lasso regression recovering $d$2 from code $d$3: $d$4 with $d$5 and $d$6 as precise entropy-based combinations over all $d$7 and $d$8. The DC-score is reported as $d$9 (Rotman et al., 2022).

6. Comparison with Gaussian-Variational Latent Models

Standard VAEs assume a latent $\varphi = (\varphi_1,\ldots,\varphi_d)$0. This leads to several contrasts:

Property	Gaussian VAE	Toroidal AutoEncoder
Latent topology	Euclidean, unbounded	Compact $\varphi = (\varphi_1,\ldots,\varphi_d)$1 manifold
Periodicity support	No native	Explicit, 1 circle per factor
Disentanglement	Imposed via penalties, never exact	Exact via product state
Regularizer	Various (TC, DIP, etc.)	KL on pre-normalized Gaussians or spring loss
Decoder input size	$\varphi = (\varphi_1,\ldots,\varphi_d)$2	$\varphi = (\varphi_1,\ldots,\varphi_d)$3 (tensor product schemes)
Handling of noncompact factors	Natural	Poor (doesn’t map to circle)

A key consequence is that Toroidal AutoEncoders strictly interpolate between training points (never extrapolate), inherently match periodic generative factors to circles, and enforce disentanglement by design. For nonperiodic, noncompact factors, the toroidal approach is less suitable (Rotman et al., 2022, Mikulski et al., 2019).

7. Empirical Results and Applications

Experiments have been conducted on datasets with known generative factors: MNIST, Teapots, 2dShapes, 3dShapes, and dSprites. On these, toroidal autoencoders using the tensor-product code ($\varphi = (\varphi_1,\ldots,\varphi_d)$4-VAE) outperform $\varphi = (\varphi_1,\ldots,\varphi_d)$5-VAE, DIP-VAE-II, and Factor-VAE in disentanglement, completeness, DC-score, and sometimes in FID/reconstruction, provided the number of circles $\varphi = (\varphi_1,\ldots,\varphi_d)$6 matches or exceeds the number of true generative factors.

Demonstrated applications include:

• Multiple-path morphing: The toroidal structure allows qualitatively distinct interpolation paths, visualized on MNIST to produce transitions that traverse different “edges” of latent space, producing diverse digit transformations.
• Learning periodic generative factors: For example, learning 2D rotation via a single $\varphi = (\varphi_1,\ldots,\varphi_d)$7-component, or potentially representing 3D rotations by augmenting with spherical coordinates.
• Topological feature learning: Enables the modeling of spaces with nontrivial topology, such as angular pose or color wheel, that are not well-captured by Euclidean latents.

Limitations include exponential growth of decoder input with $\varphi = (\varphi_1,\ldots,\varphi_d)$8 in the tensor-product scheme and difficulty with unbounded or nonperiodic generative factors (Rotman et al., 2022, Mikulski et al., 2019).