Relational Autoencoder (RAE)

• Relational Autoencoder (RAE) is a neural model that explicitly encodes relationships among data samples or latent representations through tailored regularization.
• RAEs integrate optimal transport, self-expression, and spectral losses to preserve pairwise and higher-order affinities, enhancing robustness and generalization.
• Empirical studies show RAEs achieve improved performance in knowledge base completion, clustering, and generative tasks by maintaining relational structure.

A Relational Autoencoder (RAE) is an autoencoder variant distinguished by an explicit parameterization or regularization of relations, either between data samples, entities (e.g., in a knowledge base), or between latent codes and a learnable structured prior. RAE architectures and frameworks have been developed to encourage latent representations that preserve, disentangle, or reveal essential relational structure present either in the input space, latent space, or among model parameters—yielding improved robustness, interpretability, or compositional generalization in downstream tasks. RAE encompasses a spectrum of models, including relational regularized autoencoders with Gromov–Wasserstein structures, self-expression autoencoders for representation learning, relation-preserving feature extractors, and joint factorization models in knowledge representation.

1. Architectural Motivations and Theoretical Foundations

The common goal underlying RAEs is to encode or explicitly regularize the relationships among entities, data points, or latent variables, often going beyond conventional sample-wise reconstruction. This can mean:

• Compressing high-parameter relational models (e.g., knowledge base relation matrices) into low-dimensional codes that reflect compositional structure (Takahashi et al., 2018).
• Preserving pairwise or higher-order affinities among data samples in the latent space, matching or reconstructing the relational geometry of the input space (Meng et al., 2018, Kang et al., 2020).
• Enforcing consistency or proximity, often via optimal transport-based distances (Gromov–Wasserstein, Fused Wasserstein) between aggregate posterior and prior distributions in latent variable generative models (Xu et al., 2020, Nguyen et al., 2020).
• Estimating or supervising with explicit pairwise relation matrices (e.g., self-expression coefficients) as a core module (Kang et al., 2020).
• Propagating relational information in a graph-structured context, using message-passing neural networks or variational methods (Li et al., 2020).

The RAE framework is thus not restricted to a particular mathematical machinery, but shares the aim of relationally structuring the learning process—through architectural choices, loss terms, and latent space geometry.

2. Representative Model Classes and Formulations

2.1. Joint Training with a Relation Autoencoder for Knowledge Bases

In (Takahashi et al., 2018), KB triples $\langle h,r,t\rangle$ are modeled using $d$-dimensional entity vectors and a full $d\times d$ matrix $M_r$ per relation. RAE compresses $M_r$ into a sparse, low-dimensional code $\mathbf{c}_r=A \operatorname{vec}(M_r)$ (with ReLU nonlinearity) and reconstructs via $B\mathbf{c}_r$. The joint objective is: $$\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2 - \sum_{r\in\mathcal{R}} \gamma\|M_r^\top M_r - \frac{1}{d}\operatorname{tr}(M_r^\top M_r)I\|_F^2$$ where $\mathcal{L}_1$ is a KB (triple and path) loss (with NCE), $\mathcal{L}_2$ is an autoencoder contrastive loss over relations, and the last term encourages weak orthogonality in $M_r$.

Crucially, this approach preserves and even discovers compositional constraints ($M_{r_1}M_{r_2}\approx M_{r_3}$ for path and relation matching) by including multi-hop paths and learning from distributions rather than hard rules.

2.2. Relational Regularized Autoencoders with Optimal Transport

In (Xu et al., 2020, Nguyen et al., 2020), RAE is framed as minimizing a reconstruction loss plus a relational discrepancy (often Fused Gromov–Wasserstein, FGW) between the aggregated posterior $q(z)$ and a learnable prior $p(z)$, parameterized as a mixture of Gaussians: $$\min_{\theta,\phi,p(z)}\; \mathbb{E}_{x\sim p_{\mathrm{data}}}\,\|x-G_\theta(E_\phi(x))\|^2 + \lambda\,D(q(z)\|p(z))$$ Advancements such as Sliced FGW, Spherical Sliced FGW (SSFG), and their variants (MSSFG, PSSFG) focus the regularization on important directions or subpopulations in the latent space, providing non-negative, symmetric pseudo-metric constraints that enhance generative and clustering performance (Nguyen et al., 2020).

2.3. Self-Expression and Pairwise Relation Guidance

"Relation-Guided Representation Learning" (Kang et al., 2020) develops an RAE in which the latent code $Z$ is processed by a learnable self-expression layer $C\in\mathbb{R}^{n\times n}$, enforcing $z_i = \sum_j C_{ij}z_j$. The objective combines reconstruction, locality-preserving (spectral) loss, subspace consistency (reconstruction in latent and input space via $C$), and sparsity/regularity on $C$. This design directly models inter-sample affinities during representation learning.

2.4. Pairwise Similarity Preservation in Feature Learning

(Meng et al., 2018) proposes an RAE penalizing the difference between thresholded input similarity matrix $R(X) = XX^\top$ and output similarity $R(X') = X'X'^\top$. The loss

$$L_{\text{RAE}} = (1-\alpha)\|X-X'\|_F^2 + \alpha\|\tau_t(XX^\top)-\tau_t(X'X'^\top)\|_F^2$$

yields codes with improved robustness for classification and is easily extended to sparse, denoising, and variational autoencoder settings by adding the relational consistency term.

2.5. Relational Variational Graph Autoencoders

In "R-VGAE" (Li et al., 2020), relations on a large heterogeneous concept-resource graph are encoded via a relational GCN and variational posterior, with a decoder that reconstructs adjacency using a DistMult-style scoring. This arrangement captures prerequisite relationships in an unsupervised setting, leveraging multi-relation message passing and latent probabilistic modeling.

3. Optimization Strategies and Relational Regularization

RAE architectures employ joint or alternating optimization of encoder, decoder, relational parameter (e.g., $C$, $p(z)$, GMM parameters), and in some cases direction parameters for slicing distributions (SSFG). For knowledge bases (Takahashi et al., 2018), separate learning rates are used for losses over entity/relation parameters and autoencoder weights. For graph-structured and OT-based RAEs, variants of Sinkhorn optimization, stochastic gradient descent, or proximal-gradient updates are used to solve transport or regularization terms (Xu et al., 2020).

Regularization hyperparameters (e.g., $\alpha,\beta,\gamma,\tau,\lambda$) mediate the trade-off between reconstruction fidelity and relational or structural constraints, and their sensitivity is often explored in ablation studies (Xu et al., 2020, Meng et al., 2018, Kang et al., 2020).

4. Empirical Properties and Performance Benchmarks

Relational autoencoder models have demonstrated gains across a breadth of metrics:

• In knowledge base completion, joint training with an autoencoder achieves state-of-the-art or near-state-of-the-art mean rank on WN18, FB15k, FB15k-237 benchmarks, particularly improving discrimination of difficult "long tail" queries (Takahashi et al., 2018).
• On clustering tasks, relation-guided RAEs deliver marked accuracy improvements (e.g., >98% on EYaleB, >97% on COIL20 with locality and subspace losses), and ablation removing relational terms leads to 4–6% accuracy loss (Kang et al., 2020).
• Feature extraction experiments show consistent reduction in reconstruction loss and classification error, e.g., RAE (vs. vanilla autoencoder baseline) achieves lower MSE and up to 4–5% lower classification error on MNIST and CIFAR-10 (Meng et al., 2018).
• Relational regularization via FGW, SSFG, and their variants yields improved generative modeling performance (e.g., lower Fréchet Inception Distance and mean squared error on MNIST and CelebA) compared to VAE/WAE baselines. Spherical and mixture variants further improve alignment and diversity in latent representations (Nguyen et al., 2020).
• On prerequisite chain prediction, R-VGAE models surpass deep embedding and baseline graph representation learning methods by up to 10% in accuracy and F1, highlighting the value of relationally-guided variational encoders in graph-based unsupervised tasks (Li et al., 2020).

Empirical observation emphasizes the importance of balancing relational and sample-wise losses and choosing model capacity (e.g., hidden code dimension, concentration of slicing distributions) appropriate for the task.

5. Interpretability, Compositionality, and Analysis of Learned Representations

RAEs are notable for inducing interpretable and often sparse latent codes. In (Takahashi et al., 2018), dimensions of $\mathbf{c}_r$ align with interpretable semantic clusters (e.g., currency vs. film-related relations); heatmaps reveal near one-hot or block-diagonal patterns, reflecting underlying ontology structure. Relational regularization (e.g., pairwise matrix penalties, Gromov–Wasserstein distances) leads to latent spaces that better preserve manifold geometry, result in more globally consistent embeddings, and enable unsupervised discovery of compositional relationships ($M_1M_2\approx M_3$) (Takahashi et al., 2018, Meng et al., 2018). Visual and quantitative evaluation (e.g., t-SNE, spectral affinity matrices) further confirms that relational autoencoders promote clustering, smoothness, and generalization, especially in out-of-sample and multi-view contexts (Kang et al., 2020, Xu et al., 2020).

6. Extensions and Theoretical Properties

RAE frameworks have been extended via:

• Advanced regularization incorporating power-spherical or mixture von Mises–Fisher distributions (PSSFG/MSSFG) to exploit important directions in high-dimensional spaces, offering improvements in convergence and diversity (Nguyen et al., 2020).
• Multi-view and co-training setups, with cross-space relational Gromov–Wasserstein regularization enabling effective alignment of heterogeneous and unpaired autoencoder domains (Xu et al., 2020).
• Theoretical results guarantee (pseudo-)metric properties and statistical convergence for relational discrepancy regularization under appropriate conditions, with no curse of dimensionality in expectation (Nguyen et al., 2020).
• The RAE paradigm encompasses both probabilistic (VAE-style) and deterministic (WAE-style) encoders, with approximations or exact FGW for GMM and empirical distributions (Xu et al., 2020, Nguyen et al., 2020).

7. Distinctions from Related Models and Outlook

Distinct from classical autoencoders or variational autoencoders, RAEs explicitly encode, regularize, or reconstruct relationships among samples, structured objects, or latent variables, often learning both the representations and the structure (prior, similarity, cluster assignments) jointly. RAEs thus subsume models integrating affinity learning, manifold alignment, or compositional factorization as special cases, but are characterized by end-to-end differentiable architecture, learnable relational modules, and mathematical grounding in optimal transport, factorization, and spectral graph theory.

This relational principle has led to improvements in clustering, generative modeling, knowledge base reasoning, transfer, and multi-view learning, and invites further research into scalable regularization, interpretability, and applications to novel relational and graph-structured domains (Takahashi et al., 2018, Meng et al., 2018, Kang et al., 2020, Xu et al., 2020, Nguyen et al., 2020, Li et al., 2020).