---
title: Relational Autoencoder (RAE)
url: https://www.emergentmind.com/topics/relational-autoencoder-rae
type: topic
---

# Relational Autoencoder (RAE)

A Relational Autoencoder (RAE) is an autoencoder variant distinguished by an explicit parameterization or regularization of relations, either between data samples, entities (e.g., in a knowledge base), or between latent codes and a learnable structured prior. RAE architectures and frameworks have been developed to encourage latent representations that preserve, disentangle, or reveal essential relational structure present either in the input space, latent space, or among model parameters—yielding improved robustness, interpretability, or compositional generalization in downstream tasks. RAE encompasses a spectrum of models, including relational regularized autoencoders with Gromov–Wasserstein structures, self-expression autoencoders for representation learning, relation-preserving feature extractors, and joint factorization models in knowledge representation.

## 1. Architectural Motivations and Theoretical Foundations

The common goal underlying RAEs is to encode or explicitly regularize the relationships among entities, data points, or latent variables, often going beyond conventional sample-wise reconstruction. This can mean:

- Compressing high-parameter relational models (e.g., knowledge base relation matrices) into low-dimensional codes that reflect compositional structure [1805.09547].
- Preserving pairwise or higher-order affinities among data samples in the latent space, matching or reconstructing the relational geometry of the input space [1802.03145, 2007.05742].
- Enforcing consistency or proximity, often via optimal transport-based distances (Gromov–Wasserstein, Fused Wasserstein) between aggregate posterior and prior distributions in latent variable generative models [2002.02913, 2010.01787].
- Estimating or supervising with explicit pairwise relation matrices (e.g., self-expression coefficients) as a core module [2007.05742].
- Propagating relational information in a graph-structured context, using message-passing neural networks or variational methods [2004.10610].

The RAE framework is thus not restricted to a particular mathematical machinery, but shares the aim of relationally structuring the learning process—through architectural choices, loss terms, and latent space geometry.

## 2. Representative Model Classes and Formulations

### 2.1. Joint Training with a Relation Autoencoder for Knowledge Bases

In [1805.09547], KB triples $\langle h,r,t\rangle$ are modeled using $d$-dimensional entity vectors and a full $d\times d$ matrix $M_r$ per relation. RAE compresses $M_r$ into a sparse, low-dimensional code $\mathbf{c}_r=A \operatorname{vec}(M_r)$ (with ReLU nonlinearity) and reconstructs via $B\mathbf{c}_r$. The joint objective is:
\[
\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2 - \sum_{r\in\mathcal{R}} \gamma\|M_r^\top M_r - \frac{1}{d}\operatorname{tr}(M_r^\top M_r)I\|_F^2
\]
where $\mathcal{L}_1$ is a KB (triple and path) loss (with NCE), $\mathcal{L}_2$ is an autoencoder contrastive loss over relations, and the last term encourages weak orthogonality in $M_r$.

Crucially, this approach preserves and even discovers compositional constraints ($M_{r_1}M_{r_2}\approx M_{r_3}$ for path and relation matching) by including multi-hop paths and learning from distributions rather than hard rules.

### 2.2. Relational Regularized Autoencoders with Optimal Transport

In [2002.02913, 2010.01787], RAE is framed as minimizing a reconstruction loss plus a relational discrepancy (often Fused Gromov–Wasserstein, FGW) between the aggregated posterior $q(z)$ and a learnable prior $p(z)$, parameterized as a mixture of Gaussians:
\[
\min_{\theta,\phi,p(z)}\; \mathbb{E}_{x\sim p_{\mathrm{data}}}\,\|x-G_\theta(E_\phi(x))\|^2 + \lambda\,D(q(z)\|p(z))
\]
Advancements such as Sliced FGW, Spherical Sliced FGW (SSFG), and their variants (MSSFG, PSSFG) focus the regularization on important directions or subpopulations in the latent space, providing non-negative, symmetric pseudo-metric constraints that enhance generative and clustering performance [2010.01787].

### 2.3. Self-Expression and Pairwise Relation Guidance

"Relation-Guided Representation Learning" [2007.05742] develops an RAE in which the latent code $Z$ is processed by a learnable self-expression layer $C\in\mathbb{R}^{n\times n}$, enforcing $z_i = \sum_j C_{ij}z_j$. The objective combines reconstruction, locality-preserving (spectral) loss, subspace consistency (reconstruction in latent and input space via $C$), and sparsity/regularity on $C$. This design directly models inter-sample affinities during representation learning.

### 2.4. Pairwise Similarity Preservation in Feature Learning

[1802.03145] proposes an RAE penalizing the difference between thresholded input similarity matrix $R(X) = XX^\top$ and output similarity $R(X') = X'X'^\top$. The loss
\[
L_{\text{RAE}} = (1-\alpha)\|X-X'\|_F^2 + \alpha\|\tau_t(XX^\top)-\tau_t(X'X'^\top)\|_F^2
\]
yields codes with improved robustness for classification and is easily extended to sparse, denoising, and variational autoencoder settings by adding the relational consistency term.

### 2.5. Relational Variational Graph Autoencoders

In "R-VGAE" [2004.10610], relations on a large heterogeneous concept-resource graph are encoded via a relational GCN and variational posterior, with a decoder that reconstructs adjacency using a DistMult-style scoring. This arrangement captures prerequisite relationships in an unsupervised setting, leveraging multi-relation message passing and latent probabilistic modeling.

## 3. Optimization Strategies and Relational Regularization

RAE architectures employ joint or alternating optimization of encoder, decoder, relational parameter (e.g., $C$, $p(z)$, GMM parameters), and in some cases direction parameters for slicing distributions (SSFG). For knowledge bases [1805.09547], separate learning rates are used for losses over entity/relation parameters and autoencoder weights. For graph-structured and OT-based RAEs, variants of Sinkhorn optimization, stochastic gradient descent, or proximal-gradient updates are used to solve transport or regularization terms [2002.02913].

Regularization hyperparameters (e.g., $\alpha,\beta,\gamma,\tau,\lambda$) mediate the trade-off between reconstruction fidelity and relational or structural constraints, and their sensitivity is often explored in ablation studies [2002.02913, 1802.03145, 2007.05742].

## 4. Empirical Properties and Performance Benchmarks

Relational autoencoder models have demonstrated gains across a breadth of metrics:

- In knowledge base completion, joint training with an autoencoder achieves state-of-the-art or near-state-of-the-art mean rank on WN18, FB15k, FB15k-237 benchmarks, particularly improving discrimination of difficult "long tail" queries [1805.09547].
- On clustering tasks, relation-guided RAEs deliver marked accuracy improvements (e.g., >98% on EYaleB, >97% on COIL20 with locality and subspace losses), and ablation removing relational terms leads to 4–6% accuracy loss [2007.05742].
- Feature extraction experiments show consistent reduction in reconstruction loss and classification error, e.g., RAE (vs. vanilla autoencoder baseline) achieves lower MSE and up to 4–5% lower classification error on MNIST and CIFAR-10 [1802.03145].
- Relational regularization via FGW, SSFG, and their variants yields improved generative modeling performance (e.g., lower Fréchet Inception Distance and mean squared error on MNIST and CelebA) compared to VAE/WAE baselines. Spherical and mixture variants further improve alignment and diversity in latent representations [2010.01787].
- On prerequisite chain prediction, R-VGAE models surpass deep embedding and baseline graph representation learning methods by up to 10% in accuracy and F1, highlighting the value of relationally-guided variational encoders in graph-based unsupervised tasks [2004.10610].

Empirical observation emphasizes the importance of balancing relational and sample-wise losses and choosing model capacity (e.g., hidden code dimension, concentration of slicing distributions) appropriate for the task.

## 5. Interpretability, Compositionality, and Analysis of Learned Representations

RAEs are notable for inducing interpretable and often sparse latent codes. In [1805.09547], dimensions of $\mathbf{c}_r$ align with interpretable semantic clusters (e.g., currency vs. film-related relations); heatmaps reveal near one-hot or block-diagonal patterns, reflecting underlying ontology structure. Relational regularization (e.g., pairwise matrix penalties, Gromov–Wasserstein distances) leads to latent spaces that better preserve manifold geometry, result in more globally consistent embeddings, and enable unsupervised discovery of compositional relationships ($M_1M_2\approx M_3$) [1805.09547, 1802.03145]. Visual and quantitative evaluation (e.g., t-SNE, spectral affinity matrices) further confirms that relational autoencoders promote clustering, smoothness, and generalization, especially in out-of-sample and multi-view contexts [2007.05742, 2002.02913].

## 6. Extensions and Theoretical Properties

RAE frameworks have been extended via:

- Advanced regularization incorporating power-spherical or mixture von Mises–Fisher distributions (PSSFG/MSSFG) to exploit important directions in high-dimensional spaces, offering improvements in convergence and diversity [2010.01787].
- Multi-view and co-training setups, with cross-space relational Gromov–Wasserstein regularization enabling effective alignment of heterogeneous and unpaired autoencoder domains [2002.02913].
- Theoretical results guarantee (pseudo-)metric properties and statistical convergence for relational discrepancy regularization under appropriate conditions, with no curse of dimensionality in expectation [2010.01787].
- The RAE paradigm encompasses both probabilistic (VAE-style) and deterministic (WAE-style) encoders, with approximations or exact FGW for GMM and empirical distributions [2002.02913, 2010.01787].

## 7. Distinctions from Related Models and Outlook

Distinct from classical autoencoders or variational autoencoders, RAEs explicitly encode, regularize, or reconstruct relationships among samples, structured objects, or latent variables, often learning both the representations and the structure (prior, similarity, cluster assignments) jointly. RAEs thus subsume models integrating affinity learning, manifold alignment, or compositional factorization as special cases, but are characterized by end-to-end differentiable architecture, learnable relational modules, and mathematical grounding in optimal transport, factorization, and spectral graph theory.

This relational principle has led to improvements in clustering, generative modeling, knowledge base reasoning, transfer, and multi-view learning, and invites further research into scalable regularization, interpretability, and applications to novel relational and graph-structured domains [1805.09547, 1802.03145, 2007.05742, 2002.02913, 2010.01787, 2004.10610].

Source: https://www.emergentmind.com/topics/relational-autoencoder-rae