Relational Regularized RAEs Overview

• Relational Regularized RAEs are autoencoder architectures enhanced with a relational penalty that preserves structural similarities in the input data.
• They use methods like similarity matrix matching and sliced FGW to balance reconstruction loss with latent space consistency.
• Empirical evaluations show these models improve feature extraction, classification accuracy, and generative performance in diverse domains.

Relational Regularized RAEs (Relational Regularized Autoencoders) constitute a principled class of autoencoding architectures that augment basic encoder–decoder models with structural—or relational—regularization penalties. These regularizers enforce that the latent representations, or the learned latent prior, preserve the pairwise relationships or structural similarity present in the input data. The approach applies both to feature extraction in high-dimensional data as well as to distributional generative modeling, including images, graphs, or multi-view structured domains. This entry surveys foundational models, functional objectives, algorithmic advances, and empirical findings underlying the development and application of Relational Regularized RAEs across classical, deterministic, and probabilistic autoencoding frameworks.

1. Core Principles of Relational Regularized Autoencoders

Relational Regularized RAEs are distinguished by incorporating a relational regularizer into the canonical autoencoding loss. The autoencoder consists of an encoder $f_\theta$ mapping input data $X$ to a code space (latent space) and a decoder $g_\theta$ reconstructing $X'$ from encodings. The core innovation is to supplement the typical reconstruction loss, $\|X - X'\|$, with a term penalizing discrepancies between relationships or structure in the input space and those in the latent or reconstructed space.

In the basic Relational Autoencoder model, the relational penalty is cast as the discrepancy between similarity matrices calculated on $X$ and $X'$. Given $X \in \mathbb{R}^{n \times m}$ (samples $\times$ features), the similarity matrix is $R(X) = X X^\top$. The full RAE loss is:

$$\Theta(\theta) = (1-\alpha) \sum_{i=1}^n \|x_i - x'_i\|^2 + \alpha \|\tau_t(XX^\top) - \tau_t(X'X'^\top)\|_F^2$$

where $\tau_t(\cdot)$ is a rectifier nullifying similarities below threshold $t$, and $\alpha \in [0,1]$ balances reconstruction versus relationship preservation (Meng et al., 2018).

For probabilistic and generative tasks, the regularizer becomes a distributional discrepancy, such as the fused Gromov–Wasserstein (FGW) distance or its scalable approximations (Xu et al., 2020, Nguyen et al., 2020). These variants measure both direct pairwise proximity and higher-order (relational) structural similarity between the learned posterior in latent space and a target prior, or between heterogeneous AEs in multi-view settings.

2. Mathematical Formulation of Relational Regularization

Relational penalties in RAEs are formalized primarily through two paths:

• Similarity Matrix Regularization: The RAE objective compares similarity matrices before and after autoencoding. For data $X$ and reconstructions $X'$, strong relationships (above threshold $t$) are preserved via $\|\tau_t(XX^\top) - \tau_t(X'X'^\top)\|_F^2$. This mechanism is directly extensible to sparse, denoising, and variational autoencoders by matching relationships in noisy, sparse, or probabilistic latent encodings (Meng et al., 2018).
• Fused Gromov–Wasserstein (FGW) Regularization: In distributional RAEs, the relational loss becomes the FGW distance between distributions $p_x$, $p_y$ equipped with respective metrics $d_x$, $d_y$:

$$D_{\mathrm{fgw}}(p_x, p_y; \beta) = \inf_{\pi \in \Pi(p_x, p_y)} \Bigl\{ (1-\beta) \int c(x, y) \, d\pi(x, y) \,+\, \beta \iint |d_x(x, x') - d_y(y, y')|^2 \, d\pi(x, y) d\pi(x', y') \Bigr\}$$

where $c$ is the ground cost and $\beta$ modulates trade-off between pointwise alignment and relational (pairwise) structure (Xu et al., 2020). For deterministic AEs, the sliced FGW (SFGW) projection allows efficient computation via random 1D projections and permutation.

• Spherical Sliced FGW (SSFG) and Variants: To focus regularization on maximally informative projections, SSFG replaces uniform random projections with projections distributed according to a von Mises–Fisher (vMF) or other concentrated directional distribution, maximizing the expected alignment in principal discriminative directions (Nguyen et al., 2020).

3. Model Architectures and Extensions

Relational regularized autoencoders have been instantiated in multiple forms:

• Feature Extraction RAEs: Encoder and decoder networks use tied weights, sigmoidal activations, and are trained to minimize reconstruction and relationship loss, as specified above. Generalizations support sparse (RSAE), denoising (RDAE), and variational (RVAE) extensions by augmenting the basic loss with $L_2$ penalties, input noise, or KL-divergence respectively (Meng et al., 2018).
• Regularized Graph Autoencoders (RGAE): For network data with multiple edge types, RGAE constructs per-view private GCN-autoencoders and a global shared GCN. The loss includes a reconstruction term, a similarity regularizer (aligning a global consistent embedding with shared view embeddings), and a difference regularizer enforcing orthogonality between shared and private embeddings (Wang et al., 2021).
• Probabilistic RAEs: Encoder–decoder pairs parameterize a learnable (often GMM) prior in latent space. The relational FGW regularizer couples distributions of posterior codes and the prior, tuned by $\gamma$ and $\beta$. Optimization uses hierarchical or sliced FGW, via Sinkhorn or stochastic projections (Xu et al., 2020, Nguyen et al., 2020).
• Deterministic RAEs: These completely drop encoder stochasticity (as in VAEs), regularize via code-norm penalties, decoder gradient or spectral normalization, and fit the sampling distribution ex-post via learned code statistics (Gaussian or GMM) (Ghosh et al., 2019).

4. Algorithmic Implementation and Optimization

Efficient training in Relational Regularized RAEs relies on scalable approximations to the relational penalty:

• Similarity Matrix Matching: For small- to mid-scale datasets, direct matrix Frobenius norm penalties are tractable (Meng et al., 2018).
• Sliced FGW: Projects data into 1D via randomly sampled directions, sorts points, computes closed-form 1D FGW, then averages over projections; computational complexity scales as $O(L N \log N)$ per minibatch for $L$ projections and $N$ samples (Xu et al., 2020).
• Hierarchical FGW: For GMMs, constructs pairwise Wasserstein distances between components and solves via Sinkhorn iterations, yielding $O(J N^2 K)$ per batch (for $J$ iterations, $K$ prior components, $N$ posterior samples) (Xu et al., 2020).
• SSFG/MSSFG/PSSFG: SSFG focuses the projections using vMF modes, mixture variants (MSSFG) simultaneously attend to multiple discriminative directions, and PSSFG leverages power-spherical distributions for sampling efficiency in high dimensions. Each introduces an inner maximization step over the mode parameter(s), which is tractable given mini-batch sizes and moderate numbers of projections or mixture components (Nguyen et al., 2020).
• Graph RGAEs: Alternate between parameter updates (standard backpropagation in GCNs, fixing the view-weights) and closed-form view-weight updates through Lagrange multipliers until convergence (Wang et al., 2021).

5. Empirical Evaluation and Practical Impact

Relational Regularized RAEs have demonstrated systematic gains across multiple domains and benchmarks:

• Feature Extraction and Classification: On MNIST and CIFAR-10, RAEs yield lower mean square error (MSE) and classification error than basic AEs, sparse AEs, and graph AEs. Optimal relational strength ($\alpha$) is critical: low $\alpha$ fails to exploit pairwise structure, excessive $\alpha$ can degrade data reconstruction (Meng et al., 2018).
• Graph Representation Learning: RGAEs outperform state-of-the-art multi-view graph embedding models (MVE, MNE) by 1–6% Micro-F1 on node classification (AMiner, PPI) and improve ROC-AUC/AP by ~2–4%/2–3% on link prediction tasks (YouTube) (Wang et al., 2021). Ablation confirms substantial performance losses when either similarity or difference regularization is omitted.
• Generative Modeling: Probabilistic/deterministic RAEs surpass VAE, WAE, and GMVAE baselines in test set reconstruction and sample quality (evaluated by FID) on MNIST and CelebA. The use of SSFG-based discrepancies (particularly MSSFG) lowers FID by 10–25% over earlier SFG-based models while retaining scalablility; PSSFG accelerates training 20–30% with negligible quality loss (Nguyen et al., 2020).
• Multi-view Representation: RAEs permit the co-training of heterogeneous autoencoders by imposing relational (Gromov–Wasserstein) consistency between views. This yields higher accuracy than independent AE training or direct co-regularization, even in the absence of paired data (Xu et al., 2020).

6. Theoretical Properties and Limitations

Key theoretical and practical properties of relational regularization include:

• Pseudo-metric Properties: FGW and its variants (SFG, SSFG, MSSFG, PSSFG) are pseudo-metrics satisfying non-negativity, symmetry, and weak triangle inequalities, making them suitable for alignment of structured distributions (Nguyen et al., 2020).
• Compatibility and Flexibility: Relational penalties can be integrated with deterministic, probabilistic, graph-based, or multi-view architectures. The Gromov–Wasserstein framework, in particular, is agnostic to dimensionality and geometric characteristics, promoting robustness in distribution matching (Xu et al., 2020, Nguyen et al., 2020).
• Computational Overheads: Advanced relational penalties (e.g., SSFG and MSSFG) introduce additional inner-loop optimization but remain practical with moderate hyperparameter settings (e.g., $T_{\text{inner}}=10$, $L=50$, $k=10$). Power-spherical sampling further mitigates the overhead associated with vMF sampling in high dimensions (Nguyen et al., 2020).
• Hyperparameter Sensitivity: Selection of balance coefficients ($\alpha$, $\gamma$, $\beta$), projection distributions, and mixture components is crucial for attaining optimal empirical performance. Excessive regularization can induce over-smoothing or latent code collapse (Meng et al., 2018, Wang et al., 2021).

7. Future Directions and Open Questions

Emerging work suggests several research avenues:

• Adaptive Regularization: Dynamic adjustment of concentration parameters (e.g., $\kappa$ in SSFG, number of mixture components) during training, or joint learning of projection distributions (Nguyen et al., 2020).
• Extension to Structured Generative Models: Applying relational penalties to normalizing flows, graph generators, or beyond (Nguyen et al., 2020).
• Relational Regularization in Non-Euclidean and Heterogeneous Spaces: Sophisticated construction of per-view or per-task relational metrics (as in RGAE) for more diverse data modalities (Wang et al., 2021).
• Interpretability and Latent Geometry: The optimal transport plan in FGW regularization provides a mapping between generated and real data, potentially offering interpretable latent structure and mode alignment (Xu et al., 2020).

Collectively, Relational Regularized RAEs unify the preservation of structural relationships in representation learning and generative modeling, leveraging principled optimal transport and relational similarity objectives to enhance the expressivity and generalization of autoencoder-based architectures.