M-VAE: Multimodal Variational Autoencoder

• Multimodal Variational Autoencoder (M-VAE) is a generative model that learns joint latent representations to reconstruct and generate data from heterogeneous modalities.
• It incorporates Markov Random Field (MRF) structures to aggregate modality-specific latent variables, capturing complex intermodal dependencies.
• Empirical evaluations, such as on the PolyMNIST dataset, demonstrate improved cross-modality coherence and tractable inference compared to traditional methods.

A Multimodal Variational Autoencoder (M-VAE) is a family of generative latent variable models designed to represent, reconstruct, and generate data from multiple heterogeneous modalities by learning joint probabilistic latent representations. Recent methods have focused on improving the aggregation of information across modalities to better model complex intermodal dependencies, avoid overconfidence and loss of information, and provide tractable yet expressive inference mechanisms. The Markov Random Field Multi-Modal Variational AutoEncoder (MRF-MVAE) paradigm introduces explicit structured dependence in the latent space to more faithfully reflect intermodal interactions compared to traditional product-of-experts or mixture-of-experts approaches (Oubari et al., 2024).

1. Latent Variable Structure and MRF Aggregation

Conventional multimodal VAEs aggregate modality-specific latent posteriors via products (PoE) or mixtures (MoE), which can either oversharpen (veto phenomenon in PoE) or oversmooth (MoE) the joint posterior, and do not capture complex statistical dependencies between modalities. The MRF-MVAE parameterizes the shared latent variable $z=(z_1, ..., z_M)$ as a Markov Random Field. Here, modalities correspond to graph nodes in an undirected graph $G=(V,E)$, and pairwise or higher-order dependencies between latent variable blocks are represented by clique potentials.

The prior is given in Gibbs form: $$p(z) = \frac{1}{Z} \exp\left[-\sum_{C\in\mathcal{C}} \phi_C(z_C)\right]$$ where $\mathcal{C}$ are the cliques of $G$ (typically all singletons and pairs for a fully connected MRF), with $\phi_C$ the clique potentials. In the Gaussian MRF (GMRF) case, potentials are quadratic, yielding a closed-form multivariate Gaussian prior with off-diagonal precision terms encoding intermodal correlations: $$\phi_i(z_i) = \frac{1}{2} z_i^T \Lambda_{ii} z_i - \eta_i^T z_i, \quad \phi_{i,j}(z_i,z_j) = \frac{1}{2} z_i^T \Lambda_{ij} z_j$$ Extensions include non-Gaussian (neural network-based) potentials and adversarial learning-based MRFs (ALMRF, NN-MRF).

The approximate posterior (recognition model) is specified analogously: $$q(z \mid x^1, ..., x^M) = \frac{1}{\tilde{Z}(x)} \exp\left[ - \sum_{C \in \mathcal{C}} \psi_C(z_C; x) \right]$$ Unaries $\psi_i$ depend on each modality’s encoder output, while pairwise (or higher-order) $\psi_{i,j}$ terms are predicted by a global neural aggregator.

2. Variational Objective and Analytical Tractability

The evidence lower bound (ELBO) for the MRF-MVAE is: $G=(V,E)$0 Substituting the MRF forms: $G=(V,E)$1 For the GMRF case, all expectations and partition function terms can be computed in closed form due to Gaussianity, allowing exact computation of model and variational normalization constants, Kullback–Leibler divergences, and conditional inference (by Gaussian conditioning).

For more expressive non-Gaussian cases where $G=(V,E)$2 is intractable, the ELBO can be replaced by an MMD-based penalty: $G=(V,E)$3 guaranteeing a valid bound on average (see ALMRF variant, Lemma 2 in (Oubari et al., 2024)).

3. Network Architecture and Aggregation Mechanism

Each modality $G=(V,E)$4 possesses an independent encoder—2-layer MLP, 256 ReLU units/hidden layer—outputting Gaussian parameters. Decoders mirror encoders. Off-diagonal blocks in the posterior covariance (or Cholesky factor) are predicted by a 3-layer global MLP (128 hidden units). By directly learning pairwise and possibly higher-order covariance/intermodal dependencies, the MRF-MVAE circumvents limitations of PoE (overconfident/vetoing) and MoE (over-smoothing). No hand-crafted independence/factorization assumptions are required; the model’s capacity to represent intermodal structure is limited only by the expressiveness of the clique potential parameterization.

4. Empirical Evaluation on Multimodal Datasets

MRF-MVAE was benchmarked on PolyMNIST (5-modal image variants) and a synthetic Gaussian copula-based dataset devised to test cross-modal dependencies (Oubari et al., 2024). Evaluation metrics include unconditional/conditional FID, cross-modality coherence, SSIM, and joint Wasserstein distances.

Results summary (PolyMNIST):

Model	Uncond. FID	Uncond. Coh.	Cond. FID	Cond. Coh.	Cond. SSIM
MRF-MVAE	118.2	0.321	180.8	0.869	0.995
MVAE	~95/94	~0.139/0.448	~--	~--	--
MMVAE	170/198	0.175/0.517	~--	~--	--
MoPoE-VAE	106/162	0.018/0.475	~--	~--	--
MMVAE+	87/82	0.210/0.856	~--	~--	--

Synthetic Copula results:

• MRF-MVAE achieves lowest mean Wasserstein distances: uncond. (0.86), cond. (2.6), indicating nearly perfect recovery of the complex intermodal joint.

5. Modeling Advantages and Limitations

Explicitly incorporating MRF structure into the prior and recognition networks enables the model to capture pairwise (and potentially higher) dependencies between modality latents. On PolyMNIST, this enhances cross-modality coherence and structural fidelity even at some cost of increased blur—contrasting with factorized or naively aggregated Gaussians, which either fail to capture dependencies or suffer from modal collapse/overconfidence.

For the GMRF variant, both sampling and conditional inference are analytical (via Cholesky decompositions and Gaussian conditioning). ALMRF and NN-MRF generalize to non-Gaussian potentials (including heavy-tailed or flexible neural forms), extending applicability to settings with complex or unknown interdependencies.

However, the non-Gaussian settings render the partition function $G=(V,E)$5 intractable, requiring further stochastic approximations or adversarial/MMD-based bounds, potentially impacting efficiency and introducing estimation variance.

6. Relation to Broader M-VAE Literature

MRF-MVAE can be viewed as a strict generalization of Gaussian PoE-based MVAEs (Wu et al., 2018), where the clique potentials are limited to diagonal (unimodal) and aggregation is pointwise independent. By relaxing this to an arbitrary graph and direct modeling of off-diagonal covariance, the MRF formulation subsumes and extends classic PoE/MoE schemes, and avoids the degeneracies (veto/overconfidence of PoE, over-smoothing/weathering of MoE) demonstrated in empirical evaluations (Oubari et al., 2024).

Further, unlike barycentric (Qiu et al., 2024), Hellinger-pooling (Vo et al., 10 Jan 2026), or consensus-of-dependent-experts (Mancisidor et al., 2 May 2025) strategies which reformulate aggregation in the marginal inference context but retain factorial graphical models, the MRF approach directly models and parameterizes the dependencies among latent site blocks in both generative and inference networks.

7. Extensions and Future Directions

MRF-MVAE’s capacity to directly parameterize complex latent relationships positions it as a flexible template for next-generation multimodal generative modeling with structured prior knowledge. Empirical superiority is evidenced on tasks requiring faithful recovery of intricate, cross-modal joint distributions. ALMRF and NN-MRF extensions permit highly expressive, potentially non-Gaussian, data-dependent structures, which is crucial for handling modalities with skewed, heavy-tailed, or otherwise non-normal properties. Open questions include scaling expressive MRFs to very high-dimensional problems and integrating richer context-driven clique potential parameterizations with attention or message-passing frameworks (Oubari et al., 2024).