Multimodal Variational Autoencoders (M-VAEs)

• M-VAEs are probabilistic latent variable models that extend VAEs by using shared latent variables to jointly model diverse modalities like images and text.
• They implement various aggregation schemes (e.g., MoE, PoE, MoPoE) to balance generative quality, conditional coherence, and robustness against missing data.
• They employ tailored variational objectives and regularization strategies to mitigate mode collapse and improve cross-modal data imputation.

A Multimodal Variational Autoencoder (M-VAE) is a probabilistic latent variable model that extends the classic variational autoencoder (VAE) framework to jointly model several heterogeneous data modalities (e.g., images and text). The M-VAE imposes shared latent variables that serve as the common source for each modality, allows conditional and unconditional generation, and is designed to enable both representation learning and cross-modal data imputation. Multiple M-VAE architectures have been proposed, primarily differing in their inference aggregation rules and regularization strategies, yielding distinct trade-offs in generative quality, conditional coherence, robustness against missing modalities, and scalability.

1. Core Principles and Model Formulation

The foundational structure of an M-VAE assumes $M$ modalities $x_1, \ldots, x_M$ and a latent variable $z$ shared across them. The generative model factorizes:

$$p_\theta(x_1, \ldots, x_M, z) = p(z) \prod_{m=1}^M p_\theta(x_m \mid z)$$

where $p(z)$ is typically a standard Gaussian and the $\{p_\theta(x_m \mid z)\}$ are neural network decoders. Given this structure, the modalities are conditionally independent given $z$.

Learning and inference rely on a variational lower bound (ELBO), with inference networks $q_\phi(z \mid \cdot)$ implemented in different ways across M-VAE variants to handle arbitrary subsets of observed modalities.

2. Aggregation Schemes and Posterior Approximation

The primary technical divergence across M-VAE models lies in how the joint posterior $p(z \mid x_1, \ldots, x_M)$ is approximated, particularly when only a subset of modalities is available.

Mixture-of-Experts (MoE) Aggregation

In the Mixture-of-Experts approach—often called MMVAE—the joint posterior is approximated as a mixture:

$$q_\phi(z \mid x_1, \ldots, x_M) = \sum_{m=1}^M \pi_m(x_1, \ldots, x_M) \, q_\phi^{(m)}(z \mid x_m)$$

where each expert $x_1, \ldots, x_M$0 is typically a modality-specific Gaussian, and $x_1, \ldots, x_M$1 are non-negative weights (uniform or learned through a gating mechanism) that sum to one (Wolff et al., 2022). The mixture form allows modular inference and adaptation to missing modalities.

Alternative aggregation rules include:

• Product-of-Experts (PoE): $x_1, \ldots, x_M$2, which tightly concentrates on regions where all experts agree (Wu et al., 2018).
• MoPoE: Mixture-of-Products-of-Experts, interpolating between MoE and PoE across all modality subsets (Sejnova et al., 2022).
• Barycentric: Aggregating using optimal-transport divergences (e.g., Wasserstein barycenters) (Qiu et al., 2024), or using probabilistic opinion pooling (e.g., Hellinger pooling) (Vo et al., 10 Jan 2026).
• CoDE (Consensus of Dependent Experts): Bayesian aggregation that explicitly models dependencies between modalities (Mancisidor et al., 2 May 2025).
• MRF-based aggregation: Using Markov Random Field structure to model full-covariance dependencies in the joint latent (Oubari et al., 2024).

3. Variational Objectives and Losses

The basic training loss for M-VAEs is an ELBO over all available modalities:

$x_1, \ldots, x_M$3

Depending on aggregation, variants optimize marginal, conditional, or joint ELBOs:

• Stratified ELBO (as in MMVAE): ELBOs over individual modalities are averaged or summed, typically requiring modality sub-sampling during each minibatch (Wolff et al., 2022, Sejnova et al., 2022, Daunhawer et al., 2021).
• Mixture-prior or soft-coupling regularization: Jensen–Shannon divergence between unimodal posteriors regularizes encodings without collapsing modality-specific details, enabling “soft” alignment (Sutter et al., 2024).
• Two-stage or auxiliary objectives: Separation of joint and conditional distributions via explicit regularization, e.g., coupling of joint and per-modality encoders via KL penalties or alignment constraints (Senellart et al., 2023, Senellart et al., 6 Feb 2025).
• Disentanglement penalties: When decomposing $x_1, \ldots, x_M$4 into shared and private subspaces, KL-divergences and cross-view reconstructions are partitioned to preserve disentanglement (Shi et al., 2019, Märtens et al., 2024).
• Barycentric/Opinion Pooling: Use of f-divergence or optimal-transport objective in place of KL, allowing interpolation between MoE- and PoE-like behaviors (Qiu et al., 2024, Vo et al., 10 Jan 2026).

4. Limitations of Mixture-of-Experts Inference

While MoE architectures such as MMVAE provide appealing modularity and weakly-supervised capabilities, they exhibit fundamental limitations in certain data regimes:

• Surjective (one-to-many) Mappings and Mode Collapse:

In surjective multimodal data (e.g., one class label mapping to many images), MoE posteriors cannot capture within-class variability. Formally, if $x_1, \ldots, x_M$5 is surjective, the MoE posterior’s optimal solution is to predict the mean of all possible $x_1, \ldots, x_M$6 when only $x_1, \ldots, x_M$7 is observed, collapsing within-class variation (Wolff et al., 2022).

• Irreducible ELBO Gap:

Subsampling of modality subsets during training induces a lower bound gap proportional to the conditional entropy of missing modalities. For mixture-based inference, this results in limited generative quality (FID, likelihood), particularly as the number of modalities increases or when modality-private factors dominate (Daunhawer et al., 2021).

• Trade-off Between Generative Quality and Conditional Coherence:

MoE-based models yield robust cross-modal generation (“coherence”) but compromise joint generation quality. PoE-based models do the opposite. No single model can simultaneously maximize both on complex, high-diversity data (Daunhawer et al., 2021, Sejnova et al., 2022, Märtens et al., 2024).

• Missing-Information Bottleneck:

In complex settings, mixture-based posteriors lack capacity to reconstruct missing private information, leading to limitations in generalization and cross-modal imputation (Hirt et al., 2023).

5. Empirical Benchmarking and Results

Empirical evaluation of M-VAEs features both synthetic (e.g., PolyMNIST, CdSprites+) and real-world benchmarks (e.g., CelebA, CUB, MNIST–SVHN–Text). Key metrics include:

• Conditional coherence: Consistency of generated target modality samples with given source modalities, commonly scored using pretrained classifiers.
• Joint and marginal log-likelihood: Surrogate measures for overall generative fidelity.
• FID (Fréchet Inception Distance): Image generation quality.
• Linear classification on latent codes: Probing shared and modality-specific information.

A consistent set of trends arises:

Model Type	Generative Quality	Conditional Coherence	Scalability
Product-of-Experts	High	Low	Parameter efficient
Mixture-of-Experts	Low	High	Modular, scalable
MoPoE, MWB, CoDE, Hellinger	High/intermediate	Intermediate/balanced	Intermediate
Hierarchical/Dropout-based	Comparable	Comparable	Adaptable

• MMVAE and MoPoE consistently enable better cross-modal generation at the expense of FID/log-likelihood, with performance degrading on highly multimodal or surjective datasets (Wolff et al., 2022, Daunhawer et al., 2021).
• Extensions such as MMVAMP (soft mixture-prior), CoDE-VAE (expert dependency modeling), HELVAE (Hellinger pooling), and Wasserstein-barycentric models have closed much of the performance gap, providing improved Pareto frontiers between quality and coherence (Sutter et al., 2024, Mancisidor et al., 2 May 2025, Vo et al., 10 Jan 2026, Qiu et al., 2024).
• For disentanglement and robustness against dominance of modality-specific variation, modified ELBO objectives that decouple same- and cross-view gradient flow yield the best empirical robustness (MMVAE⁺⁺) (Märtens et al., 2024).

6. Remedies, Alternative Formulations, and Open Directions

Several remedies and variants are proposed to overcome the MoE limitations:

• Avoid explicit ELBO terms that reconstruct many-to-one mappings using MoE aggregation in surjective settings (Wolff et al., 2022).
• Employ PoE, barycentric, or learned aggregator posteriors to better capture conditional variability—Wasserstein and Hellinger barycenters provide more balanced aggregation, interpolating between PoE and MoE (Qiu et al., 2024, Vo et al., 10 Jan 2026).
• Gating weights: Data-adaptive or learned gating reduces the impact of weakly informative modalities (Wolff et al., 2022, Sutter et al., 2024).
• Permutation-invariant/inclusive encoders: Flexible encoders (e.g., Set Transformers) that aggregate across arbitrary subsets mitigate inductive aggregation bias (Hirt et al., 2023).
• Alignment regularization and iterative inference: Iteratively refining unimodal posteriors via multimodal gradients and distillation closes amortization and missing-modality information gaps (Oshima et al., 2024).
• Explicit latent decomposition: Partitioning the latent into shared and private subspaces, along with targeted regularization, robustly preserves modality separation and facilitates cross-modal tasks (Märtens et al., 2024, Shi et al., 2019).
• Impartial optimization: Multitask gradient conflict resolution prevents modality collapse and improves overall coherence (Javaloy et al., 2022).
• Unsupervised or semi-supervised disentanglement: Cross-modal and same-modal gradients are routed to appropriate (private/shared) latent spaces, improving disentanglement in unbalanced or label-sparse environments (Märtens et al., 2024).

Open research questions include optimal trade-off points in aggregation strategies, principled weighting and subset selection in ELBO decompositions, integration of amortized and iterative refinement for inference, and extension to fully non-Gaussian or non-parametric expert distributions.

7. Schematic Table: MoE vs. PoE and Key Trade-offs

Inference Aggregation	Posterior Sharpness	Generative Quality (FID, LLH)	Conditional Coherence	Training Objective Complexity	Robustness to Surjective Data
Mixture-of-Experts	Low	Poor	High	Low (per-modality)	Poor (mode collapse)
Product-of-Experts	High	High	Poor	Moderate (joint product)	Good
MoPoE, Barycenter	Intermediate	Balanced	Balanced	High (power set/subset)	Good
Hellinger, CoDE	Balanced	Balanced	Balanced	Moderate (moment matching)	Robust

The dominance of MoE in enabling modularity and enabling conditional generation is offset by irreducible trade-offs in generative diversity and an inability to capture conditional variability under surjective mappings (Wolff et al., 2022, Daunhawer et al., 2021, Märtens et al., 2024). Robust variants require either more expensive aggregation (MoPoE/MWB), adaptive weighting, explicit disentanglement of latent spaces, or advanced variance regularization and alignment strategies.

---

References

• "Mixture-of-experts VAEs can disregard variation in surjective multimodal data" (Wolff et al., 2022)
• "Unity by Diversity: Improved Representation Learning in Multimodal VAEs" (Sutter et al., 2024)
• "Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkit" (Sejnova et al., 2022)
• "Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives" (Hirt et al., 2023)
• "Multimodal Generative Models for Scalable Weakly-Supervised Learning" (Wu et al., 2018)
• "Disentangling shared and private latent factors in multimodal Variational Autoencoders" (Märtens et al., 2024)
• "On the Limitations of Multimodal VAEs" (Daunhawer et al., 2021)
• "A Markov Random Field Multi-Modal Variational AutoEncoder" (Oubari et al., 2024)
• "Hellinger Multimodal Variational Autoencoders" (Vo et al., 10 Jan 2026)
• "Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders" (Mancisidor et al., 2 May 2025)
• "Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization" (Javaloy et al., 2022)