Multimodal VAE: Fusion and Aggregation

• Multimodal VAEs are deep generative models that fuse heterogeneous data by learning shared and modality-specific latent representations.
• They employ aggregation methods such as PoE and MoE to integrate unimodal encoder outputs, balancing precision and robustness.
• Advanced training objectives, like masked ELBO and synergy losses, enhance cross-modal consistency and performance under missing data conditions.

Multimodal autoencoders and variational autoencoders (VAEs) extend the autoencoding principle to settings where multiple heterogeneous data streams—such as images, audio, or text—co-occur and must be jointly modeled for generation and inference. The central theoretical and practical challenge is to design latent variable models and inference schemes that support both joint encoding/decoding and robust conditional (cross-modal) generation, even in the presence of missing modalities, private information, and limited supervision. The field has witnessed the development of several influential architectures and objective functions, including the product-of-experts (PoE) and mixture-of-experts (MoE) inference schemes, hierarchical latent structures, information-theoretic training criteria, and specialized architectural designs for modality fusion.

1. Core Model Families and Posterior Aggregation

Two dominant classes of multimodal autoencoder models are based on how they aggregate unimodal encoder outputs into a joint posterior over the latent code.

Product-of-Experts (PoE) VAEs: In PoE models such as MVAE, each modality is encoded into a Gaussian "expert" $q_{\phi_m}(z|x^m)$. The joint posterior is the normalized product of these experts, which for Gaussians has a closed-form solution via precision (inverse variance) addition. This encourages concentration in regions where all modalities agree and sharply infers the latent factors when all are present. The generative model assumes conditional independence given the latent.

Mixture-of-Experts (MoE) VAEs: In MoE models such as MMVAE, the joint posterior is a normalized sum of the unimodal posteriors, $$q_\phi(z|x^{1,\dots,M}) = \sum_{m=1}^M w_m q_{\phi_m}(z|x^m)$$. MoE models are more “mass-covering” and robust to individual expert collapse, averaging across modalities and thus favoring cross-modal consistency.

Beyond PoE/MoE: More recent work frames multimodal inference as a barycenter or probabilistic opinion pooling problem, generalizing PoE and MoE to a larger family of divergence-based aggregators. For example, PoE and MoE arise as minimizers of weighted reverse KL and forward KL divergences, respectively, while Hellinger and Wasserstein barycenters can interpolate between these extremes and promote symmetric or geometry-preserving fusion (Qiu et al., 2024, Vo et al., 10 Jan 2026).

Key Table: Posterior Aggregation Schemes

Aggregator	Formula	Strengths
Product-of-Experts	$q(z|\mathbf x) \propto p(z)\prod_m q_m(z|x^m)$	Sharp joint, efficient, analytic
Mixture-of-Experts	$q(z|\mathbf x) = \sum_m w_m q_m(z|x^m)$	Robust to mode coverage, flexibility
Wasserstein Bary.	$\arg\min_r \sum_m w_m \mathcal{W}_2^2(q_m, r)$	Preserves geometry, smooth fusion
Hellinger Pooling	See moment-matching in (Vo et al., 10 Jan 2026)	Symmetric, robust, Pareto front

2. Hierarchical, Disentangled, and Hybrid Latent Structures

To capture both shared and private variation, hierarchical and disentangled models extend the classical flat latent structure.

Shared and Private Latents: Models like DMVAE, MMVAE, and hierarchical MVAE explicitly partition the latent representation into a shared latent $z_s$ (capturing common factors) and modality-specific private latents $z_p^m$ (Lee et al., 2020, Shi et al., 2019, Märtens et al., 2024). Decoders receive both latent types, ensuring that each modality can reconstruct its unique information while joint factors are preserved across views.

Hierarchical Models: MHVAEs introduce multiple layers of latents, mirroring human convergence-divergence zones: a top-level joint latent $z^c$ is responsible for global coordination, while each modality receives its own lower-level code $z^m$ (Vasco et al., 2020).

Factor Analysis and Modularity: FA-VAE architectures use a factor analysis prior for the shared code, connecting arbitrarily many "private VAE" branches and enabling modular extensibility. This structure facilitates adding/removing modalities, interpretable latent spaces, and data-efficient transfer (Guerrero-López et al., 2022).

Hybrid Continuous/Discrete Latents: For settings involving categorical or clustered structure, hybrid VAEs combine Gumbel-Softmax and Gaussian latents to permit both discrete (class/attribute) and continuous (style/content) encoding (Lee et al., 2020).

3. Training Objectives, Loss Functions, and Information Theory

Standard ELBO and Its Limitations: Most multimodal VAEs optimize some variant of the ELBO, balancing reconstruction and KL divergence terms across all modalities. However, mixture-based training with sub-sampling over modality subsets introduces an irreducible "mixture gap" proportional to the conditional entropy $H(X_{\neg S}|X_S)$, penalizing generative quality as the number or diversity of modalities increases (Daunhawer et al., 2021). PoE models avoid this gap but may become rigid or collapse under modality-specific noise.

Sub-sampled and Masked Bounds: Masked or masked-marginal ELBO objectives, which simultaneously regularize marginal and conditional likelihoods for all subsets $$q_\phi(z|x^{1,\dots,M}) = \sum_{m=1}^M w_m q_{\phi_m}(z|x^m)$$0 of modalities, achieve tighter bounds and better identifiability. Permutation-invariant neural set encoders (Deep Sets, Transformers) are used to avoid $$q_\phi(z|x^{1,\dots,M}) = \sum_{m=1}^M w_m q_{\phi_m}(z|x^m)$$1 encoders (Hirt et al., 2023).

Partial Information Decomposition and Synergy Losses: Instead of pure ELBOs, some architectures, such as PC-VAE, minimize interaction information (difference between total, marginal, and synergistic mutual information) to directly control the complementary and redundant contributions of each modality to the output (Liang et al., 2022).

Jensen–Shannon Soft Alignment: Methods like Unity by Diversity (MM-VAMP) replace hard aggregation with a "soft" JS-divergence penalty, encouraging unimodal encoders to align but not collapse, preserving modality-specific information while fostering cross-modal coherence (Sutter et al., 2024).

Gradient Impartiality and Anti-collapse: To prevent modality collapse (degenerate focus on a subset of modalities), impartial optimization applies multitask gradient balancing (PCGrad, GradNorm) in the computational graph wherever conflicting signals arise, substantially boosting coherence and individual modality performance (Javaloy et al., 2022).

4. Missing Data, Weak Supervision, and Imputation

Multimodal VAEs are tasked with missing-modality imputation—i.e., cross-modal generation given incomplete inputs—via specific design choices:

• PoE models: At inference, simply omit unobserved experts in the product; sub-sampled training ensures valid inference for missing patterns (Wu et al., 2018).
• MoE/MMVAE: Permits arbitrary observed subsets, using mixture inference; but mixture gap limits their generative quality when modality-specific variation is large (Shi et al., 2019).
• Hierarchical Dropout: Hierarchical models like MHVAE use random dropout in the hidden encodings to mimic missing modalities and regularize the joint encoder (Vasco et al., 2020).
• Iterative Alignment and Distillation: Recent iterative amortized inference frameworks refine unimodal encoders via repeated ELBO gradient steps against a multimodal "teacher," minimizing the amortization gap and improving unimodal-inferred representations (Oshima et al., 2024).

5. Evaluation Protocols, Benchmarks, and Empirical Findings

Datasets: Across works, the principal benchmarks include PolyMNIST (multiple digit images with aligned content), CelebA (images and attributes), CdSprites+ (bimodal synthetic data with compositional structure), MNIST–SVHN–Text (triple-view digits), CUB (image-caption), and multi-omics/robotics data.

Metrics: Key quantitative criteria include generative coherence (i.e., conditional/cross-modal generation accuracy), generative quality (e.g., Fréchet Inception Distance, negative log-likelihood), latent representation classification accuracy (for downstream tasks), disentanglement measures (e.g., TC, MI), and conditional generation under modality-missing regimes.

Summary Table: Empirical Properties of Major Model Families

Model	Joint Gen Quality	Cross-Modal Coherence	Scalability	Handles Missing	Distinct Features
MVAE (PoE)	Best	Low (if high privates)	Efficient	Yes	Closed-form inference, sharp latents
MMVAE (MoE)	Moderate	High	Efficient	Yes	Mixture, robust to expert collapse
MoPoE	Moderate	Best (mid-range M)	Subset-sample	Yes	Mixture of subset PoEs
DMVAE/MMVAE+	Moderate–High	High	Efficient	Yes	Shared/private blocks, hybrid latents
Barycenter	Pareto-optimal	High	Efficient	Yes	Geometry-preserving, interpolating
PC-VAE	Moderate	High (cross-modal)	Parallel	Yes	PID loss, random encoders, no attention
HELVAE	Excellent	Pareto-optimal	Fastest	Yes	Hellinger moment-matching pooling
MRF-MVAE	Highest coherence	High	Costly (M²)	Yes	Captures full intermodal dependencies
MM-VAMP	High	High	Efficient	Yes	JS-soft alignment of posteriors

Empirical Takeaways:

• MVAE outperforms in unconditional generation, especially when modality-private variation is small and all modalities are reliable.
• MMVAE, MoPoE, and MMVAE++ are more robust for cross-modal prediction and in the presence of dominant modality-private variation (Märtens et al., 2024, Daunhawer et al., 2021).
• Barycentric (Wasserstein or Hellinger) and opinion pooling methods achieve Pareto-optimal trade-offs between generative coherence and sample quality, interpolating between PoE and MoE behaviors (Qiu et al., 2024, Vo et al., 10 Jan 2026).
• Hierarchical and modular architectures are critical for scalable and extensible systems, especially with heterogeneous data and when incorporating additional modalities (Guerrero-López et al., 2022, Vasco et al., 2020).

6. Information-Theoretic Analysis, Limitations, and Current Challenges

Information-theoretic studies reveal that

• The mixture gap in sub-sampled MoE/MoPoE schemes cannot be closed through optimization alone and grows with the number and diversity of modalities (Daunhawer et al., 2021).
• Private vs. shared latent disentanglement is nontrivial: conventional models may soak up high-dimensional private noise into the shared latent unless gradients and reconstruction pathways are partitioned explicitly as in MMVAE++ (Märtens et al., 2024).
• Modality collapse (failure of the generative model to use all inputs) often results from conflicting gradients at the decoder split/merge points and can be systematically diagnosed and mitigated using impartial optimization (Javaloy et al., 2022).
• Scheduling of the $$q_\phi(z|x^{1,\dots,M}) = \sum_{m=1}^M w_m q_{\phi_m}(z|x^m)$$2-weight in the ELBO, which scales the KL loss, is critical to avoid posterior collapse and to achieve good multimodal fusion. Info-theoretic measures such as single-modality error and loss-of-precision dissect the flow of predictive information across modalities and guide architecture/loss rebalancing in robotics and sensorimotor systems (Langer et al., 2024).

7. Special Case: Multimodal Transformer PC-VAE

The PC-VAE applies a parallel, non-learned random-projection encoder (splitting images into vertical stripes and audio into fixed-length segments, compressed with fixed Gaussian matrices) followed by a concatenative fusion and pair of decoders—one for each modality. Its loss is derived from partial information decomposition, specifically interaction information, favoring low synergy and redundancy in the cross-modal mappings. PC-VAE allows for cross-modal synthesis even when only a single input modality is available, demonstrating strong empirical performance in joint, unimodal, and cross-modal generative tasks with simple, fully parallel encoders (Liang et al., 2022).

---

References:

• Multimodal Transformer for Parallel Concatenated Variational Autoencoders (Liang et al., 2022)
• Multimodal Generative Models for Scalable Weakly-Supervised Learning (Wu et al., 2018)
• On the Limitations of Multimodal VAEs (Daunhawer et al., 2021)
• MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning (Vasco et al., 2020)
• Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization (Javaloy et al., 2022)
• Disentangling shared and private latent factors in multimodal Variational Autoencoders (Märtens et al., 2024)
• Hellinger Multimodal Variational Autoencoders (Vo et al., 10 Jan 2026)
• Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (Shi et al., 2019)
• Improving VAE generations of multimodal data through data-dependent conditional priors (Lavda et al., 2019)
• Multimodal VAE Active Inference Controller (Meo et al., 2021)
• Unity by Diversity: Improved Representation Learning in Multimodal VAEs (Sutter et al., 2024)
• Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives (Hirt et al., 2023)
• Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkit (Sejnova et al., 2022)
• Private-Shared Disentangled Multimodal VAE for Learning of Hybrid Latent Representations (Lee et al., 2020)
• Multimodal hierarchical Variational AutoEncoders with Factor Analysis latent space (Guerrero-López et al., 2022)
• Multimodal Variational Autoencoder: a Barycentric View (Qiu et al., 2024)
• Analyzing Multimodal Integration in the Variational Autoencoder from an Information-Theoretic Perspective (Langer et al., 2024)
• A Markov Random Field Multi-Modal Variational AutoEncoder (Oubari et al., 2024)
• Enhancing Unimodal Latent Representations in Multimodal VAEs through Iterative Amortized Inference (Oshima et al., 2024)
• A multimodal dynamical variational autoencoder for audiovisual speech representation learning (Sadok et al., 2023)