MoE Vision-Language VLVAE

• The paper introduces a MoE-based variational autoencoder that uses a barycentric formulation to robustly fuse vision and language data.
• It employs unimodal encoders, a gating network, and closed-form moment matching to generate disentangled, tractable joint posterior representations.
• Empirical results in radiology report generation demonstrate improved BLEU, ROUGE, and clinical F1 scores, especially under missing-modality conditions.

A Mixture-of-Experts (MoE) based Vision-Language Variational Autoencoder (VLVAE) is a probabilistic generative framework designed to learn robust, disentangled joint representations of multimodal data—specifically, images and text. Recent works conceptualize this approach through a barycentric lens, where the latent posterior is modeled as a barycenter (typically a mixture) of unimodal inference networks (experts), enabling mass-covering, tractable, and geometry-aware fusion of vision and language modalities. Such models have been particularly impactful in domains with frequent missing modalities and the requirement for precise semantic fusion, such as automatic radiology report generation.

1. Theoretical Foundations: MoE Barycenter in VLVAE

The core theoretical advance is the use of a mixture-of-experts (MoE) barycenter to approximate the joint posterior over latents given vision–language input. Given input image $x_V \in \mathbb{R}^{H \times W \times 3}$ and text $x_L$, distinct unimodal inference networks yield distributions

$$q_V(z|x_V) = \mathcal{N}(z; \mu_V, \Sigma_V), \qquad q_L(z|x_L) = \mathcal{N}(z; \mu_L, \Sigma_L).$$

The barycenter is constructed via minimization of a weighted, asymmetric divergence, specifically the forward KL divergence: $$q^* = \arg\min_q \alpha_V D_{\mathrm{KL}}(q_V \| q) + \alpha_L D_{\mathrm{KL}}(q_L \| q),$$ yielding the mixture posterior

$$q_{\mathrm{MoE}}(z | x_V, x_L) = \alpha_V q_V(z|x_V) + \alpha_L q_L(z|x_L),$$

with weights $\alpha_V, \alpha_L > 0$ and $\alpha_V + \alpha_L = 1$, learned adaptively via a gating network. This formulation ensures mass-covering and accommodates missing modalities by design. Closed-form moment matching is used to reparametrize the mixture as a tractable Gaussian for efficient sampling and backpropagation: $$\bar{\mu} = \alpha_V \mu_V + \alpha_L \mu_L,\ \bar{\Sigma} = \alpha_V (\Sigma_V + \mu_V \mu_V^\top) + \alpha_L (\Sigma_L + \mu_L \mu_L^\top) - \bar{\mu} \bar{\mu}^\top.$$ For diagonal covariance, the formulation reduces dimension-wise and admits efficient implementation (Qiu et al., 29 Dec 2024).

2. Encoder, Gating, and Decoder Design

The standard architectural instantiation consists of the following modules:

• Unimodal Encoders:
  • Vision Encoder: Deep CNNs (e.g., ResNet, VGG16) with global pooling and dual linear heads for posterior mean and log-variance.
  • Language Encoder: Transformers or Bi-LSTMs with pooled token embedding; dual linear heads analogous to vision branch.
• Gating Network:

Concatenated feature vectors $[h_V; h_L]$ are passed through a small multilayer perceptron (MLP), producing unnormalized logits $s_V, s_L$ which are softmaxed to produce $\alpha_V, \alpha_L$. Softmax temperature $\tau$ is annealed during training to control gating sharpness.

• Decoders:
  • Image Decoder: Up-convolutions or U-Net structures mapping latent $z$ to pixel logits.
  • Text Decoder: Transformer decoder or LSTM for autoregressive text generation, optionally leveraging cross-attention over multimodal latents.

The architecture supports three-factor disentangled representations for modality-specific ($z_v$, $z_l$) and shared ($z_s$) factors (Shaik et al., 8 Nov 2025), realized using dedicated encoders for each and a MoE-based latent posterior for the shared factor.

3. Variational Training Objective and Disentanglement

The optimization is governed by a multi-term variational objective: $$\begin{aligned} \mathcal{L}_{\mathrm{ELBO}} =\ &\mathbb{E}_{q_{\phi_s}(z_s|V,L)}\left[ \mathbb{E}_{q_{\phi_v}(z_v|V)}[\log p_{\theta_v}(V|z_v)] + \mathbb{E}_{q_{\phi_l}(z_l|L)}[\log p_{\theta_l}(L|z_l)]\right] \ & - D_{\mathrm{KL}}(q_{\phi_v}(z_v|V) \| p(z_v)) - D_{\mathrm{KL}}(q_{\phi_l}(z_l|L) \| p(z_l)) - \mathrm{JSD}(q_{\phi_s}(z_s|V,L) \| p(z_s)) \end{aligned}$$ where JSD regularizes the mixture-of-Gaussians shared posterior.

Explicit disentanglement is enforced via two auxiliary penalties:

• Orthogonality Loss: After whitening the latent batches, Frobenius norms of cross-covariance between $(z_v, z_l, z_s)$ penalize statistical dependence, promoting orthogonality:

$$\mathcal{L}_{\mathrm{orth}} = \|\tilde{z}_s^\top \tilde{z}_v\|_F^2 + \|\tilde{z}_s^\top \tilde{z}_l\|_F^2 + \|\tilde{z}_v^\top \tilde{z}_l\|_F^2.$$

• Contrastive Alignment (InfoNCE): Ensures $z_s$ semantically predicts both modalities, using cosine similarity and temperature $\tau$ as in:

$$\mathcal{L}_{\mathrm{align}} = -\mathbb{E}\left[\log\frac{\exp(\mathrm{sim}(z_s, z_v)/\tau)}{\exp(\mathrm{sim}(z_s, z_v)/\tau)+\exp(\mathrm{sim}(z_s, z_l)/\tau)}\right] + (\mathrm{sym.})$$

The total loss sums these components, with tuning weights $\lambda_1 = \lambda_2 = 0.3$.

4. Handling Missing Modalities

Missing-modality robustness is integral to the MoE-VLVAE paradigm. If either vision or text is absent at inference, a "null" token is input for the missing branch, and the gating network predicts logits such that the corresponding $\pi$ weight is near zero. This reduces the posterior to depend exclusively on the present modality: $$q_{\phi_s}(z_s|V, \text{NULL}) \approx q_{\phi_s}(z_s|V),$$ preserving a valid ELBO and enabling unimodal-to-multimodal and cross-modal generation without retraining or network modification. During training, random drop-out of modalities (with $p \sim 0.3$) further encourages robust inference.

5. MoE-VLVAE in Disentangled Vision–Language Medical Report Generation

The DiA-gnostic VLVAE (Shaik et al., 8 Nov 2025) exemplifies the application of disentangled MoE-based VLVAE in radiology report generation, addressing modality-missingness and feature entanglement. The encoder decouples vision-specific ($z_v$), language-specific ($z_l$), and shared ($z_s$) latents, with $z_s$ inferred via the MoE barycenter. The decoder is a compact, cross-attention LLaMA-X transformer, leveraging all three latents for efficient report synthesis.

Empirical results on IU X-Ray (2% missing context) and MIMIC-CXR (45% missing context) demonstrate that DiA-gnostic VLVAE outperforms or matches state-of-the-art models across BLEU@4, ROUGE-L, and clinical F1, with notable robustness under missing clinical context. Ablation confirms that each core element (MoE latent, disentanglement, alignment) yields additive benefits, and full triangulation achieves highest robustness, e.g., F1 of 0.621 under complete and 0.438 under missing context, compared to much lower baselines.

Dataset	BLEU@4	ROUGE-L	Clinical F1
IU X-Ray	0.266	0.516	0.298
MIMIC-CXR	0.134	0.369	0.497

Values shown for DiA-gnostic VLVAE; see (Shaik et al., 8 Nov 2025) for SOTA comparisons and ablation.

6. Practical Considerations and Extensions

Key aspects for practical deployment include:

• Gating network: 1–2 layers of width 128–256; excessive capacity can induce expert collapse.
• Latents: Dimensionality 64–256 recommended; lower dims favor tighter alignment.
• Optimization: Adam optimizer, learning rate $1\mathrm{e}{-4}$ to $5\mathrm{e}{-4}$, weight decay $1\mathrm{e}{-5}$, 100–300 epochs.
• KL weighting ($\beta$-VAE): $\beta$ in [0.5, 2] trades off generation fidelity and regularization.
• Softmax temperature: Annealing from 1 to 0.1 increases gating sharpness.
• Cross-modal contrastive loss: Encouraged where modalities are weakly aligned.
• Missing modality augmentation: Randomly zeroing out one branch per mini-batch trains the network for missing-data robustness.

As an alternative to the KL-based barycenter, a 2-Wasserstein (Bures-Wasserstein) barycenter is available for geometry-aware fusion: $$q_{WB} = \arg\min_q \lambda_V W_2^2(q_V, q) + \lambda_L W_2^2(q_L, q)$$ with closed-form solutions for means and, in the diagonal covariance case, variances, thereby improving latent interpolation and robustness to distribution shift (Qiu et al., 29 Dec 2024).

7. Significance and Impact

MoE-based VLVAE methods, with explicit disentanglement and robust posterior aggregation, provide a scalable and resilient solution to multimodal generative modeling. These frameworks enable mass-covering and tractable posterior inference, efficient handling of missing data, and semantically faithful fusion critical in high-stakes domains such as clinical reporting. Empirical superiority over knowledge-graph and static fusion approaches, particularly under missing-modality conditions, suggests these models set a new baseline for robust vision–language generation and are extensible to other domains with similar multimodal challenges.