MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation

Abstract: We introduce MUNI, an end-to-end multimodal latent diffusion framework for any-to-any generation that unifies subset-conditioned cross-modal generation and unconditional joint sampling through a shared stochastic latent. Existing multimodal generative models are largely LLM-based, which limits leveraging modality-specific generators and requires text-paired data for training. Recent diffusion- and flow-based any-to-any extensions take a different direction but still rely on text-aligned embeddings, fully-paired training, or matched-dimensionality deterministic mappings. MUNI rests on two complementary contributions, one architectural and one in the training objective. First, we extend latent diffusion to multimodal any-to-any generation end-to-end: instead of the standard two-stage recipe that precomputes a frozen latent space and then fits a prior over it, MUNI jointly trains modality-specific encoders, expressive decoders, and a single shared flow-based prior under one objective. Second, we identify that the standard aggregation rules of multimodal variational inference are insufficient once coupled with a learned prior and expressive decoders. A suitable shared latent must simultaneously satisfy coherence across generated modalities, predictive sufficiency of subset latents, and minimality of the latent content. We propose a routed training objective whose structural choices align the latent with these criteria and admit a minimal-sufficiency characterization in the realizable setting. Experiments on PolyMNIST-Quadrant-Labels and a large-scale image-text-audio benchmark show MUNI matching or exceeding the strongest baselines on conditional generation while opening its largest margins on unconditional coherence. Project page: https://muni-proj.github.io/.

• The paper introduces a unified latent diffusion model that enables any-to-any multimodal generation without relying on fully paired datasets.
• It leverages modality-specific encoders, expressive decoders, and a flow-based latent prior to ensure coherence, predictive sufficiency, and minimality in the latent space.
• Empirical evaluations demonstrate significant improvements in cross-modal alignment and unconditional generation coherence over existing state-of-the-art methods.

MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation

Overview and Motivation

MUNI introduces a unified probabilistic framework for any-to-any multimodal generation that fundamentally addresses persistent drawbacks in prior multimodal VAEs and diffusion/flow-based approaches. The core novelty is the integration of jointly-learned modality-specific encoders, expressive decoders, and a shared flow-based latent prior, enabling coherent generation from arbitrary observed modality subsets—all without requiring text-aligned embeddings, fully-paired data, or dimensionality-matching constraints across modalities.

Conventional multimodal diffusion models typically operate in two modes—subset-conditioned cross-modal generation and unconditional joint sampling—but rarely unify both in a single tractable latent variable. Previous architectures, such as those based on text-aligned latent spaces or invertible flows, suffer from information bottlenecks, data inefficiency, and inability to model complex multimodal dependencies, particularly with partially observed data. MUNI builds upon the unified latent paradigm, learning all components end-to-end, and introduces principled objective modifications to ensure three properties in the latent: coherence-sufficiency for joint generation, predictive sufficiency for cross-modal completion, and minimality for efficient prior modeling.

Technical Contributions

Unified Multimodal Latent Diffusion

MUNI generalizes latent diffusion to the multimodal case by implementing:

• Modality-specific encoders mapping each observed input to a shared latent space,
• Expressive (modality-specific) decoders for reconstructing or predicting missing modalities,
• A single learned flow-based prior over the shared latent, supporting both subset-conditioned and unconditional sampling.

Conditioning on any subset $S$ of input modalities, MUNI encodes the subset into a posterior $q_{\phi,S}(z|x_S)$ (via aggregation of unimodal posteriors), and generates the missing modalities $\{x_m: m\notin S\}$ via the corresponding decoders $p_{\theta_m}(x_m|z)$. Unconditional sampling is achieved by generating $z$ from the learned prior $q_{\psi}(z)$.

Latent-Space Criteria

The authors formally identify three necessary and jointly-sufficient properties for the shared latent variable to enable coherent multimodal generation:

1. Coherence-Sufficiency: The latent must explain all inter-modality dependencies, measured by vanishing conditional total correlation: $TC(X_{1:M}|Z)=0$.
2. Predictive Sufficiency: For any conditioning subset, the latent must contain enough information from observed modalities to predict each unobserved modality: $I(X_{A}; X_m|Z_A)=0$ for $m \notin A$.
3. Minimality: The latent should not encode extraneous, modality-specific variation unnecessary for cross-modal prediction, delegating such variance to the decoders.

Conventional mixture-based (MoE) or simple product-of-experts (PoE) aggregation do not ensure all three properties, leading to degraded many-to-one and unconditional generation performance.

Objective Design and Training

Crucially, MUNI modifies the baseline unified-latent ELBO by:

• Replacing mixture-based with non-mixture (PoE/Hellinger) aggregation for subset posteriors, ensuring all available evidence informs the latent.
• Using target-detached self-reconstruction: for self-reconstruction, encoder gradients are detached for the target modality, preventing over-encoding modality-private content in the shared latent.
• Restricting prior learning to full-modality or leave-one-out latents only, precluding under-specified priors and supporting unconditional co-generation.

The authors provide formal justification (Proposition 4.1) that this routine produces coherence- and prediction-sufficient latents with minimal redundancy, under realistic modeling and optimization assumptions.

Empirical Evaluation

PolyMNIST-Quadrant-Labels

In a synthetic, controlled setting with tightly specified shared and private factors (PolyMNIST-Quadrant-Labels), MUNI is compared against leading multimodal VAE baselines (MVAE, MMVAE, MoPoE, HELVAE) and specialist methods. MUNI not only matches but often exceeds the best conditional (label-to-image) generation performance, with the most pronounced advantage in unconditional co-generation coherence—demonstrating that the shared latent enforces consistent content across all modalities in generated tuples.

Image-Text-Audio Any-to-Any Generation

On a large-scale, unpaired image-text-audio corpus, MUNI is positioned against multimodal generalists such as CoDi, OmniFlow, FlowBind, and strong VAE-based baselines. Here, MUNI achieves:

• The best generalist performance on many-to-one cross-modal alignment, with substantial AIS improvements ($+7.14$ and $q_{\phi,S}(z|x_S)$0 points over the next-best generalist in audio-image and image-audio alignment, respectively),
• Strongest unconditional co-generation coherence, outperforming the nearest generalist by $q_{\phi,S}(z|x_S)$1 CLIP, $q_{\phi,S}(z|x_S)$2 CLAP, and $q_{\phi,S}(z|x_S)$3 AIS.

MUNI also remains competitive on one-to-one translation, indicating that its design does not trade off conditional fidelity for broader any-to-any capability.

Implications and Future Directions

Practical Implications

MUNI's architecture alleviates the need for fully-paired multimodal data, making efficient use of weakly paired or unpaired observations. This capability is critical as real-world multimodal sources are rarely fully observed or perfectly aligned. The framework’s demonstrated superiority on large-scale audio-image-text datasets further validates its scalability.

Theoretical Implications

The latent-space analysis extends the theory of multimodal variational inference: the paper specifies not only how subset posteriors should be aggregated, but also precisely which subset latents are valid targets for prior learning. This clarifies and resolves longstanding ambiguities in the literature on the interplay between latent design, aggregation, and coherence across complex multimodal distributions.

Limitations and Future Work

While MUNI’s objective is general across modalities, it does not exploit known structure or asymmetries—e.g., deterministic versus stochastic relationships—in specific modality pairings. Incorporating structurally-aware route sampling, targeted loss design, or modular architectures could enhance performance in asymmetrical multimodal domains such as speech (where text-to-speech is multi-modal, but speech-to-text is nearly deterministic). Efficient extension to even more modalities, and scaling to larger corpora, are promising research directions.

Conclusion

MUNI establishes a new baseline for coherent any-to-any multimodal generation by enforcing principled latent-space criteria and integrating expressive generative modeling in a tractable unified-latent framework. Experimental results demonstrate that careful aggregation and training of the shared latent are crucial for both cross-modal completion and unconditional joint generation. This work substantially advances both the practical capabilities and the theoretical understanding of scalable, coherent multimodal generative modeling (2606.16408).