LatentUMM: Dual Latent Alignment for Unified Multimodal Models

Abstract: Unified multimodal models (UMMs) achieve strong performance in both understanding and generation by learning a shared latent space, yet they often exhibit functional inconsistency between these two capabilities. We observe that this issue does not stem from a lack of shared representations, but from the absence of explicit alignment between the transformations that map into and out of the latent space. As a result, generation and re-encoding can follow inconsistent trajectories, leading to semantic drift under modality transitions. In this work, we propose LatentUMM, a framework that constructs an enhanced shared latent space to explicitly align these transformations and improve cross-modal consistency. LatentUMM consists of two stages. First, dual latent alignment enforces consistency at both the modality and capacity levels: cross-modal alignment uses a stronger embedding model to impose structured cross-modal semantics, while dual capacity alignment enforces bidirectional consistency under generation and re-encoding. Second, latent dynamics stabilization improves robustness via stochastic latent rollouts and preference optimization, favoring trajectories that better preserve semantic consistency. Experiments show that LatentUMM consistently improves multimodal consistency across diverse architectures. Code is available at: https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/LatentUMM.

• The paper presents a dual latent alignment framework that minimizes semantic drift between understanding and generation.
• The paper employs bidirectional cross-modal alignment and stochastic latent rollouts to enhance latent space stability.
• The paper demonstrates significant improvements on benchmarks like MME, MMVet, and DPG-Bench, evidencing robust consistency and performance.

LatentUMM: Dual Latent Alignment for Unified Multimodal Models

Introduction and Motivation

Unified multimodal models (UMMs) have advanced the fusion of multimodal understanding and generation by sharing latent representation spaces across modalities. Despite joint training on paired data, current UMMs exhibit marked inconsistency between understanding and generation: while generation-conditioned outputs (e.g., images from text) can be high-quality, the re-interpretation (understanding) of these outputs often reveals semantic drift and incomplete preservation of the intended content. This inconsistency is rooted not in an absence of shared representations, but in the lack of coordination between the mappings into and out of the shared latent space.

The paper introduces LatentUMM, a post-training framework designed to explicitly align the latent mapping functions involved in both understanding and generation. Rather than merely utilizing a common representation, LatentUMM enforces dual latent alignment—across both modalities and functional capacities (understanding vs. generation)—while introducing a robust stabilization mechanism for latent space dynamics.

Figure 1: Overview of baseline UMMs (left) versus the proposed LatentUMM (right), illustrating improved consistency and reduced semantic drift.

Methodology: Dual Latent Alignment and Stabilization

Enhanced Shared Latent Space

LatentUMM builds on a pretrained UMM by introducing a stronger, fixed embedding model (e.g., Gemini Embedding), projecting multimodal inputs into a refined latent space of matched dimensionality. The original latent fusion function is retained unless explicitly adapted, ensuring the new structure imposes minimal architectural overhead.

Dual Latent Alignment

Alignment is enforced at two levels:

• Dual Modal Alignment: Both visual and textual modalities are mapped into the refined latent space via the external embedding model. The squared Euclidean distance between paired embeddings is minimized, explicitly reducing the gap between modalities.
• Dual Capacity Alignment: Bidirectional consistency across understanding and generation is enforced. Starting with a latent embedding, the model generates an output (e.g., image), which is then re-embedded using the external model. The original and regenerated embeddings are aligned, minimizing cross-modal functional divergence.

  Figure 2: Overview of LatentUMM, highlighting the dual alignment (modal and capacity) and rollout-based stabilization pipeline.

Latent Dynamics Stabilization

Simple alignment at the instance level may lack distributional robustness, given the high complexity and variability in real applications. To counter this, stochastic latent rollouts are introduced: latent embeddings are perturbed with Gaussian noise, and the model's capacity to maintain semantic consistency across multiple candidate rollouts is assessed. A preference optimization objective focuses on maximizing the gap between the most and least consistent trajectories, stabilizing the latent dynamics against drift.

Experimental Results

Consistency and Downstream Task Performance

LatentUMM's efficacy is demonstrated across a battery of benchmarks for multimodal understanding, generation, editing, and unification, covering both in-domain and out-of-domain settings.

• Multimodal Understanding: LatentUMM achieves the strongest gains on MME (1696.1), MMVet (67.2), and MathVista Free-Form (65.65), outperforming state-of-the-art post-training methods. Improvement is particularly pronounced on open-ended reasoning, reflecting successful coordination between generative and interpretive pathways.
• Multimodal Generation: On DPG-Bench, LatentUMM yields superior scores—especially in entity, attribute, and "Other" dimensions—indicating enhancement in both canonical and non-standard content composition.
• Editing: Quantitative advances are noted for semantic correctness and perceptual quality, reflecting robust editability with preserved intent and visual fidelity.

Consistency Improvement and Analysis

Direct evaluation using unified benchmarks (Unified-Bench, RealUnify) reveals systematic reductions in semantic drift throughout repeated cross-modal transformations, with cumulative error low and divergence growing significantly slower than in baselines.

Figure 3: Latent space analysis showing tighter alignment of image and text embeddings in LatentUMM, as indicated by lower pairwise gaps and condensed CDF of distances.

Case studies on sequential interaction tasks reveal that LatentUMM preserves both temporal and spatial logic throughout generative and interpretive cycles, whereas baselines reorder or lose information during loopbacks.

Ablation Studies

• Latent Space Construction: Explicit construction of an enhanced latent space (vs. alignment in the original space) is a major contributor to improvements.
• Embedding Model: Gemini Embedding offers marginal gains over CLIP/SigLIP; however, LatentUMM's gains are not wholly dependent on one variant.
• Rollouts and Noise: Moderate noise and rollout depth yield maximal performance; excessive perturbation leads to collapse or representational drift.

Generalization and Efficiency

LatentUMM improvements generalize robustly across core UMM architectures (e.g., Bagel, Janus-Pro, Harmon), with the most pronounced gains on weaker baselines. Architectural-agnosticism is thus established.

Training efficiency is maintained, with additional computational overhead from embedding projections and rollouts being minor relative to backbone cost.

Qualitative Image Generation

Figure 4: Qualitative image generation, evidencing LatentUMM's enhancements in multi-object handling, attribute fidelity, structured layouts, and fine detail rendering.

Qualitative outputs demonstrate concrete gains in the spatial arrangement of multiple objects, attribute accuracy, and maintenance of scene composition, highlighting the practical impact of explicit latent alignment.

Limitations

Two primary failure modes are identified: (1) excessive latent noise during rollouts can destabilize the semantic manifold, and (2) over-weighted consistency constraints can reduce generative diversity, leading to representational collapse. Performance remains sensitive to hyperparameters such as embedding model, noise scale, and consistency weighting.

Figure 5: Expected output illustrating proper semantic composition.

Figure 6: Output under excessive rollout-induced noise or over-regularization, showing partial semantic collapse.

Theoretical and Practical Implications

LatentUMM demonstrates that shared representation alone is insufficient for functional multimodal unification. Explicit bidirectional and cross-modal alignment, augmented by robust stabilization in the latent space, is essential for dependable compositionality and reasoning. As UMMs are applied to increasingly complex real-world reasoning, consistent latent-geometric dynamics will become a critical requirement.

Practically, LatentUMM offers an architecture-agnostic, efficient post-training solution that measurably improves downstream consistency and accuracy. It points toward a methodological shift in UMM research: away from scaling alone, toward coordinated latent-space regularization and functional coupling.

Conclusion

LatentUMM introduces a dual latent alignment framework for UMMs, coupling multimodal understanding and generation via both geometric and functional consistency in a refined latent space. Extensive evaluations show statistically significant gains in consistency and generalization across tasks and architectures. The framework sets a methodological precedent for explicit control over latent-space dynamics, inviting future work on semantic stability, latent supervision, and further decomposition of cross-modal reasoning pathways (2605.17766).