Semantic Generative Tuning for Unified Multimodal Models

Abstract: Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture. However, prevailing training paradigms independently optimize understanding via sparse text signals and generation through dense pixel objectives. Such a decoupled strategy yields misaligned representation spaces, isolating visual understanding from generation and hindering their mutual reinforcement. This work presents the first systematic investigation into generative post-training, where we formulate hierarchical visual tasks as generative proxies to bridge the isolation in UMMs. Our empirical investigation reveals that high-level semantic tasks, particularly image segmentation, serve as optimal proxies. Unlike low-level tasks that distract models with texture details, segmentation provides structural semantics that significantly enhance both vision-centric perception and generative layout fidelity. Building upon these insights, we introduce Semantic Generative Tuning (SGT), a novel paradigm that leverages segmentation as a generative proxy to align and synergize multimodal capabilities. Mechanistic analyses further demonstrate that SGT fundamentally improves feature linear separability and optimizes visual-textual attention allocation pattern. Extensive evaluations show that SGT consistently improves both multimodal comprehension and generative fidelity across mainstream benchmarks. Our code is available on the https://song2yu.github.io/SGT/.

• The paper introduces Semantic Generative Tuning (SGT) to align visual understanding and generation using segmentation as a proxy.
• SGT improves multimodal benchmarks with high structural abstraction and reduces feature entanglement in vision-centric tasks.
• Evaluation shows a 6.02% increase in performance with SGT-BAGEL, achieving high scores in text-to-image generation.

Semantic Generative Tuning for Unified Multimodal Models

Motivation and Context

Unified Multimodal Models (UMMs) are designed to consolidate visual understanding and visual generation within a single architecture. Despite recent progress, existing training paradigms have been fundamentally decoupled: visual understanding is optimized via sparse textual supervision, while visual generation relies on dense, low-level pixel-level objectives. This representational misalignment impedes bidirectional transfer and prevents the emergence of true synergy between understanding and generation modalities, as can be seen in the disjoint feature spaces produced by conventional pipelines (Figure 1).

Figure 1: Comparison of alignment strategies for UMMs, contrasting traditional decoupled optimization, pixel-level proxies, and SGT's semantic-level alignment.

To address this, the paper introduces the first systematic investigation of generative post-training using hierarchical visual tasks as generative proxies. It establishes that high-level semantic tasks, particularly image segmentation, serve as optimal proxies for coupling modality spaces. Semantic Generative Tuning (SGT) is proposed, leveraging segmentation as a generative proxy to align and synergize multimodal capabilities.

SGT Paradigm and Hierarchical Visual Proxies

SGT formalizes generative tuning as a conditional generation process, with input modalities processed via dual ViT and VAE encoders to extract semantic and structural features. The training objective is to maximize alignment by mapping RGB images and concise textual instructions into a shared representation space, then generating high-level semantic targets such as segmentation masks rather than low-level pixel reconstructions (Figure 2).

Figure 2: SGT generative tuning overview, demonstrating modality encoding and semantic-level proxy optimization via segmentation.

Empirical analysis is performed using a hierarchical taxonomy: high-level (segmentation, object detection), mid-level (depth estimation, inpainting), and low-level (edge detection, image restoration). Evaluations indicate the following:

• High-level semantic tasks consistently outperform low-level counterparts in multimodal understanding benchmarks, providing superior structural abstraction and mitigating feature entanglement.
• Visual supervision primarily enhances perception rather than logical reasoning; improvements are notable in vision-centric, spatial reasoning, and hallucination resistance benchmarks, but knowledge-intensive tasks remain static.
• Spatial layout fidelity in generative tasks is robust across proxy tasks, indicating that proxy-guided reconstruction imposes explicit spatial constraints regardless of semantic granularity.

These findings motivate the adoption of segmentation as the sole generative proxy in SGT, resulting in better alignment and mutual reinforcement between modalities.

Strong Numerical Results and Claims

Comprehensive evaluation across multiple UMM architectures (BAGEL, OmniGen2) and benchmarks validates SGT:

• SGT-BAGEL achieves a 6.02% increase over baseline on CV-Bench and scores 90% on GenEval for text-to-image generation.
• Ablation studies confirm segmentation as the optimal proxy: SFT+SGT outperforms SFT+Edge and SFT+Reconstruction across vision-centric, spatial, and hallucination benchmarks.
• Performance scales monotonically with the quantity of segmentation data; optimal intra-batch mixture is 1:2 VQA-to-segmentation.
• SGT integration accelerates convergence and enhances robustness across vision-centric tasks.

These results are consistent across architecture scale and design (Figure 3).

Figure 3: Training dynamics with different SFT:Seg ratios, illustrating accelerated convergence and higher final scores with segmentation data integration.

Qualitative results further demonstrate SGT's advantage in compositional text-to-image generation, including adherence to spatial and color instructions (Figure 4), and highlight the higher visual fidelity and diversity of generated outputs (Figure 5).

Figure 4: Qualitative comparison on compositional text-to-image generation, confirming semantic adherence and improved alignment.

Figure 5: High-quality, diverse images generated by SGT across wide-ranging prompts.

Mechanistic Analysis

The paper provides mechanistic insights into why SGT unlocks synergy:

• Feature linear separability is improved; t-SNE visualizations reveal that segmentation supervision yields discriminative embeddings with clear class separation, avoiding cluster entanglement in confusable categories (Figure 6).

Figure 6: Feature space analysis on fine-grained classes, confirming improved intra-class cohesion and inter-class separation with SGT.

• Vision-language attention allocation is optimized, with deeper transformer layers allocating more attention to visual tokens, mitigating linguistic prior over-reliance (Figure 7).
• Token-level attention during generation demonstrates amplified focus on semantically salient tokens, particularly those specifying object, position, and color constraints (Figure 8).

Figure 7: Analysis of cross-modal attention patterns, showing increased visual focus in deeper layers.

Figure 8: Token-level attention distribution during image generation, with SGT promoting critical token weighting for spatial and object placement.

Practical and Theoretical Implications

SGT establishes a semantic-level alignment strategy that overcomes the long-standing optimization divergence between visual understanding and generation. Practically, this paradigm delivers improved generalization, enhanced layout fidelity, and accelerates training convergence for UMMs, making it suitable for advanced tasks like in-context visual editing or compositional image synthesis. Theoretically, it offers a framework for unified representation learning, emphasizing the necessity of high-level semantic proxies for cross-modal coupling.

SGT is primarily advantageous as a foundational alignment method. Its current reliance on segmentation constrains performance in knowledge-intensive or symbolic tasks; integration with VQA data or further fusion with explicit generative supervision is required for optimal multimodal generality. This points to future research avenues incorporating reinforcement learning and more complex proxy mixtures.

Conclusion

Semantic Generative Tuning is proposed as a principled paradigm for UMMs, shifting multimodal alignment from pixel-level reconstruction to semantic-level structure. Empirical and mechanistic analyses confirm that segmentation-driven generative proxy fundamentally enhances feature separability and optimizes attention allocation, resulting in substantial and scalable improvements in visual understanding and generation. This approach provides a robust foundation for developing cohesive, versatile multimodal systems and has broad implications for future AI research and deployment (2605.18714).