Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

Abstract: Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts. While existing paradigms like Mixture-of-Transformers (MoT) mitigate this conflict through structural isolation, they fundamentally sever cross-modal synergy and suffer from capacity fragmentation. In this work, we present Symbiotic-MoE, a unified pre-training framework that resolves task interference within a native multimodal Mixture-of-Experts (MoE) Transformers architecture with zero-parameter overhead. We first identify that standard MoE tuning leads to routing collapse, where generative gradients dominate expert utilization. To address this, we introduce Modality-Aware Expert Disentanglement, which partitions experts into task-specific groups while utilizing shared experts as a multimodal semantic bridge. Crucially, this design allows shared experts to absorb fine-grained visual semantics from generative tasks to enrich textual representations. To optimize this, we propose a Progressive Training Strategy featuring differential learning rates and early-stage gradient shielding. This mechanism not only shields pre-trained knowledge from early volatility but eventually transforms generative signals into constructive feedback for understanding. Extensive experiments demonstrate that Symbiotic-MoE achieves rapid generative convergence while unlocking cross-modal synergy, boosting inherent understanding with remarkable gains on MMLU and OCRBench.

• The paper presents a novel bimodal expert grouping strategy that unifies text and visual processing with significant routing balance and improved multimodal performance.
• It demonstrates enhanced stability in both generation and understanding by maintaining modality-aware expert partitions, outperforming Standard MoE on benchmarks like POPE and GQA.
• The work underscores the importance of shared semantic pathways and structural synergy to prevent catastrophic forgetting and enable robust compositional generation.

Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

Architectural Rationale and Modality-Aware Expert Disentanglement

Symbiotic-MoE introduces a bimodal expert grouping strategy that partitions MoE experts into a unified Understanding Group (Text+ViT) and a specialized Generation Group (VAE). The empirical justification for this architecture is established via routing dynamics analyses, revealing a structurally coupled subset of "core experts" within the pre-trained foundation VLM. Macro-level routing statistics yield imbalance ratios frequently exceeding 2.5 and reaching up to 4.26, evidencing severe concentration of capacity usage among a minority of experts, hence justifying non-uniform modality splits (Figure 1).

Figure 1: Imbalance Ratio across MoE layers demonstrates capacity concentration and the necessity for modality-aware expert grouping.

Micro-level analysis in Layer 16 shows significant entanglement of modalities within the same core experts. Text and ViT tokens, although distinct, consistently activate overlapping top experts (e.g., Experts 16, 52, and 122), empirically demonstrating shared semantic pathways between linguistic and visual processing (Figure 2).

Figure 2: Micro-level modality entanglement showing shared expert activation for Text and ViT tokens.

Ablation results (not shown in this essay) confirm that tripartite splits violently disrupt these semantic couplings, yielding catastrophic performance degradation. Therefore, the proposed bimodal grouping maintains pre-trained stability and enables further generative adaptation.

Quantitative Benchmarking and Routing Capacity Dynamics

Symbiotic-MoE is evaluated via an expanded suite of multimodal benchmarks, including POPE, GQA, ChartQA, AI2D, TQA, and MME. The model preserves and improves understanding and reasoning capabilities, with explicit avoidance of performance regression across specialized domains. Notably, Symbiotic-MoE surpasses Standard MoE and MoT on POPE (74.5 vs 60.3/63.1) and GQA (48.0 vs 38.5/40.9), demonstrating effective resistance to catastrophic forgetting and alignment collapse.

Routing dynamics are visualized to verify modality-specific load balancing. Disentangled capacity rate curves show all modalities exceeding 0.90 throughout the co-training phase, ruling out partial routing collapse and confirming architectural robustness (Figure 3, Figure 4).

Figure 3: Individual modality capacity rates; all maintain high utilization, confirming system-wide routing balance.

Figure 4: Cumulative token consumption dynamics validating rigorous, modality-proportional co-training.

These results demonstrate that Symbiotic-MoE transforms generative gradient signals into constructive regularization, rather than destructive interference, bridging the understanding-generation synergy gap endemic to previous architectures.

Training Protocol, Data Mixture, and Evaluation Methodology

Symbiotic-MoE leverages the Hunyuan-VL-30B-A3B backbone unified by the Transfusion framework, strictly restricting experiments to the pre-training (PT) phase at a base pixel budget ($256 \times 256$). The corpus comprises a balanced mixture (T2I:T2I-Long:LM:MMU = 3:3:2:2) of text, aligned pairs, and interleaved multimodal data, ensuring both generative experts and understanding modules are deeply regularized during adaptation.

The evaluation rigorously decouples generative and understanding objectives with metrics such as GenEval, FID, CLIPScore, HPSv2, and T2I-CompBench for generation; natural language and perception tasks (MMLU, OCRBench, POPE, GQA, TQA, ChartQA, AI2D, MME) for understanding. This holistic methodology provides granular evidence for both preservation of pre-trained capabilities and enhancement of compositional generation.

Qualitative Synthesis and Structural Synergy

Extended qualitative comparisons reveal that Standard MoE suffers irreversible structural collapse due to unconstrained gradient interference, while MoT, despite structural isolation, fails to preserve semantic grounding, resulting in attribute hallucination and geometric incoherence. Symbiotic-MoE, in contrast, synthesizes semantically precise, photorealistic, and geometrically consistent results, as demonstrated by accurate attribute binding and structure retention in complex compositional scenarios (Figure 5).

Figure 5: Side-by-side generative comparison; Symbiotic-MoE achieves stabilizing semantic and geometric fidelity.

The shared experts act as an effective semantic bridge, enabling the generative module to query deeply aligned text representations, strictly binding fine-grained textual attributes to visual entities while avoiding holistic collapse or hallucination.

Implications and Forward Trajectories

Practically, Symbiotic-MoE enables large-scale multimodal models to maintain multi-domain reasoning and robust generative capacity within a fixed memory and computational budget, suitable for deployment in resource-constrained environments. Theoretically, it exemplifies how intrinsic semantic couplings in pre-trained sparse MoEs must be respected to achieve optimal understanding-generation synergy, challenging prior assumptions regarding strict modality or task isolation.

Future developments may explore dynamic expert allocation via learned amalgamation, further scaling unified architectures in both modality and task heterogeneity. Advances in routing algorithms, more complex data mixtures, and integration with in-context learning frameworks are anticipated to enhance both universal intelligence and compositional generation without sacrificing foundational stability.

Conclusion

Symbiotic-MoE empirically validates a modality-aware, shared-expert approach to unify generation and understanding in foundation models. Rigorous analyses demonstrate profound semantic coupling at both macro and micro routing levels, structural stability across training dynamics, and robust qualitative synthesis. The architectural paradigm unlocks cross-modal synergies unattainable by prior modalities or task-isolated systems, providing a scalable path forward for multi-domain, multimodal intelligence (2604.07753).