Effectiveness of MoS under Early-Fusion Training

Determine whether the Mixture of States (MoS) multimodal diffusion architecture achieves effective performance when trained in early-fusion settings where the understanding transformer and the generation transformer are jointly trained with bidirectional interactions, specifically by equipping the MoS router with multiple projection layers to establish transformer connections in both directions.

Background

The Mixture of States (MoS) framework introduced in this paper uses a dual-tower architecture consisting of an understanding transformer (frozen during training) and a generation transformer, with a lightweight learnable router enabling adaptive, token-level, timestep-dependent interactions. The training strategy adopted is multi-stage and focuses compute on the generation tower while keeping the understanding tower fixed.

Prior work on Mixture of Transformers (MoT) has shown strong scalability under early-fusion training regimes where modalities are fused and trained jointly with symmetric layer correspondences. The authors note that while MoS performs well for multimodal generation in their current setup, it remains unknown whether MoS would retain or improve effectiveness under early-fusion training. They propose a principled extension—adding multiple projection layers to the router to enable bidirectional transformer connections—but defer experimentation due to computational and data constraints.