The Physics of Multimodal Pretraining: Knowledge Flow and Early Unification
This presentation examines the mechanistic laws governing unified multimodal pretraining through controlled experiments on vision-language models. The researchers uncover asymmetric knowledge flow between modalities, reveal how task complexity determines synergy versus competition, demonstrate the necessity of early simultaneous unification, and propose efficient data recipes that achieve state-of-the-art performance with minimal generative data.Script
Language doesn't just help vision. It transforms it. The authors discovered that knowledge flows strictly one way across modalities: language boosts both visual understanding and generation, understanding enables generation, but generation offers almost nothing in return.
The researchers found that complexity determines whether modalities cooperate or compete. When paired with simple synthetic images, language modeling improves. But introduce real-world visual complexity, and the two modalities fight for the same neural capacity, degrading both.
The solution lies in selective architectural decoupling. Giving each modality its own feed-forward networks while keeping attention and normalization shared unlocks maximum synergy. Each modality gets specialized capacity for transformation while attention routes information across the boundary.
Timing matters more than anyone expected. The authors prove that introducing vision only after extensive language pretraining permanently cripples visual capabilities. Sequential training, even with replay buffers to prevent forgetting, cannot match the performance of models trained on all modalities from day one.
Late fusion creates what the researchers call vision laziness. When language dominates early training, the model's vision pathways atrophy. Vision tokens receive less attention, their embeddings drift away from the language manifold, and vision-specific parameters activate weakly, as if the visual system became a vestigial afterthought.
Armed with these mechanistic insights, the authors propose a radically asymmetric training recipe: 70 percent language, 25 percent visual understanding, just 5 percent generation. This lopsided mixture achieves state-of-the-art generative performance at a fraction of the compute cost, because the foundation for generation comes from language and understanding, not brute force data volume. Explore the full physics of multimodal pretraining and create your own explanatory videos at EmergentMind.com.
