Enabling general-purpose multimodal interaction at scale

Determine effective approaches to enable general-purpose multimodal interaction within large-scale models trained on interleaved vision–language data, so that a single system can flexibly process and generate interleaved inputs and outputs across tasks.

Background

Beyond learning from long videos, the paper aims for a unified interface that naturally accepts and emits interleaved text and visual content across diverse tasks. The authors explicitly state uncertainty about how to achieve general-purpose multimodal interaction, indicating unresolved design and training principles for scalable, transferable interfaces.

Emu3.5 employs unified pretraining, supervised fine-tuning, and reinforcement learning to move toward this capability, but how to systematically enable and scale such interaction remains unclear and is identified as a foundational open question.