Fine-grained Cross-modal Control for Expressive Multi-speaker Dialogue

Develop models and training strategies that achieve fine-grained cross-modal control across speech, vision, and text to generate expressive multi-speaker dialogues, enabling nuanced controllability of interactional style while maintaining cross-modal alignment.

Background

The paper surveys recent advances in spoken dialogue generation and cinematic video generation, noting that while modality-specific capabilities have improved, coordinated control across modalities for dialogues remains difficult.

To address this gap, the authors introduce MM-DIA and MM-DIA-BENCH to facilitate research on conditional multimodal dialogue generation, but they explicitly acknowledge that achieving fine-grained cross-modal control is still unresolved.