Role of natural language supervision in processing nuanced multimodal social signals

Determine the role and effectiveness of natural language supervision in processing nuanced multimodal social signals for Social-AI systems, including whether language can reliably guide social perception and behavior generation across modalities.

Background

Social signals during interactions are fine-grained and interleaved across modalities (e.g., gaze, posture, vocal emphasis), and small changes can significantly affect social meaning. While language has been shown to scaffold understanding in vision and audio and to guide motion generation, its effectiveness for nuanced multimodal social signal processing is not established.

The authors propose examining language as an intermediate representation for integrating multimodal social cues and question whether some social signals may be ineffable in language, highlighting the need to clarify the role of natural language supervision in this context.