Role of natural language supervision in processing nuanced multimodal social signals

Determine the role and effectiveness of natural language supervision in processing nuanced multimodal social signals for Social-AI systems, including whether language can reliably guide social perception and behavior generation across modalities.

Background

Social signals during interactions are fine-grained and interleaved across modalities (e.g., gaze, posture, vocal emphasis), and small changes can significantly affect social meaning. While language has been shown to scaffold understanding in vision and audio and to guide motion generation, its effectiveness for nuanced multimodal social signal processing is not established.

The authors propose examining language as an intermediate representation for integrating multimodal social cues and question whether some social signals may be ineffable in language, highlighting the need to clarify the role of natural language supervision in this context.

References

While scientists are studying the capacity of language to scaffold visual understanding, audio understanding, and virtual agent motion generation, the role of natural language supervision for processing nuanced multimodal social signals remains an open question.

— Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions  (2404.11023 - Mathur et al., 2024) in Section 4, Subsection (C2) Nuanced Signals, C2 Opportunities and Open Questions

Ensuring reliable multimodal emotion reasoning remains an open challenge.

— Multi-Faceted Evaluation and Mitigation of Emotion Hallucinations in MLLMs  (2609.11154 - Zeng et al., 10 Sep 2026) in Section II-A, “Emotion Multimodal Large Language Models”

Models capable of jointly processing audio and text fall into two broad categories: audio-language foundation models (e.g., Wav2Vec 2.0 (Baevski et al., 2020), CLAP (Elizalde et al., 2022), and SpeechT5 (Ao et al., 2022)), which learn general-purpose audio representations; end-to-end conversational multimodal LLMs (e.g., Qwen3-Omni, GPT-5.1, and Gemini 3. Pro), which map speech directly into semantic representations within unified dialogue frameworks. This work does not adopt the latter due to their substantial computational requirements and uncertain impact on specialized ER tasks, leaving their exploration to future work.

— Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue  (2609.39072 - Hu et al., 30 Sep 2026) in Section 2.3, Available Multimodal Models for Audio and Text Data