What is Multimodality?

Abstract: The last years have shown rapid developments in the field of multimodal machine learning, combining e.g., vision, text or speech. In this position paper we explain how the field uses outdated definitions of multimodality that prove unfit for the machine learning era. We propose a new task-relative definition of (multi)modality in the context of multimodal machine learning that focuses on representations and information that are relevant for a given machine learning task. With our new definition of multimodality we aim to provide a missing foundation for multimodal research, an important component of language grounding and a crucial milestone towards NLU.

• The paper introduces a task-relative definition of multimodality that shifts focus from traditional rigid categories to context-specific data integration.
• It critiques conventional definitions by highlighting their limitations in addressing the nuances of diverse data interactions in ML tasks.
• It emphasizes that aligning modality selection with specific task needs may drive innovations in language grounding and natural language understanding.

The paper "What is Multimodality?" addresses the evolving landscape of multimodal machine learning and argues that traditional definitions of multimodality are inadequate for contemporary applications. The authors propose a novel, task-relative definition of multimodality that emphasizes the importance of context-specific representations and information relevant to specific machine learning tasks.

Key Points

1. Critique of Traditional Definitions:
   • The paper highlights that existing definitions of multimodality tend to be rigid and outdated. These definitions typically focus on broad categorizations like vision, text, or speech without considering the nuances of how these modalities interact in specific tasks.
2. Task-Relative Definition:
   • The authors propose a new conceptual framework where multimodality is defined relative to the task at hand. This means considering the modes of input that contribute meaningfully to solving a particular task, rather than adhering to predefined categories.
3. Role in Multimodal Machine Learning:
   • By adopting a task-relative approach, the paper argues that researchers can better align their models with the actual needs of a given task, potentially leading to more effective and efficient machine learning solutions.
4. Implications for Language Grounding and NLU:
   • The discussion ties into broader themes in natural language understanding (NLU) and language grounding, suggesting that a deeper comprehension of multimodality can enhance these fields by providing more robust foundational concepts.
5. Potential for Innovation:
   • The authors see this new definition as a foundational step that could drive future innovations in multimodal research, paving the way for advancements in machine learning applications that require the integration of diverse data types.

Overall, the paper calls for a shift in how researchers conceptualize and operationalize multimodality, advocating for definitions that are flexible and directly tied to the specificities of each task. This approach is positioned as a crucial development for advancing both the theory and application of multimodal machine learning.