Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

Abstract: Sarcasm is often expressed through several verbal and non-verbal cues, e.g., a change of tone, overemphasis in a word, a drawn-out syllable, or a straight looking face. Most of the recent work in sarcasm detection has been carried out on textual data. In this paper, we argue that incorporating multimodal cues can improve the automatic classification of sarcasm. As a first step towards enabling the development of multimodal approaches for sarcasm detection, we propose a new sarcasm dataset, Multimodal Sarcasm Detection Dataset (MUStARD), compiled from popular TV shows. MUStARD consists of audiovisual utterances annotated with sarcasm labels. Each utterance is accompanied by its context of historical utterances in the dialogue, which provides additional information on the scenario where the utterance occurs. Our initial results show that the use of multimodal information can reduce the relative error rate of sarcasm detection by up to 12.9% in F-score when compared to the use of individual modalities. The full dataset is publicly available for use at https://github.com/soujanyaporia/MUStARD

• The paper introduces MUStARD, a multimodal dataset that integrates audiovisual cues and dialogue context to enhance sarcasm detection.
• The paper demonstrates that combining textual and visual features reduces the relative error rate by up to 12.9% compared to unimodal methods.
• The paper explores the role of conversational context and speaker characteristics, highlighting both benefits and challenges in multimodal analysis.

Towards Multimodal Sarcasm Detection

The paper advances the field of sarcasm detection by advocating for a multimodal approach, aiming to integrate verbal and non-verbal cues. While sarcasm detection has predominantly focused on textual data, this paper underscores the potential improvements achievable when considering various modalities. The proposed resource, named the Multimodal Sarcasm Detection Dataset (MUStARD), comprises audiovisual utterances derived from popular TV shows, annotated with sarcasm labels. This dataset serves as a foundation for exploring how multimodal signals can enhance sarcasm detection models.

Key Contributions and Results

The authors make several contributions through this work:

1. Dataset Creation: The paper introduces MUStARD, which includes audiovisual data accompanied by contextual dialogue history. This dataset paves the way for research into understanding sarcasm as a multimodal phenomenon.
2. Multimodal Approaches: Through initial experiments leveraging the dataset, the authors demonstrate a reduction of up to 12.9% in relative error rate when using multimodal information compared to individual modalities. Specifically, combinations of textual and visual features yielded significant improvements over single modality baselines.
3. Contextual and Speaker Information: The study explores the role of conversational context and speaker characteristics in sarcasm detection, pointing out scenarios where such additional information could be beneficial, albeit with mixed results based on the experimental setup.

Insights and Analysis

The work presents substantial evidence supporting the role of multimodality in sarcasm detection. The utilization of audiovisual cues alongside text enables the model to capture the nuanced, often subtle, indicators of sarcasm that might be lost in a unimodal analysis. However, the challenges associated with disentangling multimodal cues are acknowledged, particularly when considering variables such as speaker biases and contextual dependencies.

Additionally, while the dataset provides a robust starting point, the authors note that current fusion techniques could be expanded upon to better integrate and exploit cross-modal relationships and incongruities that often characterize sarcastic expression.

Implications and Future Directions

The practical implications of this research lie in augmenting the performance of automatic sarcasm detectors deployed in sentiment analysis systems and human-computer interactions, promising richer, more nuanced understanding of sarcasm beyond mere textual cues. Theoretically, this work invites further exploration into fusion strategies and context modeling frameworks, inviting interdisciplinary approaches that span linguistics, cognitive sciences, and computer vision.

Future developments could explore enhancing context modeling through understanding speaker intentions and dependencies, localizing main speakers in multiparty dialogues, and leveraging learned speaker-specific patterns, even with the constraints of dataset size which currently limit the application of more complex neural architectures.

In conclusion, this work provides a significant step towards understanding sarcasm as a multimodal phenomenon. The developed dataset and preliminary findings lay a foundational basis for future research that should continue to explore advanced fusion and contextual modeling techniques, potentially leading to significant advancements in sarcasm detection tasks.