Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features

Abstract: Hateful memes are a growing menace on social media. While the image and its corresponding text in a meme are related, they do not necessarily convey the same meaning when viewed individually. Hence, detecting hateful memes requires careful consideration of both visual and textual information. Multimodal pre-training can be beneficial for this task because it effectively captures the relationship between the image and the text by representing them in a similar feature space. Furthermore, it is essential to model the interactions between the image and text features through intermediate fusion. Most existing methods either employ multimodal pre-training or intermediate fusion, but not both. In this work, we propose the Hate-CLIPper architecture, which explicitly models the cross-modal interactions between the image and text representations obtained using Contrastive Language-Image Pre-training (CLIP) encoders via a feature interaction matrix (FIM). A simple classifier based on the FIM representation is able to achieve state-of-the-art performance on the Hateful Memes Challenge (HMC) dataset with an AUROC of 85.8, which even surpasses the human performance of 82.65. Experiments on other meme datasets such as Propaganda Memes and TamilMemes also demonstrate the generalizability of the proposed approach. Finally, we analyze the interpretability of the FIM representation and show that cross-modal interactions can indeed facilitate the learning of meaningful concepts. The code for this work is available at https://github.com/gokulkarthik/hateclipper.

• The paper introduces Hate-CLIPper, a novel architecture that uses a cross-modal Feature Interaction Matrix to merge CLIP-generated text and image features for precise hateful meme detection.
• Hate-CLIPper’s intermediate fusion approach captures fine-grained attribute correlations, achieving an AUROC of 85.8 on the challenging Hateful Memes Challenge dataset.
• The design enhances interpretability by analyzing the FIM’s salient dimensions, paving the way for greater transparency in multimodal AI applications.

Hate-CLIPper: A Multimodal Approach to Hateful Meme Classification

The proliferation of hateful memes on social media platforms poses a significant challenge, amplified by the inherently multimodal nature of such content, which traditionally combines images and text. The complexity arises from the fact that the textual and visual modalities in memes can independently appear innocuous but may convey harmful messages when combined. This paper presents Hate-CLIPper, a novel architecture designed to address this challenge by leveraging the capabilities of multimodal pre-training and cross-modal interaction modeling.

Key Contributions

Hate-CLIPper introduces an integrated approach that employs the Contrastive Language-Image Pre-training (CLIP) model, explicitly focusing on cross-modal interactions. The architecture advances beyond existing methodologies by combining multimodal pre-training with intermediate fusion through a Feature Interaction Matrix (FIM). This novel representation captures fine-grained attribute correlations between image and text features, which is critical for accurately detecting hateful intent.

Experimental Evaluation

Significant performance improvements underscore Hate-CLIPper's efficacy, achieving an AUROC of 85.8 on the Hateful Memes Challenge (HMC) dataset, surpassing human performance benchmarks. The paper meticulously evaluates Hate-CLIPper across multiple meme datasets, including Propaganda Memes and TamilMemes, to demonstrate its generalizability. The alignment of CLIP-generated image and text representations ensures that even a simple classifier can deliver state-of-the-art results without relying on additional features like bounding boxes or facial detection.

Methodological Insights

The architecture capitalizes on the rich, pre-aligned feature spaces generated by CLIP, employing intermediate fusion that avoids the pitfalls of early or late fusion techniques. Traditional early fusion assumes a descriptive relationship between text and images, unsuitable for memes where text may not directly describe the image. Conversely, late fusion models lack the nuance to integrate multimodal interactions comprehensively. Hate-CLIPper's use of a bilinear pooling strategy to generate the FIM represents a significant departure here, fostering more efficient and accurate multimodal feature integration.

Interpretability and Implications

A noteworthy aspect of the paper is its exploration of the interpretability of the FIM matrix, an area often sidelined in deep learning research. By identifying salient dimensions contributing to classification decisions, the work highlights potential pathways for enhancing model transparency. It becomes evident that the FIM facilitates the learning of coherent conceptual mappings across modalities, which could inform future efforts in explainable AI.

Future Directions

Although Hate-CLIPper represents a robust solution to the hateful meme classification challenge, future research could explore optimizing its computational efficiency, particularly the high-dimensional nature of the FIM model. Furthermore, extending this architecture to effectively handle low-resource languages, where multimodal training data is scarce, remains an open challenge. Additionally, refining methodologies to provide granular interpretability consistent with potential ethical applications could greatly enhance the model's deployment potential in real-world scenarios.

In conclusion, Hate-CLIPper establishes a promising foundation for multimodal hateful content detection by expertly balancing pre-trained capabilities with innovative fusion techniques. As hateful online content continues to evolve, sophisticated methods like Hate-CLIPper are increasingly vital in mitigating its spread and fostering safer digital environments.