Supervised Multimodal Bitransformers for Classifying Images and Text

Abstract: Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised multimodal bitransformer model that fuses information from text and image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks, outperforming strong baselines, including on hard test sets specifically designed to measure multimodal performance.

• The paper introduces a multimodal bitransformer that projects image embeddings into the text token space, enabling effective fusion without full retraining.
• Experimental evaluations show the model surpasses state-of-the-art benchmarks, achieving 61.6 macro-F1 on MM-IMDB and 92.1% accuracy on Food101.
• The study highlights how integrating unimodal pretrained encoders allows for robust performance, even when one modality is missing, setting a solid baseline for future research.

Supervised Multimodal Bitransformers for Classifying Images and Text

The advent of transformer models, particularly BERT, has heralded significant advancements in various textual classification domains. However, the increasing prevalence of multimodal data in digital content necessitates solutions that accommodate more than just textual input, such as images alongside text. The paper under consideration introduces a straightforward yet effective solution to this challenge: a supervised multimodal bitransformer model that aligns image embeddings to the token space of text embeddings using transformers pretrained unimodally on text and images.

Proposed Architecture

The authors propose a model that combines separate unimodally pretrained text and image encoders. The image embeddings are projected into the text token embedding space, thereby enabling the Multimodal Bitransformer (MMBT) to perform multimodal fusion. This architecture retains the unimodally pretrained weights and does not necessitate retraining from scratch, which aligns with the premise that developing improved encoders for each modality can enhance overall performance without redesigning the entire system.

Experimental Evaluation

The authors evaluated their model on text-heavy multimodal classification tasks, namely MM-IMDB, Food101, and V-SNLI. The performance of the proposed model was consistent with, and sometimes surpassed, state-of-the-art multimodal models that utilize extensive multimodal pretraining, like ViLBERT. The key findings include:

1. MM-IMDB: On the multilabel classification task for movie genre prediction, the MMBT achieved macro-F1 and micro-F1 scores of 61.6 and 66.8, respectively, surpassing other unimodal baselines and fusion techniques by a notable margin.
2. Food101: The multiclass task of food category classification yielded a 92.1% accuracy, indicating effective multimodal fusion capability for this application.
3. V-SNLI: In the task involving entailment classification in textual and visual premise and hypothesis pairs, the MMBT achieved a 90.4% accuracy, demonstrating that it can effectively generalize beyond bimodal datasets.

Furthermore, the authors constructed hard test sets by selecting examples where unimodal models differed significantly from the ground truth. The MMBT consistently demonstrated superior performance on these challenging subsets, underscoring its ability to excel in multimodal reasoning scenarios.

Implications and Future Work

The supervised MMBT presents several significant practical implications:

• Integration of Improved Models: As the model relies on separate pretrained components, it allows for easy replacement with newer text or image models to further improve performance without requiring modification of the fusion technique.
• Robustness to Missing Modalities: Experiments reveal that the MMBT maintains resilience even when one of the input modalities is missing during training.
• Baseline for Future Multimodal Research: The effectiveness of unimodal pretraining combined with a simple projection mechanism provides a strong baseline that challenges the necessity of complex multimodal pretraining strategies.

This work proposes that while self-supervised multimodal models like ViLBERT are gaining traction, simpler approaches using unimodally pretrained components can still achieve competitive performance. Future developments may explore techniques that combine these advantages, perhaps introducing a layer of multimodal pretraining atop unimodal pretrained systems to enhance multimodal tasks further. Additionally, exploration into more complex fusion architectures could yield further gains in scenarios with highly imbalanced multimodal importance. This research underscores an exciting direction for achieving simplicity and effectiveness in multimodal machine learning models.