Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (1606.01847v3)

Abstract: Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features. We extensively evaluate MCB on the visual question answering and grounding tasks. We consistently show the benefit of MCB over ablations without MCB. For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation. This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.

Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

The paper "Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding" by Fukui et al. explores the use of multimodal compact bilinear pooling (MCB) to combine visual and textual information for tasks such as visual question answering (VQA) and visual grounding. The authors propose that current techniques, such as element-wise sum or product and concatenation, lack the expressiveness of an outer product but introduce a computationally feasible alternative through MCB, which approximates the outer product in high-dimensional space.

Core Contributions

The paper introduces MCB as a method that efficiently combines visual and textual features for multimodal tasks. Specifically, the authors:

1. Propose MCB for Efficient Multimodal Pooling:
   • MCB combines vectors from different modalities, projecting them into a higher-dimensional space using the Count Sketch algorithm and leveraging Fast Fourier Transform for efficient computation.
2. Evaluate MCB in VQA and Visual Grounding:
   • A sophisticated VQA architecture utilizing MCB improves state-of-the-art performance on the VQA dataset.
   • Integration of MCB pooling into the visual grounding task demonstrates improved localization accuracy over various baselines.

Detailed Methodology

Multimodal Compact Bilinear Pooling (MCB)

MCB addresses the infeasibility of directly computing the outer product due to its high dimensionality by compressing the bilinear feature space. It uses the Count Sketch algorithm to project individual vectors into a higher-dimensional space and then applies element-wise product in the Fourier domain to simulate the outer product:

• Count Sketch Projection: Projects vectors into a higher dimension where interactions between all elements are modeled.
• Fast Fourier Transform: Converts convolution operations in the time domain to efficient multiplications in the frequency domain.
• Bilinear Pooling: By combining Count Sketch and FFT, MCB efficiently models higher-order interactions between vectors, providing a richer feature representation suitable for tasks requiring nuanced multimodal feature integration.

VQA Architecture

The VQA architecture leveraging MCB includes several notable components:

• Image Representation: Extracted via ResNet-152.
• Question Representation: Encoded using LSTM.
• Attention Mechanism: Uses MCB to predict attention weights over spatial features and combines attended visual features with the question representation through MCB.
• Answer Prediction: Treated as a multi-class classification problem with a 3000-answer vocabulary.

The paper shows that attention mechanisms combined with MCB pooling significantly outperform non-bilinear methods and simple attention mechanisms, marking an enhancement in model performance on the VQA task.

Visual Grounding Architecture

The grounding method replaces concatenation of visual and text features with MCB, leading to better performance on the benchmark datasets Flickr30k Entities and ReferItGame. The model successfully localizes textual queries in images by leveraging the rich feature interactions facilitated by MCB pooling, demonstrating superior phrase localization accuracy.

Experimental Results

Visual Question Answering (VQA)

Extensive experiments on the VQA dataset indicate that MCB, coupled with attention, outperforms existing methods. Key results include:

• MCB with Attention: Achieves 64.2% accuracy, outperforming concatenation and other baselines significantly.
• MCB with Augmented Training Data: Integrating Visual Genome data and GloVe vectors further enhances performance, with an ensemble of models achieving 66.7% accuracy on the VQA test-dev split.

Visual Grounding

MCB integration into the visual grounding task significantly boosts accuracy:

• Flickr30k Entities: MCB achieves 48.69% accuracy, outperforming other fusion methods including element-wise product and concatenation.
• ReferItGame: Similarly, MCB outperforms baselines with a grounding accuracy of 28.91%.

Implications and Future Work

The paper highlights the effectiveness of MCB in both VQA and visual grounding, illustrating that MCB can capture complex interactions between visual and textual modalities more expressively than existing methods. Theoretically, this suggests that higher-order interactions are crucial in multimodal understanding tasks. Practically, the results indicate that MCB can be a powerful tool for enhancing performance in multimodal AI applications.

Future research may explore further optimization of MCB, its applications to other multimodal tasks, and the integration of MCB with alternative architectures and embedding techniques to advance the state-of-the-art in multimodal learning.

Conclusion

In summary, this paper presents a compelling method for multimodal pooling using MCB that significantly enhances performance in VQA and visual grounding tasks. The demonstrated improvements underscore the importance of expressive combination techniques in multimodal AI, with MCB providing a feasible and effective solution.