Data-Efficient Multimodal Fusion on a Single GPU (2312.10144v4)

Abstract: The goal of multimodal alignment is to learn a single latent space that is shared between multimodal inputs. The most powerful models in this space have been trained using massive datasets of paired inputs and large-scale computational resources, making them prohibitively expensive to train in many practical scenarios. We surmise that existing unimodal encoders pre-trained on large amounts of unimodal data should provide an effective bootstrap to create multimodal models from unimodal ones at much lower costs. We therefore propose FuseMix, a multimodal augmentation scheme that operates on the latent spaces of arbitrary pre-trained unimodal encoders. Using FuseMix for multimodal alignment, we achieve competitive performance -- and in certain cases outperform state-of-the art methods -- in both image-text and audio-text retrieval, with orders of magnitude less compute and data: for example, we outperform CLIP on the Flickr30K text-to-image retrieval task with $\sim ! 600\times$ fewer GPU days and $\sim ! 80\times$ fewer image-text pairs. Additionally, we show how our method can be applied to convert pre-trained text-to-image generative models into audio-to-image ones. Code is available at: https://github.com/layer6ai-labs/fusemix.

• The paper introduces FuseMix, a mixup-inspired method that aligns latent spaces of pretrained unimodal encoders for efficient multimodal fusion on a single GPU.
• It demonstrates competitive text-to-image retrieval, outperforming CLIP with 600 times fewer GPU days and 80 times less data.
• The approach decouples unimodal training from fusion, providing a practical and resource-efficient pathway for advancing multimodal research.

Data-Efficient Multimodal Fusion on a Single GPU

This paper introduces "FuseMix," a framework designed to achieve efficient multimodal fusion by leveraging unimodal pre-trained encoders. The research addresses the significant computational and data demands associated with traditional multimodal alignment methods, which typically require large datasets and computational resources. FuseMix offers a method that operates efficiently on a single GPU, utilizing fewer data points while maintaining or even surpassing the performance of existing state-of-the-art methods.

Methodology

The authors propose an augmentation scheme named FuseMix, inspired by mixup, which operates on the latent spaces of pre-trained unimodal encoders. This approach effectively aligns these spaces without backpropagating through the entire network, thereby making the process computationally lightweight. The architecture consists of frozen pre-trained encoders and fusion adapters, which are lightweight MLPs that align the encoded data into a shared multimodal space. By strategically employing pre-computed latent representations from these encoders, FuseMix efficiently performs fusion using minimal multimodal data.

Results and Analysis

The paper's experimental results demonstrate the effectiveness of FuseMix in both image-text and audio-text retrieval tasks. In text-to-image retrieval, it outperforms CLIP on the Flickr30K dataset using approximately 600 times fewer GPU days and 80 times less data. Such efficiency is achieved without compromising on performance, achieving competitive results even with limited data.

An ablation paper further reveals insights into dataset characteristics impacting performance: quality, diversity, and quantity. Human-annotated data, which is considered higher quality, significantly enhances retrieval outcomes, as does ensuring data diversity via determinantal point processes.

Implications and Future Directions

FuseMix illustrates a promising direction for reducing resource constraints in multimodal machine learning. By effectively decoupling the relationship between unimodal training and multimodal fusion, this research paves the way for models that can integrate advances across different modalities flexibly and economically. This decoupling makes FuseMix particularly suitable for scenarios with limited access to large computational resources or extensive multimodal datasets.

Looking forward, one potential area of development involves exploring efficient fine-tuning methods to enhance the semantic capacity of pre-trained encoders, potentially integrating techniques like LoRA or QLoRA to improve the alignment phase. Such developments could address the current limitation where performance is tied to the semantic robustness of the pre-trained encoders used. Another intriguing possibility is the application of the FuseMix methodology in contexts where APIs provide unimodal encoders, broadening the scope of available models for fusion.

The paper showcases a significant stride towards democratizing research in multimodal fusion tasks, encouraging a paradigm that balances performance with practical resource considerations.