FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion (2405.04883v2)

Abstract: Unified multi-model representation spaces are the foundation of multimodal understanding and generation. However, the billions of model parameters and catastrophic forgetting problems make it challenging to further enhance pre-trained unified spaces. In this work, we propose FreeBind, an idea that treats multimodal representation spaces as basic units, and freely augments pre-trained unified space by integrating knowledge from extra expert spaces via "space bonds". Specifically, we introduce two kinds of basic space bonds: 1) Space Displacement Bond and 2) Space Combination Bond. Based on these basic bonds, we design Complex Sequential & Parallel Bonds to effectively integrate multiple spaces simultaneously. Benefiting from the modularization concept, we further propose a coarse-to-fine customized inference strategy to flexibly adjust the enhanced unified space for different purposes. Experimentally, we bind ImageBind with extra image-text and audio-text expert spaces, resulting in three main variants: ImageBind++, InternVL_IB, and InternVL_IB++. These resulting spaces outperform ImageBind on 5 audio-image-text downstream tasks across 9 datasets. Moreover, via customized inference, it even surpasses the advanced audio-text and image-text expert spaces.

• The paper introduces FreeBind, a framework that fuses expert multimodal spaces using innovative space bonds.
• It employs space displacement and combination bonds, along with sequential and parallel bonds, to effectively integrate diverse representation spaces.
• Experimental results on five downstream tasks demonstrate that FreeBind’s flexible coarse-to-fine inference strategy consistently outperforms baseline models.

The paper "FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion," published in May 2024, addresses the challenges encountered in enhancing pre-trained unified multimodal representation spaces. These challenges include the complexity brought about by billions of model parameters and the issue of catastrophic forgetting, where models tend to forget previously learned information upon acquiring new knowledge.

Key Contributions:

1. FreeBind Framework: The paper introduces FreeBind, a framework designed to treat multimodal representation spaces as fundamental units. The innovative idea behind FreeBind is to augment pre-trained unified spaces by integrating knowledge from additional expert spaces through mechanisms called "space bonds."
2. Space Bonds:

FreeBind utilizes two primary types of space bonds: - Space Displacement Bond: This bond allows adjustment of the unified space by displacing it towards the expert space representation. - Space Combination Bond: This bond facilitates the combination of the unified space with the expert space, thereby enriching the representation.

1. Complex Sequential & Parallel Bonds: Building on the basic bonds, the paper designs more sophisticated sequential and parallel bonds to integrate multiple representation spaces effectively and simultaneously. This allows for the creation of complex, enriched multimodal spaces benefiting from the strengths of various expert spaces.
2. Coarse-to-Fine Customized Inference Strategy: In line with the modularization concept, the authors propose a flexible inference strategy that can be adjusted from coarse to fine granularity. This customization enables the fine-tuning of enhanced unified spaces for specific tasks or requirements, improving overall performance and adaptability.

Experimental Validation:

The authors validate FreeBind by binding ImageBind with additional image-text and audio-text expert spaces, resulting in three main derivatives:

• ImageBind++
• InternVL_IB
• InternVL_IB++

These enhanced spaces are tested on five audio-image-text downstream tasks across nine datasets, consistently outperforming the original ImageBind model. Notably, the customized inference strategy enables these enhanced spaces to even surpass the performance of advanced audio-text and image-text expert spaces.

The innovative approach of FreeBind thus provides a practical and effective solution for augmenting multimodal representation spaces. By enabling the integration of knowledge from various expert spaces, FreeBind significantly enhances the capabilities and performance of pre-trained unified multimodal spaces.