IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities (2408.12902v2)

Abstract: In the field of multimodal LLMs (MLLMs), common methods typically involve unfreezing the LLM during training to foster profound visual understanding. However, the fine-tuning of such models with vision-language data often leads to a diminution of their NLP capabilities. To avoid this performance degradation, a straightforward solution is to freeze the LLM while developing multimodal competencies. Unfortunately, previous works have not attained satisfactory outcomes. Building on the strategy of freezing the LLM, we conduct thorough structural exploration and introduce the Inner-Adaptor Architecture (IAA). Specifically, the architecture incorporates multiple multimodal adaptors at varying depths within the LLM to facilitate direct interaction with the inherently text-oriented transformer layers, thereby enabling the frozen LLM to acquire multimodal capabilities. Unlike previous approaches of freezing LLMs that require large-scale aligned data, our proposed architecture is able to achieve superior performance on small-scale datasets. We conduct extensive experiments to improve the general multimodal capabilities and visual grounding abilities of the MLLM. Our approach remarkably outperforms previous state-of-the-art methods across various vision-language benchmarks without sacrificing performance on NLP tasks. Code and models are available at https://github.com/360CVGroup/Inner-Adaptor-Architecture.