MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model (2406.11193v2)

Abstract: Projecting visual features into word embedding space has become a significant fusion strategy adopted by Multimodal LLMs (MLLMs). However, its internal mechanisms have yet to be explored. Inspired by multilingual research, we identify domain-specific neurons in multimodal LLMs. Specifically, we investigate the distribution of domain-specific neurons and the mechanism of how MLLMs process features from diverse domains. Furthermore, we propose a three-stage mechanism for LLM modules in MLLMs when handling projected image features, and verify this hypothesis using logit lens. Extensive experiments indicate that while current MLLMs exhibit Visual Question Answering (VQA) capability, they may not fully utilize domain-specific information. Manipulating domain-specific neurons properly will result in a 10% change of accuracy at most, shedding light on the development of cross-domain, all-encompassing MLLMs in the future. The source code is available at https://github.com/Z1zs/MMNeuron.

"MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal LLM" presents significant advancements in understanding how Multimodal LLMs (MLLMs) process and interpret features from various domains, particularly by identifying and manipulating domain-specific neurons.

The core of the paper focuses on the internal mechanisms by which MLLMs handle visual features projected into the word embedding space, a prevalent fusion strategy in current research. Inspired by techniques used in multilingual models, the authors set out to uncover and characterize domain-specific neurons within these multimodal systems. Their research aims to understand how these neurons are distributed and operate when MLLMs process multimodal data.

Key Contributions and Findings:

1. Identification of Domain-Specific Neurons:
   • The paper identifies neurons within MLLMs that distinctly respond to specific domains (e.g., visual vs. textual data). This is analogous to how certain neurons in multilingual models respond to specific languages.
2. Three-Stage Framework Hypothesis:
   • The authors propose a three-stage framework to describe how LLM modules within MLLMs handle projected image features:
     1. Feature Extraction: Extracting salient features from visual inputs.
     2. Feature Projection: Projecting these features into the word embedding space.
     3. Feature Integration: Integrating and interpreting these projected features alongside textual data.
   • This hypothesis is tested and verified using the logit lens technique, which provides insight into intermediate representations within neural networks.
3. Experimental Validation:
   • Extensive experiments indicate that while current MLLMs demonstrate capabilities in tasks like Visual Question Answering (VQA), they may not be fully leveraging domain-specific information. Specifically, the manipulation of domain-specific neurons can result in up to a 10% change in accuracy, suggesting significant yet underutilized potential in these models.
4. Implications for Future MLLM Development:
   • The findings highlight the importance of domain-specific neuron identification and manipulation in enhancing the performance of MLLMs. Understanding and harnessing these neurons can lead to the creation of more versatile multimodal models capable of seamlessly integrating information across different domains.

These insights provide a pathway for future research and development in the field of MLLMs, particularly in creating models that can more effectively and efficiently process and integrate multimodal data. The release of the code upon paper notification further promises to enable the replication and extension of this work by other researchers in the field.