Matryoshka Multimodal Models (2405.17430v2)

Abstract: Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in visual-linguistic reasoning. These models first embed images into a fixed large number of visual tokens and then feed them into a LLM. However, this design causes an excessive number of tokens for dense visual scenarios such as high-resolution images and videos, leading to great inefficiency. While token pruning/merging methods do exist, they produce a single length output for each image and do not afford flexibility in trading off information density v.s. efficiency. Inspired by the concept of Matryoshka Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens that capture information across multiple coarse-to-fine granularities. Our approach offers several unique benefits for LMMs: (1) One can explicitly control the visual granularity per test instance during inference, e.g. , adjusting the number of tokens used to represent an image based on the anticipated complexity or simplicity of the content; (2) M3 provides a framework for analyzing the granularity needed for existing datasets, where we find that COCO-style benchmarks only need around ~9 visual tokens to obtain accuracy similar to that of using all 576 tokens; (3) Our approach provides a foundation to explore the best trade-off between performance and visual token length at sample level, where our investigation reveals that a large gap exists between the oracle upper bound and current fixed-scale representations.

Matryoshka Multimodal Models: Enhancing Efficiency and Flexibility in Visual Token Representation

Large Multimodal Models (LMMs) have demonstrated significant progress in visual-linguistic reasoning tasks. Traditional models like LLaVA embed input images into a fixed number of visual tokens for subsequent processing by a LLM. However, this approach leads to inefficiencies, especially in dense visual contexts such as high-resolution images and long videos. The proposed solution by the authors is the introduction of the Matryoshka Multimodal Models (M$^3$), designed to represent visual content with nested sets of visual tokens, capturing information from coarse-to-fine granularities. This summary explores the methodology, implications, and performance of M$^3$, providing insights into its contributions and potential future developments.

Methodology

The central innovation of M$^3$ is the representation of visual content as multiple nested sets of visual tokens, enabling explicit control over the visual granularity at inference time. This methodology draws inspiration from Matryoshka dolls, where larger structures encompass smaller, detailed components. Specifically, M$^3$ modifies the visual token generation process by pooling tokens in a hierarchical manner, thereby producing token sets of varying granularity that can be selectively used based on the complexity of the visual input.

The training objective is straightforward yet powerful: it involves maximizing the likelihood of the predicted tokens matching the ground-truth answers, averaged over all scales of visual tokens. This approach involves no additional learnable parameters beyond those in the visual encoder and LLM. Rather, it optimizes the existing architecture to accommodate and leverage the hierarchical token representations.

Experimental Evaluation

The performance of M$^3$ was evaluated on several benchmarks focusing on both image and video understanding tasks. Notably, the results demonstrated that M$^3$ achieved comparable or superior performance to existing models while offering significant efficiency gains. For instance, in the MMBench evaluation, M$^3$ with 9 tokens per image performed on par with models using far more tokens, such as Qwen-VL-Chat with 256 tokens.

In video understanding tasks, M$^3$ showcased its ability to maintain performance while reducing the number of tokens. Interestingly, certain video tasks benefited from the compact representation offered by M$^3$, where models using fewer tokens outperformed those using the full token set.

Implications and Future Directions

The implications of M$^3$ span both practical and theoretical dimensions. Practically, the ability to adjust the granularity of visual tokens dynamically allows for more efficient deployment of LMMs, particularly in resource-constrained environments. This is particularly valuable for applications involving high-resolution images or long videos, where traditional models struggle with inefficiency.

Theoretically, M$^3$ highlights the potential of hierarchical representations in enhancing model performance and efficiency. It provides a foundation for further exploration into adaptive token length strategies and the underlying biases in visual benchmarks. The significant performance gap between models using full tokens and the oracle upper bound suggests that there is considerable room for optimization, potentially through the development of sophisticated token length predictors.

Conclusions

The introduction of M$^3$ marks a significant step forward in the efficient representation of visual content within LMMs. The model's ability to dynamically adjust visual granularity during inference offers both improved performance and efficiency. The results demonstrated across various benchmarks affirm the robustness and flexibility of M$^3$. Future research can build on these findings to develop models that optimize token usage further and extend the principles of hierarchical representation to other domains, such as text and dense vision tasks.