Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration (2501.05179v1)

Abstract: Multimodal LLMs (MLLMs) have attracted considerable attention due to their exceptional performance in visual content understanding and reasoning. However, their inference efficiency has been a notable concern, as the increasing length of multimodal contexts leads to quadratic complexity. Token compression techniques, which reduce the number of visual tokens, have demonstrated their effectiveness in reducing computational costs. Yet, these approaches have struggled to keep pace with the rapid advancements in MLLMs, especially the AnyRes strategy in the context of high-resolution image understanding. In this paper, we propose a novel token compression method, GlobalCom$^2$, tailored for high-resolution MLLMs that receive both the thumbnail and multiple crops. GlobalCom$^2$ treats the tokens derived from the thumbnail as the ``commander'' of the entire token compression process, directing the allocation of retention ratios and the specific compression for each crop. In this way, redundant tokens are eliminated while important local details are adaptively preserved to the highest extent feasible. Empirical results across 10 benchmarks reveal that GlobalCom$^2$ achieves an optimal balance between performance and efficiency, and consistently outperforms state-of-the-art token compression methods with LLaVA-NeXT-7B/13B models. Our code is released at \url{https://github.com/xuyang-liu16/GlobalCom2}.

Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration

The paper "Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration" presents a novel approach, termed GlobalCom$^2$, designed to address the computational inefficiencies associated with Multimodal LLMs (MLLMs) when processing high-resolution visual data. The increasing demand for models like LLaVA-NeXT that can handle complex vision-language tasks has highlighted the quadratic complexity problem caused by the extended sequence length of visual token inputs. This research provides a significant contribution to the field of AI by proposing a method that accelerates the inference of MLLMs without the need for retraining or significant architectural changes.

The core contribution of this work is the development of GlobalCom$^2$, a training-free token compression technique. This approach is specifically designed to optimize the processing of high-resolution images in MLLMs by effectively compressing visual tokens. GlobalCom$^2$ achieves this by employing a "global-to-local" guidance strategy, where global thumbnail information is utilized to direct the compression process for individual image crops. This ensures that semantically redundant tokens are pruned while critical local details are conserved, enhancing the model's efficiency and performance.

The methodology is divided into two key stages. Firstly, the thumbnail image acts as a global commander, assessing the importance of each crop and determining the retention ratios for token preservation. This allocation is computed using the scaled attention values between tokens and the [CLS] token from the visual encoder's attention maps, ensuring that more informative segments maintain a higher proportion of tokens. Secondly, the method evaluates token importance within each crop by considering both local relevance and global significance, preserving those critical for accurate multimodal understanding.

Experimentally, the authors validate the efficacy of GlobalCom$^2$ across ten benchmarks, demonstrating its superior performance in maintaining accuracy while significantly reducing token quantity. Notably, with a retention of only 10% of the original visual tokens, GlobalCom$^2$ achieves over 90% of the original accuracy of the LLaVA-NeXT models, outperforming state-of-the-art training-free token compression methods. These results suggest that the proposed approach is highly effective in not only retaining essential visual information but also substantially speeding up the inference process.

From a practical standpoint, this work has substantial implications for deploying MLLMs in resource-constrained environments, where computational efficiency and memory utilization are critical. Theoretically, the approach underscores the importance of considering both global and local visual contexts in model architecture, which could influence future model designs and token handling strategies in multimodal machine learning.

Future research could explore the extension of GlobalCom$^2$ into more diverse modalities and tasks, potentially broadening its applicability in real-world scenarios. Furthermore, integrating this method with other efficiency-boosting techniques like quantization or knowledge distillation could yield even more substantial improvements in performance and resource utilization. As the demand for real-time, multimodal processing grows, innovations like GlobalCom$^2$ will undoubtedly play a crucial role in shaping the future landscape of AI technologies.