VoCo-LLaMA: Towards Vision Compression with Large Language Models (2406.12275v1)

Abstract: Vision-LLMs (VLMs) have achieved remarkable success in various multi-modal tasks, but they are often bottlenecked by the limited context window and high computational cost of processing high-resolution image inputs and videos. Vision compression can alleviate this problem by reducing the vision token count. Previous approaches compress vision tokens with external modules and force LLMs to understand the compressed ones, leading to visual information loss. However, the LLMs' understanding paradigm of vision tokens is not fully utilised in the compression learning process. We propose VoCo-LLaMA, the first approach to compress vision tokens using LLMs. By introducing Vision Compression tokens during the vision instruction tuning phase and leveraging attention distillation, our method distill how LLMs comprehend vision tokens into their processing of VoCo tokens. VoCo-LLaMA facilitates effective vision compression and improves the computational efficiency during the inference stage. Specifically, our method achieves minimal performance loss with a compression ratio of 576$\times$, resulting in up to 94.8$\%$ fewer FLOPs and 69.6$\%$ acceleration in inference time. Furthermore, through continuous training using time-series compressed token sequences of video frames, VoCo-LLaMA demonstrates the ability to understand temporal correlations, outperforming previous methods on popular video question-answering benchmarks. Our approach presents a promising way to unlock the full potential of VLMs' contextual window, enabling more scalable multi-modal applications. The project page, along with the associated code, can be accessed via $\href{https://yxxxb.github.io/VoCo-LLaMA-page/}{\text{this https URL}}$.

VoCo-LLaMA: Towards Vision Compression with LLMs

The paper "VoCo-LLaMA: Towards Vision Compression with LLMs" introduces an innovative method for compressing vision tokens within the framework of Vision-LLMs (VLMs). This work addresses a significant bottleneck in VLMs caused by the limited context window and high computational costs associated with processing high-resolution image inputs and videos. It puts forth VoCo-LLaMA as a pioneering approach to efficiently compress vision tokens using LLMs, leveraging their inherent ability to distill visual information.

Summary

Motivation and Background

The efficacy of VLMs in multimodal tasks has been well-documented, especially with enhancements in image understanding through high-resolution image encoding and incorporating more video frames. However, the large number of vision tokens generated from such high-resolution inputs heavily occupies the context window of LLMs, increasing computational costs substantially. Previous solutions involved compressing vision tokens with external modules, which often led to significant visual information loss. VoCo-LLaMA proposes a novel internal approach to vision compression by utilizing LLMs themselves to compress and understand vision tokens.

Methodology

VoCo-LLaMA introduces Vision Compression (VoCo) tokens during the vision instruction tuning phase and leverages attention distillation to encode visual information into a compact format. The approach comprises:

1. Vision Compression:
   • Vision tokens ($\mathcal{V}$) obtained from image encoders are converted into a fewer number of compression tokens ($\mathcal{C}$), termed VoCo tokens.
   • A two-stage attention mechanism is employed where text tokens attend only to VoCo tokens and not directly to vision tokens, ensuring information compression without disrupting the LLM's understanding of visual data.
2. Attention Mask Adjustment:
   • The attention mask is modified so that text tokens interact exclusively with VoCo tokens, facilitating effective distillation and compression of visual information.
3. Temporal Modeling:
   • For video inputs, the compressed tokens representing individual frames are processed sequentially, capturing temporal correlations.

Results

Experimental evaluations demonstrate that VoCo-LLaMA achieves significant vision compression while retaining high performance in visual understanding tasks. Key results include:

• Compression Performance:
  • VoCo-LLaMA achieves an average compression retention rate of 83.7% across several benchmarks, utilizing a single VoCo token to represent 576 vision tokens.
  • Compared to previous methods like Q-Former and average pooling, VoCo-LLaMA exhibits superior retention of visual information while significantly reducing computational costs.
• Inference Efficiency:
  • VoCo-LLaMA reduces CUDA time by up to 69.6% and FLOPs by 94.8%, with a 99.8% reduction in cache storage compared to traditional full-caching strategies.
• Video Understanding:
  • VoCo-LLaMA outperforms state-of-the-art methods on video question-answering benchmarks like MSVD-QA, MSRVTT-QA, and ActivityNet-QA, showcasing robust performance even with compressed visual inputs.

Implications and Future Directions

Practical Implications

The practical implications of VoCo-LLaMA are profound. By efficiently compressing vision tokens, this method significantly enhances the scalability of VLMs for processing high-resolution images and videos in limited context windows. The reduction in computational overhead and storage requirements facilitates real-time deployment and wider applicability in resource-constrained environments.

Theoretical Implications

Theoretically, VoCo-LLaMA introduces a new paradigm in vision-LLMing by demonstrating that LLMs can effectively compress and retain visual information without relying on external modules. This underscores the potential for deeper integration and optimization within multimodal systems, paving the way for future innovations in cross-modal learning and understanding.

Future Developments

Future developments can expand on this work by exploring:

1. Adaptive Compression Mechanisms: Adjusting the number of VoCo tokens dynamically based on input complexity to balance compression efficiency and performance.
2. Cross-Task Generalization: Applying VoCo-LLaMA to other vision-language tasks beyond comprehension and answering, such as image generation and editing.
3. Interoperability with Other Models: Integrating VoCo-LLaMA with other advanced LLMs and visual encoders to enhance its robustness and generalizability.

Conclusion

VoCo-LLaMA represents a significant step forward in the efficient processing of visual information using LLMs. By leveraging the inherent capabilities of LLMs to distill and compress vision tokens, VoCo-LLaMA offers a scalable and computationally efficient solution that maintains high performance across various vision-language tasks. This work provides a promising foundation for future research in enhancing the efficiency and scalability of multimodal AI applications.