Efficient Multimodal LLMs: A Comprehensive Survey
Prepping the Ground for Efficient MLLMs
Multimodal LLMs (MLLMs) have shown impressive abilities in tasks like visual question answering and visual understanding. Yet, their huge model sizes and the demanding cost of training and using them have limited their wider application. This survey paper provides an in-depth review of efficient MLLMs, particularly in light of their potential use in edge computing scenarios. The focus is on lightweight models that maintain strong performance while using fewer resources.
Architecture: Breaking It Down
Core Components
Efficient MLLMs follow the basic framework of conventional MLLMs but are designed with an eye on reducing computational costs. The architecture can generally be divided into three main parts:
- Vision Encoder: Processes visual inputs.
- LLM: Handles multimodal signals and reasoning.
- Vision-Language Projector: Bridges the two modalities.
Key Approaches
Multiple Vision Encoders: Combining different vision encoders offers a diverse range of visual representations, enhancing the model's understanding of visual data. Models like Cobra integrate DINOv2 and SigLIP for better performance.
Lightweight Vision Encoder: Approaches like ViTamin focus on creating smaller vision models without sacrificing accuracy. This makes them suitable for tasks with high-resolution requirements.
Vision-Language Projector: Most use simple MLPs, while others like BLIP2 introduce transformers like Q-Former to capture richer visual features using latent queries.
Small LLM: Efficient MLLMs typically use LLMs with less than 3 billion parameters to save resources while maintaining strong performance. Models like Phi-2 and Gemma-2B serve as excellent examples.
Vision Token Compression: Techniques like token processing and multi-scale information fusion reduce the computational load imposed by high-resolution visual inputs. Methods such as the compression module in LLaVA-UHD help maintain a balance between detailed perception and efficiency.
Efficient Structures: MoE-based models like MoE-LLaVA enhance computational efficiency and performance by using sparseness effectively. Meanwhile, models like VTW improve inference by strategically dropping tokens.
Training: From Scratch to Fine-Tuned
Pre-Training
Pre-training usually involves large datasets of image-caption pairs to build initial multimodal representations. Efficient strategies include assembly of multi-stage pre-training with different image resolutions for optimal performance.
Instruction-Tuning
Instruction-tuning fine-tunes the models using task-specific datasets, including curated conversations and instructions. Approaches like LaVIN manage to reduce training cost significantly while retaining high performance across tasks.
Diverse Training Steps
Some efficient models, like TinyGPT-V, employ multi-stage training processes to iteratively refine their capabilities from basic understanding to advanced multi-task learning.
Parameter-Efficient Transfer Learning
Methods like MemVP propose using visual prompts to inject new visual knowledge into the model, thus reducing the computational burden significantly during both training and inference.
Data and Benchmarks: The Backbone of Performance
Pre-Training Data
Datasets like CC595k and LAION provide extensive image-text pairs necessary for building robust initial models. However, high-quality, fine-grained datasets created with the help of models like GPT-4V offer better performance but come at a higher cost.
Instruction-Tuning Data
Instruction-tuning datasets are derived from a mix of task-specific and general-purpose data, aimed at refining the models' responsiveness to various instructions.
Benchmarks
Performance is evaluated using established benchmarks like VQA and GQA, where efficient models often show competitive results against larger counter-parts. This highlights the success of efficient architectures in maintaining high-quality outputs.
Applications: Spanning Domains
Biomedical Analysis
Efficient MLLMs like MoE-TinyMed have found applications in medical scenarios, providing strong performance with fewer parameters. Models such as LLaVA-Rad outperform larger models in generating radiology reports.
Document Understanding
Efficient models such as TinyChart integrate strategies for fine-grained perception to enhance document understanding, paving the way for applications that require detailed visual and textual analysis.
Video Comprehension
Methods like Video-LLaVA excel by unifying visual representation into a language space, enabling efficient processing of multiple frames in video understanding tasks.
Final Thoughts and Future Directions
Efficient MLLMs are making strides in various fields by balancing performance and resource consumption. However, there's always room for improvement. Broadening input and output modalities, enhancing zero-shot capabilities, and developing embodied agents are some promising future directions that could further establish efficient MLLMs as versatile tools in AI.
In summary, the paper comprehensively surveys the landscape of efficient MLLMs, pointing to robust strategies and promising avenues that hold the potential to bring advanced AI capabilities into practical, resource-constrained environments.