BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices (2411.10640v1)

Abstract: The emergence and growing popularity of multimodal LLMs (MLLMs) have significant potential to enhance various aspects of daily life, from improving communication to facilitating learning and problem-solving. Mobile phones, as essential daily companions, represent the most effective and accessible deployment platform for MLLMs, enabling seamless integration into everyday tasks. However, deploying MLLMs on mobile phones presents challenges due to limitations in memory size and computational capability, making it difficult to achieve smooth and real-time processing without extensive optimization. In this paper, we present BlueLM-V-3B, an algorithm and system co-design approach specifically tailored for the efficient deployment of MLLMs on mobile platforms. To be specific, we redesign the dynamic resolution scheme adopted by mainstream MLLMs and implement system optimization for hardware-aware deployment to optimize model inference on mobile phones. BlueLM-V-3B boasts the following key highlights: (1) Small Size: BlueLM-V-3B features a LLM with 2.7B parameters and a vision encoder with 400M parameters. (2) Fast Speed: BlueLM-V-3B achieves a generation speed of 24.4 token/s on the MediaTek Dimensity 9300 processor with 4-bit LLM weight quantization. (3) Strong Performance: BlueLM-V-3B has attained the highest average score of 66.1 on the OpenCompass benchmark among models with $\leq$ 4B parameters and surpassed a series of models with much larger parameter sizes (e.g., MiniCPM-V-2.6, InternVL2-8B).

• The paper demonstrates a novel co-design strategy that optimizes both algorithm and system architecture for mobile multimodal language models.
• It employs a dynamic resolution scheme and token downsampling with mixed-precision quantization to balance speed and accuracy under mobile constraints.
• The study reports substantial improvements with 24.4 tokens per second generation speed and competitive benchmark results, setting a new standard for mobile MLLMs.

BlueLM-V-3B: Streamlining Algorithms for Multimodal LLMs on Mobile Devices

In the era of digital ubiquity, the integration of multimodal LLMs (MLLMs) into mobile environments presents unique challenges and opportunities. The paper entitled "BlueLM-V-3B: Algorithm and System Co-Design for Multimodal LLMs on Mobile Devices" presents a structured approach to efficiently deploying MLLMs on mobile platforms, addressing prevalent constraints such as computational capability and memory limitations. This essay aims to dissect and analyze the comprehensive methodology outlined by the authors, focusing on their algorithmic and system design advancements for optimizing MLLMs on mobile devices.

Core Highlights

BlueLM-V-3B is proposed as a condensed, yet high-performance, MLLM tailored for mobile usage, featuring a LLM with 2.7 billion parameters and a vision encoder with an additional 400 million parameters. Noteworthy aspects of this model include exceptional speed and competitive performance, achieving a generation speed of 24.4 tokens per second with 4-bit weight quantization and superior benchmark results (66.1 on the OpenCompass benchmark).

Algorithmic & System Innovations

The authors apply a holistic co-design strategy, revisiting traditional architectures prevalent among mainstream MLLMs to align with mobile specifications:

1. Dynamic Resolution Scheme: The authors introduce a relaxed aspect ratio matching technique, optimizing the model's interaction with high-resolution images without unnecessary computational overhead by reducing image token numbers. This adjustment permits efficient model training and inference, pivotal for resource-constrained deployment.
2. System Architecture Adjustments: To leverage hardware acceleration, innovations such as batched image encoding and pipeline parallelism are integrated, facilitating concurrent processing of image patches. This practice efficiently utilizes available hardware, optimizing runtime inference speeds.
3. Token Downsampling and Mixed Precision: By employing token downsampling, the authors manage excessive image token lengths, crucial for inference on devices with limited processing rates. They also incorporate mixed-precision quantization techniques, balancing memory usage against model accuracy, with weight operations in INT4 and activations in higher precision formats.

Model Evaluation and Results

Upon evaluation, BlueLM-V-3B upholds superior efficacy in terms of model accuracy and efficiency. The deployment on mobile devices demonstrates a token throughput speed almost five times faster than compared models, like the MiniCPM-V. The balance between parameter economy and performance output stands as a testament to the thorough system-analysis process undertaken by the authors.

Implications and Future Directions

The implications of this research extend into both practical and theoretical realms. Practically, mobile devices accessing sophisticated MLLMs open avenues for enhanced real-time applications such as language translation and augmented communication tools. Theoretically, BlueLM-V-3B sets a precedent for future model designs embodying co-design principles, emphasizing symbiosis between software prowess and hardware limitations.

Future developments may explore greater efficiency through adaptive learning techniques or further compression strategies in parameter and computation handling. Additionally, the expansion of dataset diversity utilized for training could enhance model robustness across more contexts and languages, unlocking greater multi-functional capacity for global user bases.

In summary, BlueLM-V-3B manifests a strategic confluence of algorithmic refinement and system design, demonstrating tangible advancements in deploying high-efficiency, adaptable multimodal models on mobile platforms. This convergence of innovation caters to the rising demand for intelligent systems integrated seamlessly into the ever-expanding landscape of mobile technology.