LightVLM: Fast VLM Inference Accelerator

• The paper introduces LightVLM, a training-free acceleration framework that compresses both visual tokens and KV caches to boost multimodal inference speed.
• It employs pyramid token merging during encoding and attention-based key-value cache compression during decoding to reduce redundancy and improve throughput.
• Empirical results show that LightVLM retains near-full accuracy (97.8-100%) while significantly reducing latency and memory usage across various VLM architectures.

LightVLM is a training-free inference acceleration framework for large vision-LLMs (VLMs) that is designed to plug into existing multimodal systems and accelerate both major phases of inference: encoding and decoding. Rather than introducing a new VLM architecture from scratch, it modifies the forward pass of existing models by compressing visual tokens during encoding through pyramid token merging and compressing key-value caches during decoding through attention-based KV cache compression. In the reported experiments, LightVLM retains 100% performance when only preserving 35% image tokens, maintains around 98% performance when keeping only 3% image tokens, improves network throughput by 2.02$\times$, reduces prefilling time by 3.65$\times$, and reduces long-sequence inference time by 3.21$\times$ for 4096-token generation (Hu et al., 30 Aug 2025).

1. Definition and problem setting

LightVLM addresses a central systems bottleneck in contemporary VLMs: the dominance of visual tokens in the multimodal context and the consequent cost of long-context decoding. In the formulation used by the method, a typical VLM first encodes an image or video into a large number of visual tokens through a vision encoder and projector, concatenates these with prompts and user text, and then feeds the resulting multimodal sequence into a LLM. The paper states that a single image is often mapped to 1,000–10,000+ tokens, with Qwen2.5-VL supporting up to 16,384 visual tokens, while textual input usually contributes only tens to hundreds of tokens. As a result, visual tokens can account for 95–99% of the input sequence length, and the quadratic attention complexity makes inference costly (Hu et al., 30 Aug 2025).

The method is explicitly positioned as a plug-and-play efficiency layer rather than a retraining procedure. It does not change model weights or require additional optimization of the base VLM. Instead, it exploits two empirical observations reported in the paper. First, visual tokens are highly redundant: in Qwen2.5-VL 7B, about 11% of image tokens account for 95% of attention in layer 1, while in layer 27 only about 3% of image tokens account for 90% of attention and fewer than 10% account for 95%. Second, many visual tokens in the KV cache receive negligible attention during long decoding, so preserving all of them wastes both memory and compute (Hu et al., 30 Aug 2025).

A common misconception is to treat LightVLM as a generic “small VLM.” The paper instead defines it as a training-free acceleration framework for large multimodal models. Its target is not primarily parameter count reduction, but reduction of effective sequence length and cache size during inference.

2. Inference-stage decomposition and integration strategy

LightVLM divides multimodal inference into two stages. During the encoding stage, the model processes the system prompt, image or video tokens, and user text, producing contextual representations and populating KV caches for the input tokens. During the decoding stage, the model autoregressively predicts output tokens, with each new token attending to all encoded tokens and all previously generated tokens. LightVLM attaches a different compression mechanism to each stage (Hu et al., 30 Aug 2025).

The framework assumes a standard vision encoder–projector–LLM architecture and is described as compatible with Qwen2.5-VL, InternVL2.5, LLaVA-OneVision, and MiniCPM-V. It hooks into the LLM forward pass rather than the training loop. During encoding, at selected transformer layers $K=\{k_1,k_2,\dots\}$, it merges visual tokens before later layers process them. During decoding, after some early layers, it compresses the KV cache by dropping low-importance visual-token cache entries on a per-layer, per-head basis (Hu et al., 30 Aug 2025).

The resulting division of labor is compactly summarized below.

Inference stage	LightVLM mechanism	Reported role
Encoding	Pyramid token merging	Reduce visual tokens hierarchically while preserving dominant tokens
Decoding	Attention-based KV cache compression	Remove unnecessary visual-token caches to reduce latency and memory

This two-stage design is important because earlier acceleration work frequently targeted only one side of the pipeline. LightVLM’s stated contribution is to improve both prefilling and long-sequence generation in a unified framework. The paper also emphasizes compatibility with both vanilla attention and FlashAttention, including the case where FlashAttention returns only per-token cumulative attention scores rather than full attention matrices (Hu et al., 30 Aug 2025).

3. Pyramid token merging during encoding

The encoding-stage mechanism starts from attention-derived token importance. For a layer with $N$ visual tokens and channel dimension $C$, the visual-token representation is written as

$P \in \mathbb{R}^{N \times C},$

and the head-averaged attention scores over these tokens as

$A \in \mathbb{R}^{N}.$

These scores are used to identify dominant and non-dominant tokens (Hu et al., 30 Aug 2025).

Suppose the goal is to reduce the token count by $r$, so that $N' = N-r$. LightVLM sorts the tokens by attention importance and splits them into an unmerged subset and a merged subset. The top $\times$0 tokens become

$\times$1

while the remaining $\times$2 tokens become

$\times$3

Rather than performing iterative pairwise merges, the method collapses the merge into a single weighted sum using a precalculated vector

$\times$4

and forms a merged token

$\times$5

The compressed sequence is then

$\times$6

The paper describes this as a deterministic, constant-time merging procedure with minimal overhead, designed specifically to remain compatible with FlashAttention’s reduced attention outputs (Hu et al., 30 Aug 2025).

The “pyramid” qualifier refers to repeated application at multiple depths. In the default Qwen2.5-VL 7B setting reported in the appendix, token merging is applied at layers 5, 9, and 13 of the LLM, using a constant reduction ratio chosen according to the final visual-token budget. This produces a hierarchical retention schedule in which shallow layers preserve more local detail and deeper layers retain only a small set of semantically dominant tokens. Visualizations reported in the appendix show that retained tokens concentrate on text regions, formulas, curves, legends, key objects, faces, and textual elements in both natural images and video frames (Hu et al., 30 Aug 2025).

4. KV cache compression during decoding

The decoding-side mechanism targets the KV cache, whose size grows linearly with generated sequence length, number of layers, and hidden size. In VLMs this burden is compounded by the presence of thousands of visual input tokens. The paper reports that increasing output length from 128 to 8192 tokens can add 31.2 GB of memory for KV cache alone, motivating explicit cache compression (Hu et al., 30 Aug 2025).

LightVLM uses attention scores over visual tokens to decide which cached entries to keep. For a given layer and head, it computes per-token importance scores, sorts tokens by descending importance, and retains only the smallest prefix whose cumulative attention mass exceeds a threshold $\times$7. With sorted indices $\times$8 and cumulative sum

$\times$9

it finds the minimal $\times$0 such that

$\times$1

It then keeps only the tokens $\times$2 in the KV cache and discards the rest. The appendix reports a default setting of $\times$3 (Hu et al., 30 Aug 2025).

This is performed independently per layer and per head. Because attention is more uniform in shallow layers, the same threshold retains more tokens there, while deeper layers with steeper attention distributions retain fewer tokens. The method therefore induces an adaptive depth-aware cache budget without retraining. The paper’s ablation also shows complementarity between the two compression stages: token merging alone reduces latency for short and medium sequences, KV compression alone is especially effective for long decoding, and their combination yields the best overall latency (Hu et al., 30 Aug 2025).

5. Empirical results and comparative behavior

The main experimental backbone is Qwen2.5-VL 7B, evaluated on nine image benchmarks—MMMU, MMMU-Pro, MMBench, MMStar, LVLM-eHub, WE, MM-Vet, MuirBench, and RealWorldQA—and four video benchmarks—VideoMME, MVBench, EgoSchema, and MLVU. At 35% visual-token retention, LightVLM retains 100% accuracy; at 15% retention it preserves about 99% average performance; and at 3% retention it preserves about 97.8% relative performance across benchmarks. The paper reports that competing approaches such as SparseVLM, FasterVLM, PyramidDrop, and VisionZip degrade more severely under the same compression ratios, especially at 3% token retention (Hu et al., 30 Aug 2025).

On MMMU with Qwen2.5-VL 7B, the efficiency table gives a baseline throughput of 1.86 images/s and a prefilling time of 428 ms. With LightVLM, throughput increases to 3.75 images/s, corresponding to a 2.02$\times$4 speedup, and prefilling time falls to 117 ms, corresponding to a 3.65$\times$5 reduction. For long outputs, the method reduces latency by 3.21$\times$6 at 4096 generated tokens, reduces memory consumption by 1.85$\times$7, and under an 80 GB NVIDIA A800 memory budget increases the maximum manageable output length by 3.58$\times$8 (Hu et al., 30 Aug 2025).

The framework is also evaluated across model families and scales, including InternVL2.5 8B, 26B, and 38B, LLaVA-OneVision 7B, and MiniCPM-V2.6 8B. A central empirical claim is that LightVLM can make a larger model faster than a smaller one while preserving or improving quality. The paper states, for example, that InternVL2.5 26B equipped with LightVLM achieves higher throughput than vanilla InternVL2.5 8B while also scoring higher on MMMU and MMStar. Similarly, Qwen2.5-VL 7B with LightVLM surpasses vanilla Qwen2.5-VL 3B in throughput while retaining a substantial benchmark advantage (Hu et al., 30 Aug 2025).

These results suggest that the dominant deployment bottleneck in large VLMs is often not raw parameter count alone, but multimodal sequence redundancy. LightVLM’s contribution is to reduce that redundancy in a way that preserves the LLM’s multimodal reasoning signal.

6. Position within the broader “lightweight VLM” landscape

The exact name “LightVLM” refers to the 2025 acceleration framework described above, but adjacent literature uses similar names for substantially different objectives. LiteVLM is a low-latency embedded inference pipeline for resource-constrained environments that combines patch selection, token selection, speculative decoding, and FP8 post-training quantization on NVIDIA DRIVE Thor, reporting 2.5$\times$9 end-to-end latency reduction and 3.2$K=\{k_1,k_2,\dots\}$0 speed-up with FP8 (Huang et al., 9 Jun 2025). Lite VLA uses a compact SmolVLM backbone with LoRA, 4-bit NF4 quantization, and ROS 2 integration for CPU-bound mobile robots, rather than accelerating large VLMs in datacenter settings (Williams et al., 7 Nov 2025). UL-TTA treats a frozen CLIP backbone plus an online Bayesian head over prototypes, priors, and temperature as an “ultra-light” adaptation strategy under domain shift, with less than 8% latency overhead and no backbone updates (Kim, 12 Nov 2025).

Other works use closely related naming conventions at the application-framework level. LightEMMA is a lightweight end-to-end multimodal framework for autonomous driving that hosts multiple VLMs in a common evaluation scaffold rather than compressing them internally (Qiao et al., 1 May 2025). VLMLight integrates a VLM, an LLM meta-controller, and an RL branch for traffic signal control, using heavy multimodal reasoning only in safety-critical cases (Wang et al., 26 May 2025). VLM-IMI uses large VLMs as semantic instruction generators for generative low-light enhancement, where language serves as an enhancement prior rather than an inference bottleneck to be compressed (Sun et al., 24 Jul 2025).

This naming overlap can obscure an important distinction. In the strict sense established by the dedicated paper, LightVLM is not a general label for any lightweight multimodal model. It is a specific training-free acceleration method for existing large VLMs, defined by pyramid token merging during encoding and attention-based KV cache compression during decoding (Hu et al., 30 Aug 2025).