ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning

Abstract: Large Vision-LLMs (LVLMs) incur high computational costs due to significant redundancy in their visual tokens. To effectively reduce this cost, researchers have proposed various visual token pruning methods. However, existing methods are generally limited, either losing critical visual information prematurely due to pruning in the vision encoder, or leading to information redundancy among the selected tokens due to pruning in the LLMs. To address these challenges, we propose a Visual and Textual Semantic Collaborative Pruning framework (ViTCoP) that combines redundancy filtering in the vision encoder with step-wise co-pruning within the LLM based on its hierarchical characteristics, to efficiently preserve critical and informationally diverse visual tokens. Meanwhile, to ensure compatibility with acceleration techniques like FlashAttention, we introduce the L2 norm of K-vectors as the token saliency metric in the LLM. Extensive experiments on various Large Vision-LLMs demonstrate that ViTCoP not only achieves state-of-the-art performance surpassing existing methods on both image and video understanding tasks, but also significantly reduces model inference latency and GPU memory consumption. Notably, its performance advantage over other methods becomes even more pronounced under extreme pruning rates.

• The paper introduces ViTCoP, a tuning-free, three-stage framework that combines visual and textual semantic collaborative pruning to improve the efficiency of Large Vision-Language Models (LVLMs).
• ViTCoP significantly reduces the inference latency and GPU memory usage by pruning redundant visual tokens, retaining 98.5% and 95.3% performance at 66.7% and 88.9% pruning respectively.
• The method leverages K-vector L2 norms as a saliency proxy and clustered merging of tokens, ensuring the preservation of dominant information for different tasks.

Motivation and problem statement

Large Vision-LLMs (LVLMs) such as LLaVA-1.5, LLaVA-NeXT, and LLaVA-NeXT-Video encode images or video frames into hundreds to thousands of visual tokens that are concatenated with text tokens and processed by an LLM. Because Transformer attention cost scales quadratically with sequence length, this visual token volume dominates inference latency and GPU memory. The paper's premise is that visual tokens are highly redundant: on LLaVA-1.5-7B, the top 10% of tokens by attention score account for over 60% of total attention weight, and retaining only the top 20% preserves roughly 95% of benchmark performance.

The authors argue that existing pruning methods fall into two flawed paradigms. Text-agnostic methods (e.g., VisionZip) prune in the vision encoder using purely visual saliency; they can discard task-critical regions when the query targets a specific part of the image. Text-guided methods (e.g., FastV, PyramidDrop, SparseVLM) prune inside the LLM using attention scores against the instruction; they preserve relevance but produce subsets with high informational redundancy when instructions are broad or reference similar concepts. ViTCoP is designed to combine both signals across the processing pipeline rather than choosing one.

Key empirical insights

The method rests on three observations established empirically before the method is introduced:

1. Early redundancy: a large fraction of visual tokens can be removed before entering the LLM with minimal performance impact.
2. K-vector L2 norm as saliency proxy: there is a strong negative correlation between a token's K-vector L2 norm and its attention score within the LLM; pruning by smallest norm matches or exceeds attention-based pruning. This matters practically because attention scores are inaccessible under FlashAttention-style fused kernels, whereas K norms are not.
3. Layer-dependent token roles: attention heatmaps across all 32 LLM layers show diffuse, global attention over visual tokens in shallow layers, transitioning to sparse, focused attention in deep layers — i.e., global aggregation early, local detail absorption late.

Method

ViTCoP is a tuning-free, three-stage framework:

Stage I — vision encoder pruning. At the penultimate encoder layer, tokens are ranked by summed [CLS]-to-token attention across heads and the top fraction retained, removing background and repetitive-texture redundancy before projection into the LLM.

Stage II — collaborative pruning at shallow LLM layers (layer 2). A Visual Information Clustering (VIC) algorithm groups retained tokens by feature-space density ($\rho_i$ via Gaussian kernel distances) subject to a spatial proximity threshold $\tau$, yielding semantically coherent clusters. Within each cluster, a size-proportional retention quota $q_c$ is allocated; elite tokens are those with the smallest K-vector L2 norms (highest textual relevance), and remaining tokens are merged into one representative vector per cluster by averaging. This jointly enforces diversity (via clustering) and query relevance (via K-norm selection).

Stage III — text-guided pruning at deep layers (layer 22). Tokens are ranked solely by K-vector L2 norm and aggressively pruned, since core information has already been aggregated by then.

The layer placement is shown to be sensitive: shallow pruning at layer 1 degrades performance because cross-modal information has not yet converged, and deep pruning at layer 17 causes severe degradation because critical information is discarded before absorption. Hyperparameters ($d_c=8$, $\tau=0.6$) were fixed via grid search on COCO-2017 only and held constant across all benchmarks.

Results

On LLaVA-1.5-7B across 11 image benchmarks, ViTCoP achieves the best average retention at all three compression levels tested:

Method	192 tokens (66.7% pruned)	128 tokens (77.8%)	64 tokens (88.9%)
FastV	92.8%	88.9%	72.5%
PyramidDrop	96.9%	93.2%	76.6%
SparseVLM	96.8%	90.6%	80.7%
VisionZip	97.6%	95.4%	90.4%
ViTCoP	98.5%	97.3%	94.7%

The advantage widens under extreme pruning: at 64 retained tokens ViTCoP leads VisionZip by 4.3 points and SparseVLM by 14 points. On some benchmarks (MMBench 100.3%, OK-VQA 100.4%) it slightly exceeds the unpruned model, which the authors attribute to removal of interfering information. On LLaVA-NeXT-7B (2880 tokens), ViTCoP retains 95.1% average performance at 88.9% pruning versus VisionZip's 91.4%, and 90.8% at 94.4% pruning versus 83.3%; notably, FastV, PyramidDrop, and SparseVLM could not run at the 94.4% setting. On LLaVA-NeXT-Video-7B at 88.9% pruning, ViTCoP averages 97.7% retention across four video benchmarks, exceeding SparseVLM's 95.1%, and reaches 100.8% on Video-MME — again above the unpruned baseline.

Efficiency results on LLaVA-NeXT-13B show TFLOPs reduced by ~94.6%, prefill latency reduced by ~85%, and GPU memory reduced to 88.1% of baseline, while outperforming VisionZip's accuracy by roughly 10 percentage points on POPE at comparable cost.

Ablation findings

The ablation study isolates each component. Replacing K-norm guidance with attention scores (or vice versa) yields comparable performance, confirming the K-norm as a robust, FlashAttention-compatible proxy. Removing Stage I causes catastrophic collapse (e.g., COCO CIDEr drops from 1.032 to 0.086), indicating that coarse background removal is essential for downstream stages to function. Removing Stage III slightly improves COCO captioning, suggesting fine-grained captioning tasks are sensitive to loss of detailed semantics in deep-layer pruning — a trade-off the authors acknowledge but do not fully resolve.

Limitations and open questions

Several caveats bear directly on the reported results. First, VIC has quadratic complexity in the number of Stage I survivors, $O((\pi_1 m)^2)$, so the approach's overhead grows with the retained-token budget; the paper does not evaluate scaling behavior for very large $\pi_1$. Second, hyperparameters were tuned on COCO-2017 and validated only on LLaVA-family models; generalization to other LVLM architectures (e.g., Qwen2.5-VL, InternVL) is asserted but not demonstrated. Third, the layer positions (2 and 22) were selected empirically and shown to be sensitive, raising the question of whether they transfer to LLMs with different depths. Fourth, the Stage III ablation reveals that aggressive deep pruning can hurt detail-sensitive tasks like image captioning, and the paper offers no adaptive mechanism to modulate pruning intensity per task. Finally, the claim that K-vector L2 norm reliably proxies attention-based saliency is supported by correlation analysis on specific models; whether this relationship holds across architectures and training regimes remains open.

Conclusion

ViTCoP contributes a three-stage, tuning-free pruning framework that sequences visual-saliency filtering, vision-text collaborative clustering with merging, and text-saliency pruning according to the functional evolution of visual information through the LLM. Its use of K-vector L2 norms as a FlashAttention-compatible saliency metric and its consistent superiority at extreme compression rates (94.7% retention at 88.9% pruning on LLaVA-1.5-7B; 97.7% on video tasks) represent concrete advances over prior text-agnostic and text-guided baselines, with over 94% FLOPs reduction. The main unresolved issues are the quadratic cost of the clustering stage, architecture generality, and task-adaptive control of deep-layer pruning aggressiveness.