Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models

Abstract: Network pruning is an effective technique for enabling lightweight Large Vision-LLMs (LVLMs), which primarily incorporates both weights and activations into the importance metric. However, existing efforts typically process calibration data from different modalities in a unified manner, overlooking modality-specific behaviors. This raises a critical challenge: how to address the divergent behaviors of textual and visual tokens for accurate pruning of LVLMs. To this end, we systematically investigate the sensitivity of visual and textual tokens to the pruning operation by decoupling their corresponding weights, revealing that: (i) the textual pathway should be calibrated via text tokens, since it exhibits higher sensitivity than the visual pathway; (ii) the visual pathway exhibits high redundancy, permitting even 50% sparsity. Motivated by these insights, we propose a simple yet effective Asymmetric Text-Visual Weight Pruning method for LVLMs, dubbed ATV-Pruning, which establishes the importance metric for accurate weight pruning by selecting the informative tokens from both textual and visual pathways. Specifically, ATV-Pruning integrates two primary innovations: first, a calibration pool is adaptively constructed by drawing on all textual tokens and a subset of visual tokens; second, we devise a layer-adaptive selection strategy to yield important visual tokens. Finally, extensive experiments across standard multimodal benchmarks verify the superiority of our ATV-Pruning over state-of-the-art methods.

• The paper introduces ATV-Pruning, which preserves all text calibration tokens while selecting high-drift visual tokens to address modality imbalance in LVLM pruning.
• ATV-Pruning achieves 94.00% performance retention at 50% sparsity and 77.01% at 60% on LLaVA-NeXT 8B, outperforming Wanda and TAMP across key benchmarks.
• The method takes 99.6 seconds versus 1,418 seconds for TAMP and shows that visual pathways tolerate substantial weight sparsity, while textual calibration remains highly sensitive.

Motivation and problem statement

Large Vision-LLMs (LVLMs) inherit the computational burden of their LLM backbones, which is exacerbated by long visual token sequences from high-resolution images. Post-training weight pruning—exemplified by SparseGPT and Wanda—is an attractive remedy, but these methods were developed for unimodal LLMs and assume homogeneous token statistics within a single calibration pool. When applied to LVLMs, where text and visual tokens occupy distinct clustered regions of activation space and yield pruning masks with low mutual IoU, this modality-agnostic assumption degrades accuracy. Prior LVLM-specific work such as TAMP selects multimodal calibration tokens but still treats both modalities within a shared selection process.

The paper's central contribution is a controlled disentanglement study showing that the two modalities behave asymmetrically under pruning, followed by a pruning method—ATV-Pruning—that exploits this asymmetry directly in the calibration-pool construction.

Modality-aware sensitivity analysis

Using a Mixture-of-Transformer (MoT) probe on LLaVA-NeXT 8B, the authors replicate the QKV and FFN layers into separate textual and visual pathways and prune each independently using calibration pools drawn from text-only, image-only, or mixed tokens at 50% and 60% unstructured sparsity. Two findings emerge:

Finding A: the textual pathway is highly sensitive to calibration source. At 60% sparsity, text-pathway performance on ScienceQA-IMG collapses to 35.85 (mixed) or 11.11 (visual-only) versus 61.58 with a text-only pool. Text-only calibration consistently gives the best retention for the textual pathway; mixed pooling behaves better than visual-only but still substantially worse.

Finding B: the visual pathway is highly redundant. Pruning only the visual pathway to 60% sparsity retains over 99.25% of dense performance regardless of which calibration pool is used—even a text-only pool suffices. This indicates that text-driven activation statistics already identify most weights critical to visual processing, and that weight-level redundancy in the vision pathway is distinct from the token-level redundancy targeted by visual token pruning methods.

An additional observation reinforces the asymmetry: on VizWiz at 50% sparsity, pruned models exceed the dense baseline, and the paper attributes these gains primarily to pruning within the text pathway under text-only calibration. The practical takeaway is that calibration should be text-anchored and augmented with only a compact set of salient visual tokens—a conclusion with direct implications for any activation-aware pruning pipeline applied to multimodal models.

ATV-Pruning method

ATV-Pruning builds on Wanda's importance score $I_{ij} = |W_{ij}| \cdot \|X_j\|_2$ but replaces the modality-agnostic calibration pool with an asymmetric one. For each Transformer block:

• Full text retention: all text-token positions $\mathcal{T}$ are included.
• Block-adaptive visual selection: visual saliency $s_v$ is instantiated as visual drift, the cosine distance between a token's input and output representations within the block ($$s_v = 1 - \cos(X_{\mathrm{in},v}, X_{\mathrm{out},v})$$). The block-average drift $\bar{s}$ determines a budget $$K = \lfloor \alpha \cdot \bar{s} \cdot n_{\text{text}} \rfloor$$, and the top-$K$ most drifting visual tokens per sample form $\mathcal{V}_{\text{sub}}$.
• Activation norms are then estimated only over $$\mathcal{S}_{\text{cal}} = \mathcal{T} \cup \mathcal{V}_{\text{sub}}$$.

The rationale for drift is that blocks which substantially update a visual token are actively processing it, so its activations carry more informative statistics there. With $\alpha = 1.0$ on LLaVA-NeXT, roughly 15 visual tokens per sample per block are retained. The design depends on the assumption that representational change is a faithful proxy for computational involvement—an assumption the authors validate empirically rather than theoretically.

Main results

Experiments use 128 ShareGPT4V image–text pairs for calibration, lmms-eval as the unified harness, and nine benchmarks (GQA, MMBench-EN, MME, MMMU, OK-VQA, POPE, SQA-img, TextVQA, VizWiz).

Model / setting	SparseGPT	Wanda	TAMP	ATV-Pruning
LLaVA-NeXT 8B, 50%	91.74%	88.36%	92.67%	94.00%
LLaVA-NeXT 8B, 60%	76.24%	64.45%	73.75%	77.01%
Qwen2-VL 7B, 60%	85.30%	77.78%	83.79%	85.65%

At 50% sparsity on LLaVA-NeXT, ATV-Pruning ranks first on GQA, MME, MMMU, OK-VQA, and VizWiz. At 60%, it leads MMMU, OK-VQA, SQA-img, and VizWiz, exceeding Wanda and TAMP by +12.56 and +3.26 average points respectively—the advantage over Wanda-style baselines widens with sparsity. Supplementary results extend the claim across architectures: LLaVA-OneVision 7B (88.07% retention) and Qwen2.5-VL 7B (86.44%, surpassing even SparseGPT by +0.69), plus hardware-friendly semi-structured formats where ATV-Pruning reaches 76.29% under 2:4 and 85.54% under 4:8 patterns on LLaVA-NeXT.

Efficiency

A key practical claim is that the accuracy gains come at negligible cost relative to Wanda: ATV-Pruning takes 99.6 s to prune LLaVA-NeXT 8B at 50% sparsity versus 666 s for SparseGPT and 1418 s for TAMP, while delivering +5.64 percentage points over Wanda. The modest 1.35× overhead stems from computing the cosine-distance drift. This positions ATV-Pruning as a plug-in replacement for Wanda's scoring step rather than a heavyweight pipeline like TAMP's attention-plus-MMD selection.

Ablations

Three ablations support the design choices. First, the scaling factor $\mathcal{T}$0: even $\mathcal{T}$1 (pure text-only calibration) yields 64.69 on MMBench, already beating naive Wanda (63.83), confirming the primacy of text tokens; moderate values ($\mathcal{T}$2) improve robustness, while large values ($\mathcal{T}$3) degrade toward the modality-agnostic baseline by diluting the linguistic signal. Second, replacing adaptive drift-based selection with random or fixed-budget variants reduces average retention from 98.56% to 97.50% and 98.24% respectively; alternative saliency signals (attention-based ABS, diversity-based DBS with max-min selection) perform comparably, with drift giving the best overall balance. Third—and notably—applying selective pruning to textual tokens always hurts: reducing the retained text-token ratio from 1.0 to 0.9 drops MMBench from 64.52 to 63.14 and SQA-img from 69.71 to 67.97, confirming that unlike the visual pathway, the text pathway admits no redundancy in calibration.

Limitations and open questions

Several caveats bear on the results. The sensitivity analysis and most experiments rely on LLaVA-NeXT as the analysis backbone, so the asymmetry findings, while replicated across Qwen2-VL, LLaVA-OneVision, and Qwen2.5-VL for final accuracy, have not been independently verified via the MoT probe on other architectures. The method requires a global hyperparameter $\mathcal{T}$4 whose optimal value differs across models (1.0 vs. 1.5), introducing mild tuning burden. The visual drift heuristic is validated empirically against alternatives but lacks a formal justification for why representational change correlates with calibration informativeness. Finally, evaluation covers unstructured and N:M sparsity on image-centric benchmarks; whether the asymmetric calibration principle transfers to video, audio, or broader multimodal settings—as the authors propose extending it—remains open, as does the question of whether weight-level visual redundancy persists in models with stronger vision-language entanglement.

Conclusion

This paper identifies a concrete failure mode of modality-agnostic calibration in LVLM weight pruning—dilution of linguistic signal—and resolves it with an asymmetric scheme that keeps all text tokens while layer-adaptively sampling a small set of high-drift visual tokens. Across four model families, unstructured and semi-structured sparsity regimes, and nine benchmarks, ATV-Pruning matches or exceeds state-of-the-art baselines at near-Wanda cost, with the clearest margins appearing at higher sparsity. Its main lasting contribution may be the empirical demonstration that the visual pathway tolerates 60% parameter sparsity almost losslessly, provided calibration remains text-anchored.