TGV-KV: Text-Grounded KV Eviction for Vision-Language Models

Abstract: Vision-LLMs (VLMs) inherit the auto-regressive generation paradigm and cache the keys and values (KV) of all previous tokens to accelerate inference, resulting in memory consumption that scales linearly with context length. This issue is particularly pronounced in VLMs due to substantial redundancy in the visual modality. Although KV cache eviction approaches can effectively reduce inference memory, they often incur significant performance degradation in VLMs, as most are designed for LLMs and overlook the inherent gap between text and vision. By systematically analyzing the modality gap in VLMs in this work, we argue that the importance of visual information should be grounded in textual guidance and accordingly propose a Text-Grounded KV Eviction method for VLMs (TGV-KV). TGV-KV comprises three submodules: (1) Text-Vision Budgeting (TVB) assigns budget to each layer based on the mutual information interaction. (2) Text-Weighted Ranking (TWR) assesses the priority of text and ranks vision importance based on weighted text-image attention. (3) Text-Prioritised Retention (TPR) policy strategically preserves text KV to avoid acute information loss. We evaluate TGV-KV across five models with different sizes and architectures, showing that TGV-KV preserves 99.2% full-KV accuracy on the VizWiz-VQA task with LLaVA-NeXT and boosts end-to-end throughput by 52.6% with an extreme retention budget of 5%. Code is available at https://github.com/Danielement321/TGV-KV.

• The paper proposes TGV-KV, a text-grounded framework that efficiently evicts redundant KV pairs to bridge the modality gap in VLMs.
• It introduces three modules—TVB, TWR, and TPR—to dynamically allocate retention budgets and compute text-weighted importance for vision tokens.
• Empirical evaluations demonstrate up to 52.6% throughput improvement and minimal accuracy loss under severe KV cache compression.

Text-Grounded KV Eviction for Efficient Vision-LLM Inference

Motivation and Background

Vision-LLMs (VLMs) leverage the auto-regressive generation paradigm inherited from LLMs, relying on the caching of key-value (KV) pairs for all previous tokens during inference. While efficient for accelerating generation, this mechanism induces memory consumption scaling linearly with context length. VLMs exacerbate this challenge because the visual modality introduces substantial redundancy; high-resolution images and videos can result in thousands of tokens, yet much of this information is superfluous. Existing approaches for reducing the KV cache—primarily token pruning and naive KV eviction—frequently harm performance in VLMs due to the modality gap.

The modality gap arises from three factors: (1) vision tokens exhibit high mutual redundancy, unlike the diversity of text tokens; (2) unimodal attention dominates over inter-modality attention; and (3) accumulated attention scores lead to uneven and suboptimal KV eviction, often disproportionately evicting text KVs. Figures below illustrate these phenomena.

Figure 1: Visualization of modality gap: (a) cosine similarity distributions for vision vs. text tokens, (b) low cross-modality attention, (c) sharply uneven cumulative attention-driven KV eviction.

TGV-KV Framework

TGV-KV introduces a text-grounded, synergistic three-module approach for KV cache eviction:

1. Text-Vision Budgeting (TVB): Dynamically allocates per-layer KV retention budgets based on normalized cross-modality attention. The TV attention is summed for each layer, serving as a proxy for semantic fusion and information exchange intensity.
2. Text-Weighted Ranking (TWR): Computes the importance of vision KVs by weighting text-vision attention with dominant text-token scores (from text-text attention maps), establishing a robust text-conditioned ranking for eviction.
3. Text-Prioritised Retention (TPR): Ensures maximal preservation of text KVs given the budget, only evicting text KVs if the budget is exhausted after vision KVs are removed.

   Figure 2: Overview of TGV-KV: TVB assigns layer budgets, TWR generates text-weighted vision importance, TPR enforces text retention priorities.

This formulation directly addresses the modality gap by anchoring visual information to textual relevance, blending intra- and inter-modality attention as eviction criteria and avoiding catastrophic information loss.

Attention and Importance Patterns

Analyzing average attention maps across models reveals persistent vertical patterns in text-text attention, indicating dominant tokens that guide both text and vision importance scoring. Inter-modal attention (text-vision) consistently remains low relative to intra-modal regions, warranting normalized cross-modality-driven budget allocations.

Figure 3: Average layer attention map for LLaVA-1.5-7B and Qwen3-VL-8B. Dominant text tokens manifest as persistent vertical lines in text-text regions.

Ablation experiments demonstrate that TWR consistently outperforms naive self-attention-based scoring, especially when accompanied by TPR for extreme cache reductions. TVB offers performance gains when the budget is sufficient; TPR becomes critical when budgets are stringent, preventing accuracy collapse.

Empirical Evaluation

Vision-Dominant and Text-Dominant Accuracy

TGV-KV is evaluated on five diverse VLMs including LLaVA-series and Qwen3-VL-series. On vision-dominant tasks (ChartQA, DocVQA, VizWiz, TextVQA), the method achieves:

• LLaVA-NeXT: Retains 99.2% of full-KV accuracy on VizWiz at 5% retention rate and yields robust results across all extreme settings.
• Qwen3-VL-4B: Preserves 92.5% accuracy on DocVQA at 5% budget, outperforming baselines by up to 33% in high-resolution scenarios.

In text-dominant settings (TextCaps, COCO-Caption), TGV-KV maintains performance, dropping <0.5% under tight budgets and achieving significant improvement over second-best baselines (relative boost of 57.4% over StreamingLLM with LLaVA).

Video Reasoning and Long Contexts

On Video-TT, TGV-KV achieves near-vanilla performance even under severe cache constraints, with ≤2 percentage point drops in the most challenging settings.

Figure 4: TGV-KV achieves Pareto-efficient accuracy retention under extreme KV budget compression for video reasoning tasks.

Efficiency Benchmarks

TGV-KV demonstrates substantial KV cache memory reduction and decoding throughput improvement:

• 5% retention: Reduces memory consumption from 3.91 GB to 0.20 GB (8k context), accelerates throughput by 52.6%
• Scaling: Holds improvements at 16k token context with memory reduction ≈51% and throughput increases up to 52.6%

Comparative Pruning Studies

TGV-KV outperforms token pruning solutions, maintaining superior accuracy and reducing hallucination while incurring no loss of unextracted information. System efficiency is also enhanced, as iterative decoding dominates latency and benefits from eviction.

Qualitative Visualizations

Systematic visualization of retained KVs illustrates that TGV-KV accurately preserves patches and text tokens most aligned with semantic queries. Shallow layers receive increased cache budgets, corresponding to high cross-modality interaction, and dominant instruction phrases remain intact in all retained layers.

Figure 5: Retained KVs in a POPE sample; patches most relevant to text instruction are retained (circled), evicted KVs are masked.

Additional attention maps highlight the persistent attention gap and utility of text-grounded retention.

Figure 6: Log-scale visualization of attention map; cross-modality remains suppressed, dominant text tokens drive retention.

Implications and Future Directions

The TGV-KV paradigm enables practical compression of KV cache for VLMs without performance compromise, making deployment feasible under severe memory constraints and long-context scenarios. The findings—grounding visual relevance in text-driven importance, preserving text modality, and dynamically allocating retention budgets—address the modality gap at its root, offering a generalizable approach for multimodal inference acceleration.

Future work may extend TGV-KV via adaptive importance scoring leveraging learned cross-modality representations, explore hierarchical pruning guided by semantic grounding, and further optimize cache retention for streaming or continual multimodal reasoning, as well as integrating with next-generation architectures (e.g., unified image-video tokenization).

Conclusion

TGV-KV systematically addresses multimodal KV eviction in VLMs via text-grounded attention-driven retention that bridges the modality gap. Empirical analyses support the efficacy of its three-pronged module design in preserving both accuracy and efficiency across a spectrum of tasks, modalities, and architectures. TGV-KV offers broad implications for memory-efficient multimodal inference and sets the stage for advancing robust, scalable vision-language deployments.