Vision–Text Compression Techniques

• Vision–text compression encompasses techniques that condense visual and textual data for multimodal tasks while preserving critical semantic information.
• Methods such as text rasterization, dynamic token pruning, and neural distillation enable 2–10× compression by optimizing token budgets and leveraging adaptive strategies.
• Empirical results demonstrate up to 50–90% token reduction with minimal accuracy loss, facilitating longer context handling and faster inference in vision-language models.

Vision–Text Compression encompasses a family of strategies and models that reduce the computational, memory, or representational cost of joint vision-language tasks by compressing either visual or textual information, or their mappings, in a manner that preserves task-critical semantics. Techniques include rendering text as images for compact multimodal ingestion, pruning or summarizing visual tokens, and leveraging distillation or pooling to decrease model size and accelerate inference while maintaining or minimally reducing performance. Recent advances enable scaling to extremely long contexts and document-level reasoning using vision-language pretraining and optical compression methods.

1. Rendering Text as Images for Compression

A core principle in recent vision–text compression research is the representation of text not as discrete tokens, but as rasterized images, thereby leveraging the compression capacity of vision encoders. The “ConTexImage” pipeline, as deployed in "Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMs" (Li et al., 21 Oct 2025), and further generalized by frameworks such as Glyph (Cheng et al., 20 Oct 2025), renders long text spans into high-resolution images. Key pipeline steps are:

• Text normalization and LaTeX rendering: Clean text (normalize typography, escape symbols), wrap as a LaTeX document, compile to PDF, and rasterize at controlled DPI.
• Adaptive font sizing: Maximize the fill ratio (target 0.7–0.8) to exploit image area without loss of legibility.
• Image resolution tuning: Output PNGs at resolutions such as 600×1000 or 750×1000, which determine the final visual token budget (number of patches per image).
• Batched visual tokenization: Each image is split into patches (e.g., 14×14 or 16×16), producing a manageable sequence length for standard vision-language pipelines.

This paradigm enables 2–4× compression of input sequence length, dramatically increasing effective context window for LLMs (Li et al., 21 Oct 2025, Cheng et al., 20 Oct 2025).

2. Quantitative Compression Metrics and Effectiveness

Compression effectiveness is formalized using token budget and savings equations:

• Token budgets: Let $m$ be text tokens, $k$ be visual tokens per image; the full-model input becomes $T_{\rm text} = m + |{\mathbf q}|$ (text) and $T_{\rm img} = k + |{\mathbf q}|$ (image); $k \ll m$ in practice (Li et al., 21 Oct 2025).
• Compression ratio: $$\rho = \frac{T_{\rm text}}{T_{\rm img}} \approx \frac{m}{k}$$; relative savings $C = 1 - \frac{T_{\rm img}}{T_{\rm text}}$.
• Empirical outcomes: C~0.5 (≈50% savings) is robustly achievable across retrieval and summarization benchmarks without additional fine-tuning (Li et al., 21 Oct 2025). Glyph (Cheng et al., 20 Oct 2025) achieves 3–4× compression across 100,000+ token contexts.

Furthermore, architectural integration is minimal: visual tokens from a frozen vision encoder are simply appended or interleaved with language tokens, requiring no model architecture change or parameter update (Li et al., 21 Oct 2025, Cheng et al., 20 Oct 2025).

3. Dynamic Visual Token Compression and Recovery

Token-level pruning and merging are crucial for scaling VLMs to high-resolution and multi-image scenarios.

• Training-free dynamic compression: Modules such as DFMR in LLaVA-Zip (Wang et al., 11 Dec 2024) and recoverable compression (Chen et al., 2 Sep 2024) adaptively compress visual tokens based on intrinsic image statistics (e.g., spatial variance in feature maps) or their relevance to text queries.
• Formulation: For an input feature map $\mathbf V \in \mathbb{R}^{H'\times W'\times D_v}$, DFMR pools non-overlapping windows by a factor $s$ chosen to preserve patch variance above a threshold $\tau$, yielding $r = s^2$-fold compression.
• Text-guided recovery: Local Outlier Factor (LOF) and similarity scoring (visual–text embedding dot products) identify semantically-important tokens to retain or recover, with the rest clustered and averaged (Chen et al., 2 Sep 2024).
• Effectiveness: LLaVA-Zip achieves up to 89% token reduction (s=3, 576→64 tokens) with smaller accuracy degradation than global pooling or random pruning (Wang et al., 11 Dec 2024). Recoverable Compression achieves ~10× reduction (to 8–10% of tokens) with <0.5% performance loss (Chen et al., 2 Sep 2024).

Such approaches drastically reduce VRAM usage and inference latency in multi-modal LLM settings.

4. Neural Model Compression and Distillation for Vision–Text

Beyond token sequence compression, neural model distillation and quantization techniques reduce the size and compute requirements of dual-encoder and diffusion architectures:

• Two-stage teacher–student distillation: In "Leaner and Faster," a large dual-encoder (e.g., CLIP with ViT-B/32) is compressed via (1) intra-modal contrastive distillation with unpaired data, and (2) fine-tuning with KD and hard-negative mining (Ren et al., 2022).
• Lightweight adaptors and pooling: FCoT-VL (Li et al., 22 Feb 2025) uses a 1D convolutional compression module with stride 2, iterated for 4× downsampling, combined with self-distillation and high-quality post-training to preserve task accuracy on text-oriented vision benchmarks.
• Quantization in generative models: Vector quantization (VQ) of weights in text-to-image diffusion models (e.g., SDXL) reduces model size to ~3 bits/weight with negligible FID/CLIP/qualitative degradation compared to scalar quantization at 4 bits (Egiazarian et al., 31 Aug 2024).

Resultant models see parameter count and memory drops of ~60% while maintaining or even exceeding teacher performance after fine-tuning (Ren et al., 2022, Li et al., 22 Feb 2025).

5. Task-Driven Rendering and Adaptation Strategies

Compression efficacy is influenced by task, context, and adaptive rendering:

• Genetic search for optimal rendering: Glyph (Cheng et al., 20 Oct 2025) employs an LLM-guided evolutionary search over the space of rendering configurations (DPI, font, size, cropping) to maximize token savings at a user-specified accuracy floor.
• Auxiliary objectives: OCR reconstruction and Levenshtein-based auxiliary losses stabilize text recovery at high compression, while Chain-of-Thought augmentation in FCoT-VL enables reasoning on chart and math tasks (Li et al., 22 Feb 2025).
• Multi-resolution and history fading: DeepSeek-OCR (Wei et al., 21 Oct 2025) demonstrates stage-wise optical compression (convolutional downsampling then global attention), yielding >96% OCR accuracy at 10× token compression. Progressive fading (higher compression for distant context) is used to simulate memory decay and support ultra-long document history.

Empirical results confirm that task-specific and content-aware adaptation of both vision and text modality compression yields significant improvements in real-world and benchmark scenarios.

6. Efficiency–Accuracy Trade-Offs and Limitations

Vision–text compression consistently delivers substantial speed and memory benefits with bounded accuracy impact:

• Performance retention: ≤0.7–0.8 point drop in BERTScore or ROUGE-L at 50–67% compression (text-to-image) (Li et al., 21 Oct 2025), <0.5% accuracy drop at 10× visual token compression with LOF-based recovery (Chen et al., 2 Sep 2024).
• Latency and throughput: 25–45% inference speedup from reduced decoder/input size (Li et al., 21 Oct 2025), 2× faster supervised fine-tuning training for compressed context models (Cheng et al., 20 Oct 2025), 69.6% acceleration for extreme vision token compression (r=576) (Ye et al., 18 Jun 2024).
• Compression–utility limits: At 8–10× token compression, most semantic information is preserved; at 20× or higher, degradation accelerates (e.g., DeepSeek-OCR drops from 97%→60% OCR precision), especially on rare, complex, or highly-structured text (Wei et al., 21 Oct 2025).
• Current weaknesses: Fine detail (e.g., codes, UUIDs) and rare sequences are more error-prone at high visual-text compression ratios, necessitating careful tuning and potential use of hybrid (modality-mixed) solutions for maximal reliability (Cheng et al., 20 Oct 2025, Wei et al., 21 Oct 2025).

A summary table of selected results:

Approach / Paper	Max Compression	Task(s)	Relative Accuracy	Speed/Resource Savings
ConTexImage (Li et al., 21 Oct 2025)	2×	Retrieval, Summ	>97%	≤45% faster, <2% loss
Glyph (Cheng et al., 20 Oct 2025)	3–4× (8× ext.)	Long context	–	4.8× prefill, 4.4× decode
DFMR (LLaVA-Zip) (Wang et al., 11 Dec 2024)	8.9×	VQA	~+4.8% > baselines	Up to 9× fewer tokens
Recoverable Compression (Chen et al., 2 Sep 2024)	10×	ScienceQA, VQA	<0.5% loss	80% fewer FLOPs, –7.8 GB
VoCo-LLaMA (Ye et al., 18 Jun 2024)	576×	VQA, Video QA	83.7% (1-token)	95% fewer FLOPs, 70% faster
DeepSeek-OCR (Wei et al., 21 Oct 2025)	10–20×	OCR	97% (≤10×), 60% (20×)	Top accuracy w/ few tokens
FCoT-VL (Li et al., 22 Feb 2025)	4×	OCR/Chart QA	>100% vs. orig.	2.4× faster inference

7. Research Directions and Open Problems

Vision–text compression is advancing on several axes:

• Extending context windows: Enabling million-token context handling via visual rendering (Glyph) redefines the long-context frontier for LLMs (Cheng et al., 20 Oct 2025).
• Query- and task-aware adaptivity: Dynamic, instance- and content-aware token budgets, leveraging both intrinsic visual features and external grounding (text query, document layout) (Wang et al., 11 Dec 2024, Chen et al., 2 Sep 2024).
• Generalization: Expanding from generic document understanding and VQA to code, math, legal, tabular, and open-domain reasoning, with careful management of rare or fine-structured elements.
• Hardware acceleration and further quantization: Custom hardware for LUT-based VQ inference and activation quantization remains a future opportunity to maximize on-device scaling (Egiazarian et al., 31 Aug 2024).
• Combined strategies: Hybrid approaches that first prune, then render, or iteratively adapt compression settings during inference, deliver additive gains (Li et al., 21 Oct 2025, Cheng et al., 20 Oct 2025).

Remaining limitations include rendering sensitivity, rare character handling, and maintaining reversibility and information fidelity for ultra-high compression regimes (Cheng et al., 20 Oct 2025, Wei et al., 21 Oct 2025). Adaptive, content-aware compression conditioned on the current prompt/query, and cross-modal fusion architectures for higher-level reasoning, are active research frontiers.