Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMs (2510.18279v1)

Abstract: LLMs and their multimodal variants can now process visual inputs, including images of text. This raises an intriguing question: can we compress textual inputs by feeding them as images to reduce token usage while preserving performance? In this paper, we show that visual text representations are a practical and surprisingly effective form of input compression for decoder LLMs. We exploit the idea of rendering long text inputs as a single image and provide it directly to the model. This leads to dramatically reduced number of decoder tokens required, offering a new form of input compression. Through experiments on two distinct benchmarks RULER (long-context retrieval) and CNN/DailyMail (document summarization) we demonstrate that this text-as-image method yields substantial token savings (often nearly half) without degrading task performance.

• The paper demonstrates that converting long textual contexts to images reduces required decoder tokens by nearly 50% while maintaining 97–99% accuracy.
• The paper employs a LaTeX-based typesetting pipeline to render text into visual tokens, achieving a consistent compression ratio of approximately 2 across benchmarks.
• The method outperforms token-pruning baselines in tasks such as retrieval and summarization, offering up to 45% end-to-end processing speedup.

Token Efficiency of Visual Text Inputs in Multimodal LLMs

Introduction

This paper investigates the token efficiency of visual text inputs in multimodal LLMs (MLLMs), specifically examining whether rendering textual context as images can serve as a practical compression strategy for long-context inference. The motivation stems from the quadratic scaling of Transformer self-attention with input length, which imposes significant computational and cost constraints on LLM deployment for tasks involving extended contexts. The authors propose a text-as-image pipeline that leverages the vision encoder of MLLMs to process long textual inputs as images, thereby reducing the number of decoder tokens required without sacrificing downstream performance.

Figure 1: Illustration of the text-as-image compression pipeline, showing how a 90-token context is replaced by a single image, reducing decoder token usage by nearly half.

Methodology

The core approach involves converting a long textual context $\mathbf{c}$ into a rasterized image $I = \mathcal{R}(\mathbf{c})$ using a LaTeX-based typesetting pipeline. The image is then encoded by the model's vision module into a sequence of $k$ visual tokens, which are projected and concatenated with the textual query $\mathbf{q}$ for input to the language decoder. The effective input length is thus $T_{\text{img}} = k + |\mathbf{q}|$, compared to the text-only baseline $T_{\text{text}} = m + |\mathbf{q}|$, where $m$ is the original text token count. The compression ratio $\rho \approx m/k$ quantifies the token savings.

The pipeline is model-agnostic and does not require fine-tuning or supervision. All experiments use the native vision encoder of each MLLM. The authors evaluate the approach on two benchmarks: RULER (long-context retrieval) and CNN/DailyMail (document summarization), using GPT-4.1-mini and Qwen2.5-VL-72B-Instruct as representative models.

Empirical Results

Long-Context Retrieval (RULER S-NIAH)

The RULER benchmark tests the model's ability to extract a single target ("needle") from a long distractor passage ("haystack"). The authors sweep the context length $m$ while holding the visual token budget $k$ fixed, measuring accuracy and token savings.

Figure 2: Accuracy vs. text-token size ($m$) with fixed visual tokens ($k$) for GPT-4.1-mini.

Figure 3: Qwen2.5-VL-7B-Instruct: accuracy vs. text-token size ($m$) with fixed visual tokens ($k$).

Figure 4: Maximum text tokens $m^\star$ that can be preserved without accuracy loss, plotted against the visual tokens $k$ generated from the image.

Key findings:

• Both GPT-4.1-mini and Qwen2.5-VL-72B-Instruct sustain 97–99% accuracy with up to 58% fewer decoder tokens.
• The text-token tolerance $m^\star$ (largest $m$ within 3 points of the text-only baseline) scales linearly with $k$, yielding a consistent compression ratio $\rho \approx 2$.
• Larger models (Qwen2.5-VL-72B) exhibit higher tolerance and more robust performance under compression than smaller models (Qwen2.5-VL-7B).
• Latency analysis shows that, for Qwen2.5-VL, the reduced decoder sequence yields up to 45% end-to-end speedup, despite modest vision processing overhead.

  Figure 5: Rendered image input for the Ruler task at context length 1500 ($600\times800$ image resolution), showing negligible accuracy degradation.

  

  Figure 6: Rendered image input at context length 2000 ($600\times1000$ image resolution), demonstrating scaling of resolution while preserving performance.

  

  Figure 7: Rendered image input at context length 2500 ($750\times1000$ image resolution), maintaining visual fidelity and model performance.

  

  Figure 8: Rendered image input at context length 3000 ($600\times1000$ image resolution), where accuracy degrades substantially, illustrating the tradeoff between token budget and image resolution.

Document Summarization (CNN/DailyMail)

The authors compare text-as-image compression against two token-pruning baselines: Select-Context and LLMLingua-2. At matched compression rates (retaining only ~40% of the original tokens), the text-as-image method outperforms both baselines on ROUGE and BERTScore metrics for both GPT-4.1-mini and Qwen2.5-VL-72B-Instruct. This demonstrates that visual text compression is competitive with, and orthogonal to, learned token-selection approaches.

Implementation Details

The text-to-image conversion pipeline (ConTexImage) uses adaptive font optimization to maximize fill ratio and preserve readability. Images are rendered at 300 dpi and resized to target resolutions (e.g., $600\times800$, $750\times1000$). The number of visual tokens $k$ is determined by the vision encoder architecture and image resolution. For open-weight models, inference is performed on multi-GPU setups using Hugging Face implementations; for proprietary models, API token statistics are used.

The approach is compatible with existing MLLMs and does not require model modification. It is particularly effective for tasks where the context is large and the query is short, and can be combined with token-level pruning for further gains.

Trade-offs and Limitations

• The compression ratio is bounded by the model's ability to process dense visual information; excessive compression leads to rapid accuracy degradation.
• Model scale is a critical factor: larger decoders are more robust to visual text density.
• Vision encoding adds overhead, but this is offset by reduced decoder sequence length in large models.
• The method is less effective for extremely long contexts (tens of thousands of tokens) without additional strategies (e.g., chunking, retrieval).
• Applicability to domains with critical token-level semantics (e.g., math, code) requires further investigation.

Implications and Future Directions

The findings suggest that modality shifting—offloading textual context to the vision encoder—offers a practical axis for scaling LLM efficiency. The approach is model- and task-agnostic, requires no parameter updates, and can reduce both computational cost and latency. Future work may explore hybrid strategies combining token pruning and visual rendering, extend the method to other domains, and investigate robustness across diverse tasks and architectures.

The results challenge the assumption that token-level compression is the only viable strategy for long-context efficiency, highlighting the potential of visual text inputs as an implicit compression layer in MLLMs.

Conclusion

Rendering textual context as images for multimodal LLMs enables nearly two-fold reductions in decoder token count while preserving task accuracy on both retrieval and generation benchmarks. The approach is simple, effective, and complementary to existing compression methods. The observed $\tfrac{1}{2}$ allocation between text and visual tokens provides a practical heuristic for balancing efficiency and fidelity. This work opens new avenues for modality-based context compression and efficient LLM deployment in long-context scenarios.