SeeTok: Vision-Centric Tokenization for LLMs

• SeeTok is a vision-centric tokenization paradigm that renders text as images, leveraging OCR-equipped multimodal LLMs to integrate visual-text representations.
• It compresses text inputs substantially, reducing token count and computational cost while improving performance on benchmarks like TriviaQA.
• The approach enhances robustness to typographical noise and supports multilingual generalization, proving effective for both high- and low-resource languages.

SeeTok is a vision-centric tokenization paradigm for LLMs that replaces conventional subword tokenization with human-inspired visual reading. Instead of segmenting text strings into symbolic units from a fixed vocabulary, SeeTok renders text as images and leverages pretrained multimodal LLMs—typically equipped with strong optical character recognition (OCR) and cross-modal alignment capabilities—to interpret this visual-text, providing a highly compressed and robust input representation. Developed as a challenge to the dominant subword paradigm, SeeTok demonstrates state-of-the-art efficiency, improved robustness, and enhanced cross-lingual generalization, particularly in low-resource languages (Xing et al., 21 Oct 2025).

1. Motivation for Vision-Centric Tokenization

Modern LLMs almost universally use subword tokenization algorithms such as Byte-Pair Encoding (BPE) or WordPiece. While effective for high-resource languages, this approach results in over-segmentation and diminished semantic coherence in low-resource languages, inflating sequence lengths and driving quadratic self-attention costs. Furthermore, subword tokenization discards continuous visual and morphological cues, making models brittle to orthographic variation, typographical noise, or script variants.

SeeTok’s foundation is the observation that human reading, as established in cognitive neuroscience, is fundamentally visual: textual information is first processed in the Visual Word Form Area (VWFA) of the brain, which parses word shapes and letter patterns before semantic interpretation. Empirical phenomena such as “typoglycemia” (robustness to letter scrambling) underscore the holistic and shape-driven nature of human word recognition. SeeTok’s design thus aims to replicate these cognitive mechanisms within LLMs, moving from symbolic to visually grounded input representations (Xing et al., 21 Oct 2025).

2. SeeTok Pipeline: Visual Rendering to LLM Decoding

The SeeTok pipeline comprises three stages that mimic the human reading process:

a) Visual Rendering:

Text strings are rendered into RGB images, typically size $224\times224$ pixels, using fonts such as Google Noto Sans. One image may encode dozens of words, adapting to the typographic and script density of the input language.

b) Vision-Centric Tokenization:

Rendered images are fed into a pretrained vision encoder (such as Qwen2.5-VL or JanusPro) which divides the image into patches (e.g., $14\times14$), extracts patch embeddings, and groups every four adjacent patches via a two-layer MLP. This “projector” step reduces sequence length by a factor of four while projecting patch features into the embedding space expected by the LLM’s text tower. The pipeline’s formal tokenization computation is: $$t_j = \sigma\big(W_2\,\sigma(W_1\,[\,v_{4j-3};\,v_{4j-2};\,v_{4j-1};\,v_{4j}\,])\big),\quad j=1,\dots,N_{\text{patches}}/4,$$ where $v_i$ are patch embeddings, and $t_j$ are the final “visual tokens".

c) Multimodal LLM Decoding:

These visual tokens replace subword embeddings in the downstream transformer layers, thereby integrating visual-text directly into the model’s reasoning flow. The vision encoder, projector, and language backbone are pretrained for cross-modal alignment on massive image–text corpora.

3. Training and Cross-Modal Alignment

Instruction Tuning with LoRA:

Although pretrained multimodal LLMs show strong OCR ability, they do not generalize to instruction following with visual-text without further adaptation. SeeTok inserts low-rank adapters (LoRA, rank $r=8$, $\alpha=32$) into the vision and language components (while freezing the projector) and instruction-tunes the model using rendered instruction–response corpora.

Autoregressive Loss:

Given rendered image instructions $\mathcal{X}_{\text{img}}$ and target token sequence $\mathbf{y} = (y_1, \dots, y_T)$, SeeTok is trained with the standard next-token likelihood: $$\mathcal{L} = -\sum_{t=1}^T \log P(y_t|y_{<t}, \mathcal{X}_{\text{img}})$$

Cross-Modal Procrustes Alignment:

To quantify how well visual-text and plain-text activations align after tuning, the paper employs orthogonal Procrustes analysis on intermediate representations. Smaller residuals in deeper layers post-instruction tuning indicate tighter cross-modal alignment between image-derived and text-derived activations.

4. Efficiency, Compression, and Computational Savings

SeeTok yields substantial reductions in both token counts and computational cost relative to traditional subword tokenization. For example, on TriviaQA:

• Compression Ratio: $14\times14$0
• Latency: Reduced from 5.02 s to 3.34 s (−33.5%)
• FLOPs: Reduced from 3.12 TFLOPS to 0.92 TFLOPS (−70.5%)

Across 14 languages, compression ratios are $14\times14$1 for high-resource languages and $14\times14$2 for low-resource languages. As transformer memory and attention have $14\times14$3 and $14\times14$4 costs respectively with respect to sequence length, these reductions result in near-linear computational savings proportional to the token count compression (Xing et al., 21 Oct 2025).

5. Empirical Results: Language Understanding, Multilingual Transfer, and Robustness

Language Understanding:

On benchmark QA and reasoning tasks (e.g., TriviaQA, NaturalQuestions, MMLU), SeeTok matches or slightly surpasses subword-token baselines on most metrics. For example, on TriviaQA, SeeTok achieves EM = 43.53 compared to the subword baseline EM = 41.92. Notably, visual-text alone (without SeeTok’s projection mechanism) exhibits a performance collapse, highlighting the necessity of the full SeeTok pipeline.

Multilingual Translation:

SeeTok yields superior COMET-22 and fertility metrics for both high- and low-resource to English translation, e.g., COMET = 65.17 (vs. 60.71) and 83–88% reduction in tokens per word for low-resource languages.

Surface Noise Robustness:

SeeTok exhibits significantly greater robustness to both character-level (e.g., shuffling, glyph swaps) and word-level surface perturbations, with accuracy drops only 5–10 points (vs. subword’s 15–30), leveraging holistic visual encoding to overcome typographic noise.

6. Ablation Studies and Design Considerations

• Fine-Tuning Scope: Tuning only the vision encoder and LLM (freezing the projector) yields optimal performance, likely due to improved stability of pretrained cross-modal alignment.
• Backbone Generality: SeeTok outperforms visual-only prompts in larger backbones (e.g., JanusPro, Qwen2.5-VL 7B), recovering or exceeding pure-text baselines on QA.
• Scaling and Data Requirements: The approach is robust to low-data regimes, yielding material gains on only 9K instruction–answer pairs.

Table: Comparative Token and FLOP Compression

Language Class	Compression Ratio (Δ)	FLOP Reduction (%)
High-resource	5.71×	~70.5%
Low-resource	7.85×	~70.5%

7. Limitations, Cognitive Underpinnings, and Future Directions

Limitations:

SeeTok currently depends on pretrained vision encoders with inherited OCR biases, potentially limiting generalization to out-of-domain typographies or scripts. On highly knowledge-intensive benchmarks (e.g., MMLU), SeeTok still underperforms models with extensive pure-text pretraining, reflecting a domain gap in visual-text learning resources.

Cognitive and Linguistic Parallels:

SeeTok’s design emulates the VWFA–language cortex pathway, allowing transformer-based LLMs to leverage whole-word shape cues, capturing morphological hierarchies and compositionality, and providing resilience to word-internal variance—analogous to human psycholinguistic processing.

Future Research:

Recommendations include dedicated pretraining on both image and visual-text to address knowledge transfer, exploration of unified vision-centric architectures (e.g., CLIPPO), and dynamic patch grouping or learned visual "vocabularies" to further reduce sequence length. These directions signal a potential shift in foundational LLM architecture away from purely symbolic tokenization toward multimodal, cognitively inspired input processing.

---

SeeTok establishes a vision-based alternative to symbolic tokenization in LLMs, grounded in both cognitive neuroscience and empirical machine learning efficiency. By rendering text as images and feeding visual tokens to transformer models, SeeTok achieves drastic compression, robust multilingual generalization, and resilience to typographic noise, laying the groundwork for human-like, vision-integrated LLMs (Xing et al., 21 Oct 2025).