DeepEncoder: Vision Token Compression

• DeepEncoder is a high-resolution image-to-token compressor that maintains up to 97% OCR precision at 10× compression.
• It combines local window self-attention with global dense attention and a convolutional compressor for efficient, low-memory processing.
• The architecture scales advanced OCR and multimodal dataset creation while outperforming legacy systems in token efficiency.

DeepEncoder refers to the core vision-encoding engine within DeepSeek-OCR, a system designed to compress long document contexts by mapping high-resolution document images into a highly compressed set of vision tokens, which are then decoded into text by a LLM decoder (DeepSeek-3B-MoE). DeepEncoder's design focuses on information preservation under extreme compression and low activation memory, enabling the system to process and represent lengthy contexts efficiently for applications such as optical character recognition (OCR) and large-scale document modeling (Wei et al., 21 Oct 2025).

1. Design and Functionality

DeepEncoder operates as a high-resolution image-to-token compressor, acting as the primary entry point from 2D visual content to a condensed set of vision tokens suitable for context modeling and subsequent text decoding. The encoder’s architecture is optimized for:

• High-resolution input processing: Utilizing window attention for local spatial perception to manage the computational cost of high input resolutions.
• Compression: A 16× downsampling convolutional compressor reduces the token count, with an initial phase extracting patch tokens and a secondary phase compressing those tokens while maintaining informative content.
• Low activation overhead: The design ensures computational and memory demands remain tractable, even for 1024×1024 inputs, by compressing, for instance, 4096 initial patch tokens down to 256 latent vision tokens before global encoding.

DeepEncoder combines local window self-attention (as in SAM-base) for the initial visual feature extraction with global dense attention (as in CLIP-large) for semantic encoding, bridged by convolutional layers dedicated to hierarchical token downsampling.

2. Technical Architecture

DeepEncoder is derived from a composition of two large vision transformer architectures and a novel convolutional token compressor:

• Window self-attention (SAM-base): An 80M parameter model that divides the image into fixed-size patches (patch size 16 for 1024×1024 images, resulting in 4096 patch tokens), extracting local features efficiently.
• Convolutional compressor: Two convolutional layers (kernel size 3, stride 2, padding 1) upscale channel dimension from 256 to 1024 while spatially downsampling, effectively reducing the token sequence by a factor of 16 (e.g., 4096 → 256 tokens).
• Global attention (CLIP-large): A dense, 300M parameter vision transformer module further encodes the compressed token sequence. The initial patch embedding stage is omitted since patchification is already handled upstream.

Dynamic resolution modes (Tiny, Small, Base, Large, and "Gundam") offer adaptive support for various input sizes and compression needs, using interpolated positional encodings and padding schemes. For full-page extraction and variable aspect ratios, token validity is adjusted as:

$$N_{valid} = \left\lceil N_{actual} \cdot \bigg[1-\frac{\max(w, h)-\min(w, h)}{\max(w, h)}\bigg] \right\rceil$$

where $w$ and $h$ are image width and height, respectively.

3. Compression and OCR Performance

Compression ratio is central to DeepEncoder’s design:

• 10× compression: Retains roughly 97% OCR decoding precision. For text-intensive pages where the number of text tokens does not exceed 10× the number of vision tokens, information loss is minimal.
• 20× compression: Maintains approximately 60% OCR accuracy, indicating resilience to aggressive token downsampling.
• Benchmark results: On OmniDocBench, DeepSeek-OCR equipped with DeepEncoder outperforms prior state-of-the-art OCR systems (e.g., GOT-OCR2.0), achieving higher precision with fewer vision tokens. With only 100 vision tokens, it surpasses models using 256 or more tokens per page, and with fewer than 800 vision tokens, it exceeds the performance of OCR systems handling 6000+ tokens per page.

These metrics are reported for both standard OCR tasks and long-context document modeling scenarios.

4. Applications and Integrative Implications

The DeepEncoder architecture enables several advanced use cases:

• Historical Long-Context Compression: Efficiently compresses and reconstructs large-scale, content-rich document collections and archives.
• Memory and Forgetting in LLMs: Progressive image-size reduction (which degrades from center to edge) mirrors biological memory decay, suggesting its potential for memory management in unlimited-context LLMs.
• Training Data Generation: High-throughput extraction for LLM/VLM pretraining, with a production rate exceeding 200,000 pages per day on an A100-40G GPU, supporting at-scale multimodal dataset creation.
• Vision-LLM optimization: Lower token count per page reduces both memory and computation during VLM training and inference, facilitating OCR, visual retrieval, and context understanding across large document corpora.

5. Future Research and Development

Several avenues for improvement and exploration are identified:

• Compression Optimization: Efforts to approach near-lossless 10× compression by refining the encoder and pre/post-processing pipeline.
• Digital-Optical Interleaved Modeling: Extending beyond OCR to co-train on both digital (raw text) and optical (image-based) representations, supporting hybrid document understanding.
• Vision Token Allocation: Fine-tuning the allocation of vision tokens across documents and within pages for improved efficiency and fidelity.
• Dynamic Memory/Forgetting: Investigating how adaptive image compression and resolution control relate to memory decay models in LLMs, with applications to unlimited-context management and information prioritization.

6. Experimental Throughput and Production Considerations

DeepEncoder is architected for practical deployment:

• Hardware efficiency: Low activation memory and efficient tokenization allow rapid parallel processing.
• Production-scale throughput: Enables the extraction of vision tokens from hundreds of thousands of pages per day on commodity GPU infrastructure.
• Accessibility: Model weights and code are publicly released, facilitating further research and production use.

---

DeepEncoder, as the main component of DeepSeek-OCR’s context optical compression (Wei et al., 21 Oct 2025), exemplifies a high-capacity vision encoder optimized for precise, efficient context compression in document modeling and large-scale language-vision pretraining systems. Its blend of hierarchical attention mechanisms, aggressive convolutional token compression, and performance-oriented design provides promising directions for further advances in scalable vision-language modeling, OCR, and long-context document understanding.