SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion (2503.11576v1)

Abstract: We introduce SmolDocling, an ultra-compact vision-LLM targeting end-to-end document conversion. Our model comprehensively processes entire pages by generating DocTags, a new universal markup format that captures all page elements in their full context with location. Unlike existing approaches that rely on large foundational models, or ensemble solutions that rely on handcrafted pipelines of multiple specialized models, SmolDocling offers an end-to-end conversion for accurately capturing content, structure and spatial location of document elements in a 256M parameters vision-LLM. SmolDocling exhibits robust performance in correctly reproducing document features such as code listings, tables, equations, charts, lists, and more across a diverse range of document types including business documents, academic papers, technical reports, patents, and forms -- significantly extending beyond the commonly observed focus on scientific papers. Additionally, we contribute novel publicly sourced datasets for charts, tables, equations, and code recognition. Experimental results demonstrate that SmolDocling competes with other Vision LLMs that are up to 27 times larger in size, while reducing computational requirements substantially. The model is currently available, datasets will be publicly available soon.

• The paper introduces SmolDocling, an ultra-compact (256M parameter) vision-language model leveraging a unified DocTags representation to achieve state-of-the-art end-to-end multi-modal document conversion efficiency.
• SmolDocling demonstrates impressive performance benchmarks, including converting a page in 0.35 seconds with only 0.489 GB VRAM on an A100, outperforming larger models in resource efficiency.
• Its low parameter count and efficient inference make SmolDocling highly practical for deployment in resource-constrained environments and scalable across various document types.

Overview

“SmolDocling: An ultra-compact vision-LLM for end-to-end multi-modal document conversion” (2503.11576) presents a refined system architecture designed to address extensive document conversion challenges within a compact parameter regime. The approach leverages a unified representation – DocTags – which encodes content, layout, and spatial semantics of document elements while maintaining a tractable 256M parameter footprint. This work demonstrates that a carefully engineered end-to-end VLM can match or exceed the performance of larger models typically deployed in document understanding tasks.

Model Architecture and Design Choices

The core of SmolDocling comprises two complementary components. A visual encoder based on SigLIP Base Patch-16/512 (approximately 93M parameters) serves the task of image tokenization. This encoder employs an aggressive pixel shuffle algorithm, compressing high-resolution patches (512×512) into sequences of 64 visual tokens. This yields a high pixel-to-token ratio (4096 pixels per token) and minimizes redundancy in image features.

Concurrently, a trimmed version from the SmoLLM-2 family (around 135M parameters) functions as the language backbone. By integrating these modules, the model establishes a joint vision-language representation space wherein text embeddings are seamlessly concatenated with projected visual embeddings. These combined sequences are subsequently processed by an autoregressive transformer, outputting the DocTags sequence that captures a diverse set of document elements including tables, code segments, equations, and charts.

The DocTags universal markup is particularly notable for its explicit demarcation of content and structure:

• Hierarchical Nesting: Enables encapsulation of spatial coordinates (bounding boxes) alongside content.
• Tokenization Efficiency: Special tokens delineate sub-image segments, reducing ambiguity in element boundaries.
• Structural Semantics: The vocabulary inherently supports the representation of complex elements (e.g., nested tables, multilevel lists) in a single-stage conversion framework.

Training Paradigm and Optimization Techniques

SmolDocling’s training regimen employs a curriculum-based strategy whereby the vision encoder is initially frozen to ensure stability in visual feature extraction, facilitating a smoother convergence during the subsequent joint training phase. The overall training pipeline integrates both pretraining on extensive document datasets and fine-tuning on task-specific corpora, achieving a robust alignment for document conversion tasks.

Key training optimizations include:

• Progressive Unfreezing: Gradual release of encoder weights to fine-tune the entire model ensemble.
• Loss Function Engineering: Tailored objectives combining reconstruction fidelity and layout consistency, ensuring that both textual content and spatial positioning are accurately modeled.
• Data Augmentation: In-house generated datasets for charts, tables, equations, and code have been utilized, addressing common ground truth variability issues inherent in multi-modal documents.

Experimental Results and Comparative Analysis

The performance benchmarks reported for SmolDocling are compelling, particularly considering its compact size relative to other state-of-the-art models. Noteworthy experimental highlights include:

• Competitive Layout Analysis: On full-page transcription tasks, SmolDocling achieved a page conversion time of 0.35 seconds with VRAM consumption of only 0.489 GB on an A100 GPU, a significant reduction in computational overhead compared to models such as Qwen2.5-VL (7B parameters) and GOT (580M parameters).
• Robust Recognition Capabilities: The model demonstrated strong performance in text recognition, table structure extraction, and code parsing. Bold numerical claims indicate that despite the low parameter count, the accuracy in reproducing document features, particularly in the spatial localization of elements, remains uncompromised.
• Generalization Across Document Types: The methodology efficiently extends to a variety of document formats including business documents, academic journals, technical reports, patents, and forms. This breadth of applicability is reinforced by the reported metrics across diverse datasets and tasks.

The efficient integration of a unified DocTags framework allows the model to bypass the demerits of traditional ensemble architecture, where errors typically accumulate due to the cascading effect of multiple specialized modules.

Practical Implications and Deployment Considerations

From a deployment standpoint, SmolDocling offers practical benefits for resource-constrained environments due to its low parameter count and reduced VRAM requirements:

• Inference Efficiency: The low-latency conversion (0.35 seconds per page) confirms its suitability for online document processing pipelines.
• Scalability: The model’s architecture supports scaling across varying document resolutions and complexities without significant degradation in performance.
• End-to-End Conversion: The unified approach eliminates the need for additional post-processing steps commonly encountered in multi-stage pipelines, streamlining integration into end-user applications.
• Dataset Availability: The forthcoming public release of newly curated datasets for charts, tables, equations, and code recognition will facilitate further research and fine-tuning specific to niche application domains.

In practical implementation, integrating SmolDocling into an existing document processing system would involve:

1. Preprocessing Pipeline Setup: Converting documents (PDFs, scans) into rasterized inputs compatible with the visual encoder.
2. Model Integration: Adapting inference pipelines to process batches of document pages, leveraging the presented encoder-decoder structure in a cross-modal setting.
3. Post-processing: Utilizing the DocTags to reconstruct structured documents in desired formats (e.g., HTML, XML) while retaining spatial accuracy.
4. Resource Allocation: Provisioning GPU resources similar to an A100 setup to mirror reported conversion times or exploring quantization techniques for edge deployments.

Conclusion

SmolDocling epitomizes a well-engineered balance between model compactness and performance efficiency. Its innovative design, embodied in the hierarchical DocTags framework and aggressive tokenization strategy, demonstrates that ultra-compact VLMs can effectively capture multi-modal document features. The reported experimental metrics and architectural refinements position SmolDocling as a viable alternative for real-world document conversion tasks, especially in scenarios demanding high efficiency without sacrificing detailed layout and content representation.