ModernVBERT: Towards Smaller Visual Document Retrievers (2510.01149v1)

Abstract: Multimodal embedding models are gaining prevalence, notably for document retrieval as efficient alternatives to text-only pipelines. These models are typically built by finetuning large vision-language decoders (VLMs) with contrastive losses on text-image pairs. In this work, we show that, while cost-efficient, this repurposing approach often bottlenecks retrieval performance. Through controlled experiments, we establish a principled recipe for improving visual document retrieval models. We notably measure the impact of attention masking, image resolution, modality alignment data regimes, and late interaction centered contrastive objectives which emerge as central performance factors. Building on these insights, we release ModernVBERT, a compact 250M-parameter vision-language encoder that outperforms models up to 10 times larger when finetuned on document retrieval tasks. Models and code are made available at https://huggingface.co/ModernVBERT.

• The paper proposes a novel early fusion architecture that integrates patch-level visual cues with a bidirectional language model for enhanced cross-modal retrieval.
• It demonstrates that bidirectional attention with late interaction scoring yields a +10.6 nDCG@5 improvement over causal models on ViDoRe benchmarks.
• The compact 250M-parameter ModernVBERT and its variant ColModernVBERT achieve high performance and efficiency, enabling deployment on commodity CPUs.

ModernVBERT: Towards Smaller Visual Document Retrievers

Introduction

ModernVBERT addresses the efficiency and effectiveness of visual document retrieval, a domain where multimodal embedding models are increasingly replacing text-only pipelines. The paper systematically investigates the design choices that impact retrieval performance, focusing on attention masking, image resolution, modality alignment, and contrastive objectives. The authors introduce ModernVBERT, a 250M-parameter vision-language encoder, and its document retrieval-specialized variant ColModernVBERT, which outperforms models up to 10x larger on standard benchmarks. This work provides a rigorous empirical foundation for building compact, high-performance visual document retrievers and offers practical recipes for model training and deployment.

Model Architecture and Training Paradigm

ModernVBERT employs an early fusion architecture, where patch-level visual representations from a vision encoder are projected into the input space of a bidirectional LLM. This enables joint processing of image and text tokens, facilitating fine-grained cross-modal interactions.

Figure 1: MLM-based early fusion architecture. The visual encoder produces patch representations, which are passed to a LLM. The end-to-end bidirectional attention fused architecture is trained with Masked LLMing objectives and is suited for sequence and token-level representation tasks.

The training pipeline is biphasic:

1. Modality Alignment: The vision encoder is aligned with a pretrained LLM using Masked LLMing (MLM) or Causal LLMing (CLM) objectives, depending on the attention mask. LoRA adapters are used for parameter-efficient adaptation.
2. Contrastive Post-Training: The model is further specialized for retrieval via contrastive learning (InfoNCE loss) on large-scale document-query and image-caption pairs, with task-aware batching and late interaction scoring.

The vision encoder is a SigLIP2-base-16b-512 model, and the LLM backbone is a 150M-parameter bidirectional encoder (ModernBERT). Images are processed at up to 2048px resolution, with pixel shuffling and patch concatenation to maximize information density.

Empirical Analysis of Design Choices

Attention Masking and Training Objectives

The paper demonstrates that bidirectional attention (MLM) is superior to causal attention (CLM) for document retrieval, especially when combined with late interaction (multi-vector) scoring. While single-vector performance is similar between MLM and CLM, late interaction with bidirectional attention yields a substantial +10.6 nDCG@5 improvement on ViDoRe benchmarks.

Figure 2: Impact of attention masks and training objectives on document retrieval performances. Average nDCG@5 on ViDoRe benchmarks for models post-trained on ColPali.

Removing the causal mask post-hoc does not recover the performance of natively bidirectional models, indicating that encoder-style pretraining is essential for optimal retrieval.

Modality Alignment Scaling

Scaling the modality alignment phase (i.e., increasing the number of aligned tokens) consistently improves document retrieval, with performance quickly surpassing the standalone vision encoder. However, natural image retrieval and classification plateau early, indicating that document and natural image tasks benefit from different mechanisms.

Figure 3: Modality alignment scaling of early fusion encoders. Retrieval tasks benefit from extended modality alignment, especially in document retrieval.

Image Resolution

Higher image resolutions during modality alignment and contrastive training significantly boost document retrieval performance, with diminishing or negative returns for natural image tasks. A "cooldown" phase with high-resolution images at the end of alignment further improves results.

Contrastive Training Mix and Data Regimes

Incorporating text-only pairs into the contrastive mix yields clear cross-modal transfer, improving document retrieval by +1.7 nDCG@5. Adding high-quality synthetic image-caption pairs (NatCap) further enhances generalization, especially for natural image tasks. Increasing the text-to-image ratio in the contrastive mix consistently benefits retrieval.

Figure 4: Optimal text-to-image ratio in contrastive training mix. Increasing the ratio in retrieval tasks consistently improves performance.

Scaling the volume of in-distribution contrastive pairs continues to improve retrieval, but gains plateau for captioning and classification, highlighting the need for diverse data.

Figure 5: Contrastive training scaling. Performance improves with more in-distribution data, surpassing the baseline on document benchmarks and narrowing the gap on image captioning.

ModernVBERT and ColModernVBERT: Implementation and Results

Training Recipe

• Backbone: ModernBERT (150M, bidirectional) + SigLIP2-16B-512 (100M)
• Modality Alignment: MLM objective, 10B tokens, with a 2048px resolution cooldown
• Contrastive Training: Document-query and text-only pairs, 2:1 text-to-image ratio, hard negatives, late interaction scoring

Performance

ColModernVBERT achieves 81.2/56.0 nDCG@5 on ViDoRe(v1/v2), matching or exceeding models 10x larger (e.g., ColPali, ColQwen2.5) and outperforming all other sub-1B parameter models by a wide margin. Inference latency is an order of magnitude lower than larger models, enabling CPU deployment.

(Figure 1 from the paper, not shown here, demonstrates the Pareto efficiency of ColModernVBERT.)

Ablations and Generalization

• Late interaction benefits extend to non-documental retrieval tasks, with multi-vector models outperforming single-vector and dual-encoder baselines.
• Model merging (e.g., SLERP) can mitigate performance drops on some tasks but degrades document retrieval, indicating task-dependent trade-offs.
• ColModernVBERT achieves competitive text-only retrieval performance, suggesting potential for unified multimodal retrieval.

Practical Implications and Deployment Considerations

• Resource Efficiency: The 250M-parameter footprint enables deployment on commodity CPUs, reducing inference cost and environmental impact.
• Data Efficiency: Cross-modal transfer from text-only data reduces reliance on expensive document-image pairs.
• Task Specialization: Bidirectional attention and late interaction are critical for document retrieval; causal decoders and single-vector models are suboptimal.
• Scalability: The observed trends are robust at the sub-1B scale; future work should validate scaling laws at larger model sizes and in multilingual settings.

Limitations and Future Directions

• The analysis is limited to English and relatively small models; scaling to multilingual and larger architectures remains open.
• The focus is on retrieval and zero-shot classification; the modality-aligned encoder could be fine-tuned for token-level tasks (e.g., OCR error detection, visual NER).
• Further research is needed on efficient model merging, quantization, and index compression for large-scale deployment.

Conclusion

ModernVBERT provides a principled, empirically validated recipe for building compact, high-performance visual document retrievers. The synergy of bidirectional attention, early fusion, high-resolution alignment, and late interaction enables ColModernVBERT to match or exceed the performance of much larger models at a fraction of the computational cost. The findings challenge the prevailing reliance on large, decoder-based VLMs for retrieval and establish new baselines for efficient multimodal embedding. The released models and codebase facilitate reproducibility and further research on efficient, scalable, and versatile vision-language retrieval systems.