ModernVBERT: Compact Vision–Language Encoder

• ModernVBERT is a compact vision–language encoder that leverages bidirectional attention masking to enhance document retrieval performance.
• It systematically investigates architectural choices such as image resolution handling, modality alignment with MLM, and contrastive losses to optimize embedding quality.
• Empirical results demonstrate that the 250M-parameter model outperforms much larger models in retrieval accuracy, highlighting efficient parameter usage.

ModernVBERT is a compact vision–language encoder model specialized for visual document retrieval, addressing the limitations of contemporary multimodal embedding approaches by systematically investigating the impact of architectural and training choices on retrieval effectiveness. Developed as a 250M-parameter model, ModernVBERT is empirically shown to outperform vision–LLMs up to ten times larger when finetuned for document-level retrieval tasks. The model facilitates efficient and accurate retrieval in settings where both textual and visual information are present, which is essential in domains such as digital archiving, enterprise document management, and legal or scientific literature search.

1. Architectural Overview

ModernVBERT departs from prevalent vision–language embedding paradigms that repurpose large Vision-LLM (VLM) decoders, typically optimized for generative objectives, for retrieval via contrastive finetuning. The model is constructed explicitly as an encoder-only transformer, integrating both visual (image) and textual modalities into a shared embedding space. Key architectural features include:

• Bidirectional Attention Masking: ModernVBERT employs bidirectional self-attention within the transformer layers, in contrast to causal masks common in autoregressive decoders. This design is empirically linked to improved information exchange between tokens, crucial for dense retrieval tasks where full context is needed.
• Image Resolution Handling: The architecture allows flexible handling of input image resolutions, facilitating ablation studies on the influence of image granularity on embedding quality.
• Parameter Efficiency: With 250M parameters, ModernVBERT emphasizes architectural compactness while achieving top-tier retriever performance, suggesting superior parameter utilization over much larger VLMs.

2. Attention Masking and Retrieval Performance

A central experimental focus of ModernVBERT is the systematic analysis of attention masking strategies. The following masking regimes are contrasted:

• Bidirectional Attention: All tokens (textual and visual) can attend to each other, enabling full context integration.
• Causal (Unidirectional) Attention: Enforces autoregressive dependencies, which, while beneficial for generation, impede performance in encoding-based retrieval tasks.

Experimental results demonstrate that bidirectional attention yields marked improvements in document retrieval accuracy, reflecting better global context modeling. This finding establishes an evidence-based preference for bidirectional masking in visual document embedding architectures.

3. Data Regimes, Modality Alignment, and Objectives

ModernVBERT’s design process rigorously benchmarks the effects of:

• Alignment Data Regimes: The quantity and nature of paired image–text data used for pretraining and finetuning directly affect modality alignment. Controlled experiments show that regime selection is pivotal for retrieval effectiveness.
• Masked LLMing as Modality Alignment: Employing masked LLMing (MLM) facilitates robust alignment between textual and visual representations, bolstering semantic coherence in the joint embedding space.
• Contrastive Losses and Late Interaction: The model leverages contrastive losses with late interaction mechanisms, where representations are aligned at higher layers rather than early in processing. This yields superior matching between document-level visual and textual cues.

These factors collectively contribute to improved robustness and generalizability across heterogeneous document types and imaging conditions.

4. Image Resolution and Embedding Fidelity

An explicit empirical variable in ModernVBERT is the input image resolution. The controlled manipulation of resolution demonstrates its nontrivial, task-dependent effect on retrieval performance. Lower image resolutions may bottleneck visual signal encoding, while excessively high resolutions impose computational costs with diminishing returns in retrieval accuracy.

• Resolution Selection: The optimal trade-off balances embedding fidelity and efficiency, dependent on the distribution of document layouts and visual complexity in the target corpus.

5. Performance and Comparative Evaluation

ModernVBERT is benchmarked against prevailing vision–LLMs, including those an order of magnitude larger in parameter count. When finetuned for document retrieval tasks, ModernVBERT exhibits superior ranking and recall metrics, thereby challenging the presumption that only large-scale VLMs excel in retrieval. The released models and code, accessible at https://huggingface.co/ModernVBERT, enable replication and further assessment in diverse retrieval settings.

<Table> | Aspect | ModernVBERT | Large VLM Decoders | |--------------------------|----------------------------|-------------------------| | Params | 250M | 2B–3B+ | | Attention Masking | Bidirectional | Often causal | | Modality Alignment | MLM & Contrastive late-inter| Contrastive early/mixed | | Retrieval Performance | State-of-the-art (per size)| Often lower (per size) | </Table>

6. Scope and Exclusions

ModernVBERT is strictly focused on multimodal embedding and retrieval. The model and its associated research do not address techniques from computer graphics or animation—such as Bézier curves, B-splines, in-betweening, quaternion-based rotation, spherical geometry, or interpolative spline approximations. Its design and analysis are solely concerned with the alignment and embedding of vision and text data for information retrieval applications. Any inference of overlap with geometric or animation-related interpolation methods would be incorrect; ModernVBERT does not reference or apply these techniques.

7. Contextual Significance and Implications

ModernVBERT provides a data-driven blueprint for constructing compact yet high-performing vision–language retrievers, emphasizing the importance of architectural choices tailored to retrieval rather than generation. Its empirical findings on attention masking and modality alignment offer actionable guidelines for subsequent multimodal model development, particularly under resource constraints. While not extending to animation, graphics, or geometric modeling, ModernVBERT serves as a foundational work in the optimization of retrieval-centric multimodal architectures (Teiletche et al., 1 Oct 2025). A plausible implication is that further research in parameter-efficient multimodal encoders may benefit from similar systematic ablations along the lines of attention, objective selection, and input processing regimes.