M3DR: Towards Universal Multilingual Multimodal Document Retrieval (2512.03514v1)

Abstract: Multimodal document retrieval systems have shown strong progress in aligning visual and textual content for semantic search. However, most existing approaches remain heavily English-centric, limiting their effectiveness in multilingual contexts. In this work, we present M3DR (Multilingual Multimodal Document Retrieval), a framework designed to bridge this gap across languages, enabling applicability across diverse linguistic and cultural contexts. M3DR leverages synthetic multilingual document data and generalizes across different vision-language architectures and model sizes, enabling robust cross-lingual and cross-modal alignment. Using contrastive training, our models learn unified representations for text and document images that transfer effectively across languages. We validate this capability on 22 typologically diverse languages, demonstrating consistent performance and adaptability across linguistic and script variations. We further introduce a comprehensive benchmark that captures real-world multilingual scenarios, evaluating models under monolingual, multilingual, and mixed-language settings. M3DR generalizes across both single dense vector and ColBERT-style token-level multi-vector retrieval paradigms. Our models, NetraEmbed and ColNetraEmbed achieve state-of-the-art performance with ~150% relative improvements on cross-lingual retrieval.

• The paper introduces M3DR, which synthesizes large-scale parallel corpora and novel retrieval architectures to support 22 languages and diverse scripts.
• It details the deployment of both dense and ColBERT-style models with a Matryoshka embedding mechanism that balances storage, speed, and accuracy.
• Experimental results demonstrate significant improvements in cross-lingual and monolingual NDCG@5, validating its practical and theoretical contributions.

Universal Multilingual Multimodal Document Retrieval: A Technical Analysis of M3DR (2512.03514)

Motivation and Research Context

The surge in digital documents worldwide has fostered urgent demand for document retrieval systems that natively support numerous languages and modalities. Existing vision-language retrieval pipelines achieve state-of-the-art performance on English-centric tasks but demonstrate catastrophic degradation when confronting content in non-English scripts and layouts. This is principally due to their reliance on OCR-based preprocessing, English-heavy pretraining, and insufficient cross-lingual model adaptations. The M3DR framework comprehensively tackles universal multilingual multimodal document retrieval by synthesizing scalable training data, novel retrieval architectures, and rigorous benchmarking across 22 typologically diverse languages.

Framework Overview and Data Synthesis

M3DR is predicated on large-scale synthetic parallel corpora that rigorously encode both visual and textual document features. The pipeline comprises three pillars: layout-aware document translation, authentic visual rendering with language-specific typography, and diversified VLM-generated query synthesis, enabling training over $\sim$1M document-query pairs spanning Latin, Devanagari, Dravidian, CJK, and Arabic scripts.

Figure 1: The M3DR framework, encompassing synthetic data generation, multilingual neural translation, visual rendering, query synthesis, model training, and cross-modal document retrieval.

For benchmarking, the Nayana-IR suite is introduced; it consists of 23 datasets supporting evaluation in cross-lingual, monolingual, and mixed-language regimes, compatible with BEIR formats. The linguistic diversity and precise layout preservation allow for systematized paper of cross-script retrieval robustness—an unmet requirement in prior benchmarks.

Retrieval Architectures and Matryoshka Embedding

M3DR generalizes across two principal retrieval paradigms: single dense vector ("bi-encoder") models and ColBERT-style multi-vector models, each supporting flexible backbone scaling from 256M to 4B parameters. Dense models process document images and queries to fixed-size vectors ($h = 2560$ for the Gemma3 4B backbone), leveraging last token pooling and L2-normalized cosine similarity for efficient nearest neighbor search.

Matryoshka representation learning is a key innovation, enabling embeddings to be truncated (768, 1536, 2560 dimensions) at deployment, maintaining near-maximal performance with substantial storage reduction.

Figure 2: NetraEmbed workflow: Document indexing into shared semantic vectors, cross-lingual query processing, and semantic matching across languages/scripts.

ColBERT-style variants retain token-level embeddings and apply late interaction via MaxSim, tailoring semantic matching for fine-grained correspondence at the cost of storage and inference speed. Empirical findings show that dense bi-encoder models provide optimal cross-lingual accuracy-efficiency trade-offs.

Training Objectives and Strategy

Contrastive learning with the InfoNCE loss is used pervasively, with ablations demonstrating that simple in-batch negative sampling is markedly superior to document-level or hard negative mining, especially in high-diversity multilingual settings. Explicit hybrid and pairwise losses yield instability without metric improvement. Last token pooling in decoder-based VLMs outperforms mean pooling by a wide margin.

Figure 3: Comparison showing in-batch negative (positive-only) training yields better, stabler convergence than more complex negative mining strategies.

Experimental Outcomes and Benchmarking

NetraEmbed, a single-vector dense model, achieved a 0.716 NDCG@5 on cross-lingual document retrieval and 0.738 NDCG@5 for monolingual tasks, representing 152% and 80% relative improvements over ColPali-v1.3, the strongest prior system. This high cross-lingual accuracy is preserved without sacrificing monolingual (e.g., English) competitiveness (0.554 NDCG@5 on ViDoRe v2).

Figure 4: Consistent retrieval performance for NetraEmbed across all 22 languages and script families; baseline models deteriorate sharply outside English.

Critical ablations reveal that scaling from 6 to 22 languages results in robust gains in accuracy, and embedding dimension truncation (Matryoshka) enables up to 70% storage savings with only a $\sim$5% accuracy penalty.

Figure 5: Graceful degradation of NDCG@5 performance as Matryoshka embedding dimensionality decreases.

Dense vector architectures outperform ColBERT-style multi-vector models by 10+ points on cross-lingual NDCG@5 while maintaining $\sim$10$\times$ smaller storage and $\sim$10$\times$ higher retrieval speed.

Analysis of Multilingual Alignment and Model Behavior

PCA analysis of embedding evolution demonstrates progressive cross-lingual convergence—language-specific clusters vanish across training epochs and morph into coherent, language-agnostic semantic groupings. Attention consistency experiments reveal that Nayana-tuned ColGemma3 models activate identical visual regions for cross-lingual queries, validating true cross-modal, cross-script alignment rather than language-specific heuristics.

Implications, Limitations, and Prospects

The M3DR framework demonstrates that explicit multilingual multimodal training is imperative, as English-centric VLMs—even when retrained—fail on non-English retrieval tasks. Practically, M3DR enables scalable, language-agnostic document retrieval for enterprise knowledge management, education, and digital heritage, offering equitable access across global linguistic communities. The Matryoshka mechanism facilitates efficient deployment under varying storage constraints.

Theoretically, this work validates that cross-lingual transfer can be robustly encoded in VLMs via synthetic layout-aware data generation, casting doubt on the assumption that English-centric pipelines are generalizable. Limitations include moderate performance degradation at rare language pairs and difficulties with complex tabular data and passage-level retrieval; future work should explore advanced cross-lingual alignment objectives, region-level retrieval, and scaling to low-resource or unseen scripts.

Conclusion

M3DR establishes new standards for universal multilingual multimodal document retrieval by synthesizing large-scale parallel corpora, pioneering efficient embedding architectures, and rigorously benchmarking across 22 languages and diverse scripts. Explicit multilingual training, architectural flexibility (dense vs. ColBERT), and Matryoshka embeddings yield state-of-the-art results, with practical and theoretical implications for the deployment and design of inclusive AI document intelligence systems.