Trade-off between multilingual parameter allocation and visual understanding

Ascertain how allocating parameters to additional languages in multilingual bidirectional vision–language encoders for document retrieval trades off against understanding of the visual modality, and quantify the extent to which increasing the number of supported languages penalizes English retrieval performance.

Background

The study focuses on English-only training and evaluation, while suggesting the potential to release multilingual variants. Multilingual scaling introduces a resource allocation tension between language capacity and visual modality understanding, which may affect English retrieval quality.

Clarifying and quantifying this trade-off would inform architecture and data budgeting for multilingual visual document retrievers, guiding parameter distribution across languages without unduly harming performance in primary target languages.

References

While we expect the broad trends to generalize and see clear value in releasing multilingual variants, it remains unclear how allocating parameters to additional languages trades off against the understanding of the vision modality, and to what extent this penalizes English retrieval performance as the number of languages are scaled \citep{pmlr-v202-fernandes23a}.

— ModernVBERT: Towards Smaller Visual Document Retrievers  (2510.01149 - Teiletche et al., 1 Oct 2025) in Conclusion, Future Work and Limitations

We do not address the capacity limits of a single token space, leaving open the question of how many languages can be mapped into the English token space and how those would interfere.

— Why Pretraining Fails to Share Cross-Lingual Knowledge  (2609.19291 - Gaber et al., 16 Sep 2026) in Section 7, Limitations, subsection “Scalability to Massively Multilingual Settings”

The result is specific to French documents with English queries; broader language-pair coverage remains future work.

— MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval  (2609.01316 - Mahata et al., 1 Sep 2026) in Section 3.2, Cross-lingual analysis