Trade-off between multilingual parameter allocation and visual understanding

Ascertain how allocating parameters to additional languages in multilingual bidirectional vision–language encoders for document retrieval trades off against understanding of the visual modality, and quantify the extent to which increasing the number of supported languages penalizes English retrieval performance.

Background

The paper focuses on English-only training and evaluation, while suggesting the potential to release multilingual variants. Multilingual scaling introduces a resource allocation tension between language capacity and visual modality understanding, which may affect English retrieval quality.

Clarifying and quantifying this trade-off would inform architecture and data budgeting for multilingual visual document retrievers, guiding parameter distribution across languages without unduly harming performance in primary target languages.