Multilingual Training and Evaluation Resources for Vision-Language Models

Abstract: Vision LLMs (VLMs) achieved rapid progress in the recent years. However, despite their growth, VLMs development is heavily grounded on English, leading to two main limitations: (i) the lack of multilingual and multimodal datasets for training, and (ii) the scarcity of comprehensive evaluation benchmarks across languages. In this work, we address these gaps by introducing a new comprehensive suite of resources for VLMs training and evaluation spanning five European languages (English, French, German, Italian, and Spanish). We adopt a regeneration-translation paradigm that produces high-quality cross-lingual resources by combining curated synthetic generation and manual annotation. Specifically, we build Multi-PixMo, a training corpus obtained regenerating examples from Pixmo pre-existing datasets with permissively licensed models: PixMo-Cap, PixMo-AskModelAnything, and CoSyn-400k. On the evaluation side, we construct a set of multilingual benchmarks derived translating widely used English datasets (MMbench, ScienceQA, MME, POPE, AI2D). We assess the quality of these resources through qualitative and quantitative human analyses, measuring inter-annotator agreement. Additionally, we perform ablation studies to demonstrate the impact of multilingual data, with respect to English only, in VLMs training. Experiments, comprising 3 different models show that using multilingual, multimodal examples for training VLMs aids is consistently beneficial on non-English benchmarks, with positive transfer to English as well.

• The paper introduces a novel regeneration-translation paradigm for constructing high-quality, legally robust multilingual vision-language datasets.
• It presents three datasets—Multi-PixMo-Cap, Multi-PixMo-AskModelAnything, and Multi-PixMo-CoSyn-400k—comprising over 1.1M images and 2.8M examples across five languages.
• Empirical results show that multilingual training improves cross-lingual accuracy without degrading English performance, validated via human reviews and BERTScore F1 metrics.

Multilingual Resources for Vision-LLM Training and Evaluation

Motivation and Challenges

Vision-LLMs (VLMs) have advanced rapidly, but their development has been predominantly reliant on English-centric datasets and benchmarks. This linguistic bias introduces significant constraints: it hampers the cross-lingual performance of multimodal systems and limits the construction of inclusive benchmarks. The scarcity of large-scale, high-quality multilingual multimodal resources is compounded by restrictive licensing on many existing datasets and a lack of systematic evaluation protocols for multiple languages.

Regeneration-Translation Paradigm for Data Construction

The paper introduces a controlled regeneration-translation methodology for building comprehensive cross-lingual datasets. Rather than direct translation, the approach leverages a permissively licensed multimodal model (Qwen3-VL-235B-A22B-Instruct) conditioned on both the image and the source text to generate new annotations. Native speakers are involved in prompt refinement cycles and manual blind review, ensuring semantic fidelity, linguistic naturalness, and legal homogeneity. This process yields three distinct datasets:

• Multi-PixMo-Cap: Image-caption pairs generated across five languages, with rigorous grounding to the source transcription.
• Multi-PixMo-AskModelAnything: Visual question-answering (VQA) triples, where questions are translated and answers regenerated with image and question as context. Systematic numerical errors (e.g., counting tasks) are manually corrected.
• Multi-PixMo-CoSyn-400k: Synthetic text-rich images (charts, diagrams, tables) with QA pairs fully regenerated in target languages (not translation-based), filtering out low-quality content.

The resulting corpus exceeds 1.1M images and 2.8M examples, balancing English (40%) with French, German, Italian, and Spanish (15% each).

Multilingual Evaluation Benchmarks

A suite named MEVBench (Multimodal European Vision Benchmark) is constructed by translating prominent English evaluation datasets into four additional European languages using state-of-the-art LLMs. MEVBench includes:

• ScienceQA [Lu2022ScienceQA]
• MMBench [liu2023mmbench]
• AI2D [kembhavi2016ai2d]
• POPE [Li2023POPE]
• MME [Fu2023MME]

These benchmarks facilitate systematic assessment of visual perception, reasoning, hallucination detection, and text-rich image understanding across languages, using consistent task formats (MCQ, counting, binary classification) and metrics (Exact Match, composite scores).

Empirical Validation and Error Analysis

Quantitative Human Evaluation

The datasets are subjected to dual-stage human review by expert annotators in each language, measuring binary correctness for content adherence, linguistic form, translation quality, and factual precision. Positive judgments of linguistic form prevail (>96%), with content adherence also high (>85%) except for French (slightly lower but consistently within positive margins). Inter-annotator agreement, measured by Gwet's AC1, is robust (0.8–1.00), with strongest reliability in English, Italian, and Spanish. Most disagreements trace to borderline subjective interpretation or visual ambiguity.

Qualitative Error Analysis

Error attribution reveals:

• Propagation of upstream data noise (e.g., transcription inaccuracies, synthetic image artifacts),
• Systematic VLM capability failures (e.g., spatial reasoning, numerical counting, color identification, unverified inference/hallucination),
• Cases where the regeneration setup corrects source errors by aligning to visual evidence,
• Concise style of regenerated annotations, with average length decreased by 30–40% but retaining semantic coverage superior to legacy datasets.

Experimental Results

Training Configurations

Three model architectures (LLaMA3-3B, Qwen3-4B, LLaMA3-8B) are evaluated under four training protocols differentiating monolingual vs multilingual data in two stages (projection layer alignment with image-caption pairs; instruction-tuning with VQA/chart QA). Comparative runs are performed against English-only baselines.

Impact on English and Non-English Benchmarks

Multilingual training does not degrade English performance—differences are minor and occasionally positive, especially for larger models. On MEVBench, accuracy gains are pronounced in non-English splits, consistently benefiting German, Spanish, French, and Italian. For example, in MMBench, LLaMA3-3B cap$_{mu}$\,\to\,$ama$_{mu}$exceeds cap$_{en}$\,\to\,$ama$_{en}$ by +1.5 points; Qwen3-4B similarly improves across benchmarks. Incorporating Multi-PixMo-CoSyn-400k (cos) in instruction tuning further augments scores in chart/diagram-intensive tasks (e.g., AI2D), but can have negative effects under domain mismatch (e.g., ScienceQA for smaller models).

Supervised evaluation using BERTScore F1 (XLM-RoBERTa-Large) confirms that multilingual training boosts generative quality across all language splits, with regularization effects also improving English. On generative benchmarks (Multi-VQAv2), multilingual tuning yields a +20.6 point improvement (Qwen3-4B eng/multi vs eng/eng). Zero-shot transfer evaluation on external benchmarks (MTVQA, CVQA) demonstrates modest but consistent gains in unseen multilingual tasks.

Comparison with External Baselines

Publicly available instruction-tuned VLMs (Qwen2-VL-2B-Instruct, SmolVLM2-2.2B-Instruct) are evaluated on MEVBench. The new resources and training pipeline produce stronger and more balanced cross-lingual performance, particularly in text-rich generative tasks and diagram understanding, validating the practical advantages of systematic multilingual supervision.

Practical and Theoretical Implications

This work delivers openly licensed, large-scale resources for both training and systematic evaluation of VLMs across European languages. The regeneration-translation paradigm sets a new benchmark for dataset creation balancing semantic control, annotation quality, and legally unrestricted reuse. Empirical results demonstrate that multilingual multimodal supervision directly translates into improved generalization and cross-lingual transfer, with no tradeoff in English. The suite enables rigorous benchmarking of VLMs in settings that more closely match global and practical deployment requirements.

Theoretically, the workflows and results highlight the significance of parallel annotation protocols, visual grounding constraints during regeneration, and the regularization effects of linguistic diversity in multimodal optimization. The findings align with trends seen in multilingual NLP, reinforcing the value of language-adaptive supervision [conneau2020xlmrlarge], but extend them decisively into the vision-language domain.

Speculatively, as more permissively licensed multimodal models and datasets emerge, such methodologies are likely to underpin future state-of-the-art VLMs and accelerate development in domains beyond English, including scientific, medical, and culturally contextual applications.

Conclusion

The resource suite and systematic evaluation protocol introduced provide a scalable, legally robust foundation for multilingual VLM research and deployment. Empirical evidence confirms the consistent benefit of multilingual multimodal data for both cross-lingual generalization and English tasks. The release of both training corpora and evaluation benchmarks is positioned to drive more inclusive, culturally relevant, and linguistically broad VLM development, facilitating rigorous benchmarking and open scientific progress (2604.18347).