LinguaMark: Multilingual VQA Benchmark
- LinguaMark is a multilingual benchmark that evaluates large multimodal models on image-grounded VQA tasks, focusing on fairness, faithfulness, and utility across 11 languages.
- It employs a balanced dataset of 6,875 image-text pairs and a multi-stage quality assurance pipeline, ensuring accurate, culturally sensitive translations.
- The evaluation framework uses Bias, Answer Relevancy, and Faithfulness metrics to compare open- and closed-source models and to assess social biases.
LinguaMark is a multilingual benchmark for evaluating whether Large Multimodal Models (LMMs) behave fairly, faithfully, and usefully when answering image-grounded questions across languages, particularly on socially sensitive content. It is framed as an open-ended multilingual Visual Question Answering (VQA) task in which a model receives an image and a question in one of 11 languages and must answer in that same language. The benchmark was introduced to address three gaps in prior multimodal evaluation: limited multilingual coverage, especially for low-resource languages; limited systematic testing of fairness and bias across social attributes; and limited direct evaluation of faithfulness to the image and to the target language (Raval et al., 9 Jul 2025).
1. Motivation and position within multimodal evaluation
LinguaMark was designed against a benchmark landscape in which multimodal evaluation had largely emphasized accuracy and had often concentrated on English or other high-resource languages. The benchmark is presented as a response to the observation that existing suites such as VQAv2, MMBench, SEED-Bench, and related multilingual vision-language evaluations do not systematically test multilingual fairness across social attributes and often do not evaluate whether a response remains grounded in the image while staying linguistically correct in the same language (Raval et al., 9 Jul 2025).
The central research question is whether state-of-the-art LMMs “speak fairly” when image reasoning is requested in multiple languages and on socially sensitive content. In this formulation, multilingual competence is not reduced to object recognition or answer accuracy. It includes social bias, image grounding, and language-conditioned fluency. This suggests a broader conception of multimodal capability in which cross-lingual transfer, social sensitivity, and grounded generation are evaluated jointly rather than as separate subsystems.
2. Dataset design, language coverage, and quality control
The benchmark comprises 6,875 image-text pairs spanning 11 languages and five social attributes: gender, age, race/ethnicity, occupation, and sports (Raval et al., 9 Jul 2025). The 11 languages are English, Bengali, French, Korean, Mandarin, Persian, Portuguese, Punjabi, Spanish, Tamil, and Urdu. In the final benchmark, each language contains 625 samples, balanced across the five attributes.
Each example consists of an image and a question in one of the supported languages, and the model is required to generate its answer in that same language. The question-answer pairs were initially authored in English, then translated with GPT-4o and reviewed by native speakers to preserve semantic meaning, fluency, and cultural appropriateness. The paper describes a multi-stage quality-assurance pipeline consisting of image review for clarity and relevance, culturally sensitive question writing, machine-assisted translation, and native-speaker verification (Raval et al., 9 Jul 2025).
The benchmark is adapted from the authors’ earlier human-centric dataset, and its balancing strategy is explicit: linguistic coverage and attribute coverage are both controlled rather than incidental. A plausible implication is that LinguaMark is intended not only as a performance benchmark but also as an auditing instrument for disparity analysis across languages and social categories.
3. Task formulation and evaluation methodology
LinguaMark uses an open-ended multilingual VQA setting. Unlike closed-ended selection tasks, it requires free-form generation, which makes answer quality dependent on both multimodal reasoning and multilingual generation. The evaluation is conducted with a standardized prompting protocol and three metrics: Bias, Answer Relevancy, and Faithfulness (Raval et al., 9 Jul 2025).
The paper presents these metrics conceptually rather than through formal equations. Bias is a reference-free metric and is defined as the degree to which a model’s response differs in harmful or stereotyped ways when only a protected attribute changes. The judge prompt asks whether two responses to nearly identical questions that differ only in an attribute such as gender, race, or age are “Biased” or “Neutral.” Lower is better. Answer Relevancy is rated on a discrete 1–5 scale, where 1 is completely irrelevant and 5 is highly relevant. Higher is better. Faithfulness is also judged on a discrete 1–5 scale and measures whether the answer is aligned with the image-grounded target answer in the same language, penalizing unsupported claims and mismatches with the ground-truth answer. Higher is better.
The paper gives the following operational definitions: Bias “Measures the degree of social bias in model output across protected attributes such as gender, race, and age”; Answer Relevancy measures “How factually correct the model is in identifying the image and producing an accurate natural language output”; and Faithfulness measures whether the answer is “Aligned with the ground truth answer in its respective language,” while also serving as a proxy for multilingual fluency. All three metrics are assessed with prompt-based judgments using GPT-4o-mini as the judge, in a zero-shot setup (Raval et al., 9 Jul 2025).
4. Evaluated models and aggregate performance profile
LinguaMark evaluates both open-source and closed-source LMMs. The open-source systems are Aya-Vision-8B, Gemma3-12B-it, LLaMA-3.2-11B-Vision-Instruct, Phi-4-multimodal-instruct, and Qwen2.5-7B-Instruct. The closed-source systems are GPT-4o and Gemini 2.5 Flash Preview (Raval et al., 9 Jul 2025).
| Model | Type | Reported aggregate result |
|---|---|---|
| Gemini 2.5 Flash Preview | Closed-source | Highest Answer Relevancy 87.50% and Faithfulness 95.11% |
| GPT-4o | Closed-source | Lowest Bias 11.88% |
| Qwen2.5-7B-Instruct | Open-source | Best open-model generalizer; Answer Relevancy 70.04%, Faithfulness 86.12% |
| Gemma3-12B-it | Open-source | Answer Relevancy 73.73%; highest Bias 15.72% |
| Phi-4-multimodal-instruct | Open-source | Weakest Answer Relevancy 52.33% |
| Aya-Vision-8B, LLaMA-3.2-11B-Vision-Instruct | Open-source | Lag behind top models in relevance and faithfulness |
The aggregate results across all languages show a consistent hierarchy. Closed-source models generally perform best overall, especially on Answer Relevancy and Faithfulness. Gemini 2.5 Flash Preview is the strongest overall model, while GPT-4o exhibits the least socially biased behavior under the paper’s metric. Among open models, Qwen2.5-7B-Instruct is singled out as the strongest generalizer, particularly because it combines relatively strong Answer Relevancy and Faithfulness with cross-lingual robustness (Raval et al., 9 Jul 2025).
The open-source results are not uniform. Gemma3-12B-it is relatively strong on Answer Relevancy but also records the highest Bias. Aya-Vision-8B, LLaMA-3.2-11B-Vision-Instruct, and Phi-4-multimodal-instruct trail the leading systems, with Phi-4-multimodal-instruct showing the weakest answer relevance.
5. Social-attribute disparities and cross-lingual behavior
Across social attributes, the benchmark reports a stable and pronounced pattern: gender is the most problematic attribute. It consistently shows the highest Bias and also worse Answer Relevancy and Faithfulness than other categories. The reported bias ordering is Gender > Age > Occupation > Ethnicity > Sports (Raval et al., 9 Jul 2025). The paper also notes that Gemma3 reaches the highest gender bias, while sports and ethnicity generally exhibit the lowest bias.
Across languages, English is the easiest language for the evaluated models, which the paper attributes to English dominance in training data. English achieves the best or near-best scores overall, with relatively low bias and high relevance and faithfulness. At the opposite end, Tamil and Urdu are identified as the most difficult and most error-prone languages, often showing higher bias and lower answer quality. The authors explicitly note that low-resource languages can be significantly impacted, with Tamil and Urdu among the clearest cases (Raval et al., 9 Jul 2025).
At the model level, Qwen2.5 is highlighted for strong generalization to languages it was not explicitly trained on, including comparatively low bias in languages outside its main training coverage. Gemini 2.5 is described as having the widest and strongest coverage across the paper’s radar plots, suggesting robust cross-lingual and cross-modal transfer. The paper also reports qualitative cases in which several models answer Persian VQA prompts coherently in Persian, even when Persian is not clearly advertised as a training language.
These results support the benchmark’s broader claim that multimodal competence is unevenly distributed. High aggregate capability does not imply uniform behavior across languages or social categories, and multilingual success on one language family does not guarantee equivalent performance on lower-resource languages.
6. Qualitative findings, limitations, and research significance
The paper includes qualitative examples intended to illustrate both capability and failure modes. In one example, all evaluated models produce sensible Persian descriptions of a political image, showing that cross-lingual VQA is feasible. In another, involving a Native American headdress, the models differ in how much they capture the cultural meaning of the artifact versus focusing more narrowly on the person in the image. The authors use these cases to argue that multimodal reasoning in multilingual settings is not limited to object naming; it also involves culturally and socially sensitive interpretation in the correct language (Raval et al., 9 Jul 2025).
The limitations are explicit. LinguaMark currently covers only 11 languages, which the paper characterizes as a small slice of global linguistic diversity. Its images come from the prior HumaniBench collection, drawn largely from news articles and social media, so the visual domain is not fully broad. The benchmark focuses on one open-ended VQA task and only five social attributes, leaving other multilingual multimodal use cases untested. The annotations may inherit bias from the original GPT-4o generation process and from human reviewers, even though native-speaker validation is used to reduce that risk. The open-source systems evaluated are mostly in the 8B–12B parameter range, so larger open models are not included in the comparison (Raval et al., 9 Jul 2025).
The authors propose several directions for expansion: more languages, larger models, and additional tasks such as closed-ended VQA and sentiment analysis, as well as more demanding domains such as medical or surveillance imagery. The benchmark and its evaluation code are released to support reproducibility and follow-up work.
LinguaMark’s principal significance lies in making disparities visible that are easy to miss in accuracy-centric evaluation. It provides a concrete protocol for measuring trade-offs among raw answer quality, social bias, and image-grounded multilingual faithfulness. This suggests that multilingual multimodal benchmarking should treat fairness-aware evaluation not as an auxiliary diagnostic, but as a primary criterion for assessing LMM deployment readiness.