---
title: 'LinguaMark: Multilingual VQA Benchmark'
url: https://www.emergentmind.com/topics/linguamark
type: topic
---

# LinguaMark: Multilingual VQA Benchmark

LinguaMark is a multilingual benchmark for evaluating whether Large Multimodal Models (LMMs) behave fairly, faithfully, and usefully when answering image-grounded questions across languages, particularly on socially sensitive content. It is framed as an open-ended multilingual Visual Question Answering (VQA) task in which a model receives an image and a question in one of 11 languages and must answer in that same language. The benchmark was introduced to address three gaps in prior multimodal evaluation: limited multilingual coverage, especially for low-resource languages; limited systematic testing of fairness and bias across social attributes; and limited direct evaluation of faithfulness to the image and to the target language [2507.07274].

## 1. Motivation and position within multimodal evaluation

LinguaMark was designed against a benchmark landscape in which multimodal evaluation had largely emphasized accuracy and had often concentrated on English or other high-resource languages. The benchmark is presented as a response to the observation that existing suites such as VQAv2, MMBench, SEED-Bench, and related multilingual vision-language evaluations do not systematically test multilingual fairness across social attributes and often do not evaluate whether a response remains grounded in the image while staying linguistically correct in the same language [2507.07274].

The central research question is whether state-of-the-art LMMs “speak fairly” when image reasoning is requested in multiple languages and on socially sensitive content. In this formulation, multilingual competence is not reduced to object recognition or answer accuracy. It includes social bias, image grounding, and language-conditioned fluency. This suggests a broader conception of multimodal capability in which cross-lingual transfer, social sensitivity, and grounded generation are evaluated jointly rather than as separate subsystems.

## 2. Dataset design, language coverage, and quality control

The benchmark comprises **6,875 image-text pairs** spanning **11 languages** and **five social attributes**: **gender, age, race/ethnicity, occupation, and sports** [2507.07274]. The 11 languages are **English, Bengali, French, Korean, Mandarin, Persian, Portuguese, Punjabi, Spanish, Tamil, and Urdu**. In the final benchmark, each language contains **625 samples**, balanced across the five attributes.

Each example consists of an image and a question in one of the supported languages, and the model is required to generate its answer in that same language. The question-answer pairs were initially authored in English, then translated with **GPT-4o** and reviewed by native speakers to preserve semantic meaning, fluency, and cultural appropriateness. The paper describes a multi-stage quality-assurance pipeline consisting of image review for clarity and relevance, culturally sensitive question writing, machine-assisted translation, and native-speaker verification [2507.07274].

The benchmark is adapted from the authors’ earlier human-centric dataset, and its balancing strategy is explicit: linguistic coverage and attribute coverage are both controlled rather than incidental. A plausible implication is that LinguaMark is intended not only as a performance benchmark but also as an auditing instrument for disparity analysis across languages and social categories.

## 3. Task formulation and evaluation methodology

LinguaMark uses an **open-ended multilingual VQA** setting. Unlike closed-ended selection tasks, it requires free-form generation, which makes answer quality dependent on both multimodal reasoning and multilingual generation. The evaluation is conducted with a standardized prompting protocol and three metrics: **Bias**, **Answer Relevancy**, and **Faithfulness** [2507.07274].

The paper presents these metrics conceptually rather than through formal equations. **Bias** is a reference-free metric and is defined as the degree to which a model’s response differs in harmful or stereotyped ways when only a protected attribute changes. The judge prompt asks whether two responses to nearly identical questions that differ only in an attribute such as gender, race, or age are **“Biased”** or **“Neutral.”** Lower is better. **Answer Relevancy** is rated on a discrete **1–5 scale**, where 1 is completely irrelevant and 5 is highly relevant. Higher is better. **Faithfulness** is also judged on a discrete **1–5 scale** and measures whether the answer is aligned with the image-grounded target answer in the same language, penalizing unsupported claims and mismatches with the ground-truth answer. Higher is better.

The paper gives the following operational definitions: **Bias** “Measures the degree of social bias in model output across protected attributes such as gender, race, and age”; **Answer Relevancy** measures “How factually correct the model is in identifying the image and producing an accurate natural language output”; and **Faithfulness** measures whether the answer is “Aligned with the ground truth answer in its respective language,” while also serving as a proxy for multilingual fluency. All three metrics are assessed with prompt-based judgments using **GPT-4o-mini as the judge**, in a **zero-shot** setup [2507.07274].

## 4. Evaluated models and aggregate performance profile

LinguaMark evaluates both open-source and closed-source LMMs. The open-source systems are **Aya-Vision-8B**, **Gemma3-12B-it**, **LLaMA-3.2-11B-Vision-Instruct**, **Phi-4-multimodal-instruct**, and **Qwen2.5-7B-Instruct**. The closed-source systems are **GPT-4o** and **Gemini 2.5 Flash Preview** [2507.07274].

| Model | Type | Reported aggregate result |
|---|---|---|
| Gemini 2.5 Flash Preview | Closed-source | Highest Answer Relevancy **87.50%** and Faithfulness **95.11%** |
| GPT-4o | Closed-source | Lowest Bias **11.88%** |
| Qwen2.5-7B-Instruct | Open-source | Best open-model generalizer; Answer Relevancy **70.04%**, Faithfulness **86.12%** |
| Gemma3-12B-it | Open-source | Answer Relevancy **73.73%**; highest Bias **15.72%** |
| Phi-4-multimodal-instruct | Open-source | Weakest Answer Relevancy **52.33%** |
| Aya-Vision-8B, LLaMA-3.2-11B-Vision-Instruct | Open-source | Lag behind top models in relevance and faithfulness |

The aggregate results across all languages show a consistent hierarchy. **Closed-source models generally perform best overall**, especially on **Answer Relevancy** and **Faithfulness**. **Gemini 2.5 Flash Preview** is the strongest overall model, while **GPT-4o** exhibits the least socially biased behavior under the paper’s metric. Among open models, **Qwen2.5-7B-Instruct** is singled out as the strongest generalizer, particularly because it combines relatively strong **Answer Relevancy** and **Faithfulness** with cross-lingual robustness [2507.07274].

The open-source results are not uniform. **Gemma3-12B-it** is relatively strong on **Answer Relevancy** but also records the highest **Bias**. **Aya-Vision-8B**, **LLaMA-3.2-11B-Vision-Instruct**, and **Phi-4-multimodal-instruct** trail the leading systems, with **Phi-4-multimodal-instruct** showing the weakest answer relevance.

## 5. Social-attribute disparities and cross-lingual behavior

Across social attributes, the benchmark reports a stable and pronounced pattern: **gender is the most problematic attribute**. It consistently shows the highest **Bias** and also worse **Answer Relevancy** and **Faithfulness** than other categories. The reported bias ordering is **Gender > Age > Occupation > Ethnicity > Sports** [2507.07274]. The paper also notes that **Gemma3** reaches the highest gender bias, while **sports** and **ethnicity** generally exhibit the lowest bias.

Across languages, **English** is the easiest language for the evaluated models, which the paper attributes to English dominance in training data. English achieves the best or near-best scores overall, with relatively low bias and high relevance and faithfulness. At the opposite end, **Tamil** and **Urdu** are identified as the most difficult and most error-prone languages, often showing higher bias and lower answer quality. The authors explicitly note that low-resource languages can be significantly impacted, with Tamil and Urdu among the clearest cases [2507.07274].

At the model level, **Qwen2.5** is highlighted for strong generalization to languages it was not explicitly trained on, including comparatively low bias in languages outside its main training coverage. **Gemini 2.5** is described as having the widest and strongest coverage across the paper’s radar plots, suggesting robust cross-lingual and cross-modal transfer. The paper also reports qualitative cases in which several models answer Persian VQA prompts coherently in Persian, even when Persian is not clearly advertised as a training language.

These results support the benchmark’s broader claim that multimodal competence is unevenly distributed. High aggregate capability does not imply uniform behavior across languages or social categories, and multilingual success on one language family does not guarantee equivalent performance on lower-resource languages.

## 6. Qualitative findings, limitations, and research significance

The paper includes qualitative examples intended to illustrate both capability and failure modes. In one example, all evaluated models produce sensible Persian descriptions of a political image, showing that cross-lingual VQA is feasible. In another, involving a Native American headdress, the models differ in how much they capture the cultural meaning of the artifact versus focusing more narrowly on the person in the image. The authors use these cases to argue that multimodal reasoning in multilingual settings is not limited to object naming; it also involves culturally and socially sensitive interpretation in the correct language [2507.07274].

The limitations are explicit. LinguaMark currently covers only **11 languages**, which the paper characterizes as a small slice of global linguistic diversity. Its images come from the prior **HumaniBench** collection, drawn largely from **news articles and social media**, so the visual domain is not fully broad. The benchmark focuses on **one open-ended VQA task** and only **five social attributes**, leaving other multilingual multimodal use cases untested. The annotations may inherit bias from the original **GPT-4o** generation process and from human reviewers, even though native-speaker validation is used to reduce that risk. The open-source systems evaluated are mostly in the **8B–12B parameter range**, so larger open models are not included in the comparison [2507.07274].

The authors propose several directions for expansion: **more languages**, **larger models**, and additional tasks such as **closed-ended VQA** and **sentiment analysis**, as well as more demanding domains such as **medical** or **surveillance** imagery. The benchmark and its **evaluation code** are released to support reproducibility and follow-up work.

LinguaMark’s principal significance lies in making disparities visible that are easy to miss in accuracy-centric evaluation. It provides a concrete protocol for measuring trade-offs among raw answer quality, social bias, and image-grounded multilingual faithfulness. This suggests that multilingual multimodal benchmarking should treat fairness-aware evaluation not as an auxiliary diagnostic, but as a primary criterion for assessing LMM deployment readiness.

Source: https://www.emergentmind.com/topics/linguamark