ALM Bench: All Languages Matter Benchmark

• ALM-bench is a culturally grounded multilingual benchmark that evaluates visual question answering by testing image–text understanding across 100 languages.
• It combines a generic VQA branch with a unique cultural VQA branch, using 22,763 manually verified image–question–answer pairs to ensure cultural specificity.
• Empirical findings reveal significant performance gaps between high-resource and low-resource languages, emphasizing challenges in multilingual multimodal reasoning.

ALM-bench, short for All Languages Matter Benchmark, is a multilingual, culturally grounded benchmark for evaluating large multimodal models on image–text understanding across 100 languages, with a particular emphasis on low-resource languages and culturally specific visual knowledge (Vayani et al., 2024). It is organized as a visual question answering–style benchmark with 22,763 manually verified image–question–answer pairs, spanning 73 countries, 24 scripts, 15 language families, 19 domains, and four question types. Its central purpose is to test whether large multimodal models can interpret images and answer questions in diverse languages while reasoning about locally specific cultural content rather than relying on English-centric or culturally narrow priors (Vayani et al., 2024).

1. Definition and motivation

ALM-bench was introduced in response to three limitations in prior multimodal evaluation: limited language coverage, limited cultural coverage, and restricted task formats (Vayani et al., 2024). Existing large multimodal model benchmarks are described as often monolingual or only lightly multilingual, frequently centered on a small number of high-resource languages, and commonly focused on generic perception or a single question type. ALM-bench addresses this by combining broad multilingual coverage with culturally grounded content and mixed evaluation formats.

The benchmark covers 100 languages across 73 countries, 24 scripts, and 15 language families, and it explicitly balances 50 high-resource and 50 low-resource languages (Vayani et al., 2024). The paper frames this design choice through the claim that low-resource, non-Western, and culturally diverse languages should be treated as first-class targets in multimodal evaluation rather than as peripheral extensions of English-only systems. This suggests that ALM-bench is intended not merely as a translation stress test, but as an instrument for measuring whether multimodal models can operate under heterogeneous linguistic and cultural conditions.

A second motivation is cultural specificity. ALM-bench is not restricted to generic scene understanding. It includes 13 cultural domains and is designed to test recognition and reasoning about rituals, architecture, heritage, customs, literature, music, and other culturally situated phenomena (Vayani et al., 2024). The benchmark therefore targets failures that are not visible in standard visual QA datasets, such as confusion between visually similar but culturally distinct festivals or misidentification of local public figures.

A third motivation is evaluation depth. ALM-bench supports multiple choice, true/false, short open-ended VQA, and long open-ended VQA, allowing it to probe both decision-style recognition and free-form multimodal generation (Vayani et al., 2024). This multi-format structure matters because the benchmark’s reported results show that model rankings and failure modes differ substantially between closed-form and open-ended tasks.

2. Dataset construction and curation pipeline

ALM-bench is built through two parallel branches: a generic VQA branch and a cultural VQA branch (Vayani et al., 2024). The benchmark is not simply a translated English dataset. In the cultural portion, images and question–answer pairs are unique per language, rather than shared across languages.

The benchmark first defines country–language pairs using the World Values Survey as a reference for selecting a primary country association for each language (Vayani et al., 2024). This is intended to align each language with a dominant cultural context; for multilingual countries, multiple languages are included to represent intra-country diversity. Language selection is guided by resource balance, Western and non-Western coverage, and annotator availability.

For the generic VQA branch, the source is LLaVA-Bench (In-the-Wild). English questions are translated into the other 99 languages using GPT-4o, and then manually verified and corrected by native speakers (Vayani et al., 2024). This yields 6,000 open-ended samples across the generic domains. For the cultural VQA branch, images are scraped from the web using targeted country–language–domain queries, and only open-licensed images are retained. These are automatically filtered for resolution and redundancy, then manually reviewed for cultural relevance and representativeness; approximately 10.7% of images are removed as culturally irrelevant or low-quality (Vayani et al., 2024).

For each retained cultural image, annotators first write a caption, and then GPT-4o generates a set of English questions: 2 MCQs, 2 short open-ended questions, 1 long open-ended question, and 1 true/false question (Vayani et al., 2024). These questions are instructed to require visual grounding, emphasize cultural aspects, avoid reinforcing stereotypes, and remain answerable from the image rather than prior knowledge alone. The generated English questions are then translated into the other 99 languages, again with native-speaker review and correction.

Human verification is central to the benchmark’s construction. The paper reports more than 800 hours of human annotation, involving 60+ volunteers from 50 countries; 80.3% are native speakers and 87.9% have lived more than 15 years in the relevant country (Vayani et al., 2024). Annotators review translations for semantic, cultural, language, and grammatical errors, and they may remove redundant or culturally irrelevant question–answer pairs. The paper notes that semantic and grammatical errors are the most frequent translation problems in GPT-4o outputs, which justifies the heavy manual verification layer.

Privacy and leakage controls are also built into the pipeline. Faces and personal identifying details are blurred using a face detection model and PicdeFacer, except in the Media and Notable Key Figures categories, where public figures are allowed (Vayani et al., 2024). Watermarks and answer-leaking text are removed or blurred. This indicates that benchmark construction was treated as both a linguistic and an ethical curation problem.

3. Linguistic and cultural coverage

ALM-bench’s scope is defined by its combination of language diversity and cultural breadth. The benchmark includes 100 languages, distributed across 24 scripts and 15 language families (Vayani et al., 2024). Scripts include Latin, Cyrillic, Arabic, Devanagari, Bengali, Ge’ez, Sinhala, Oriya, Myanmar, Chinese, Georgian, Greek, Hebrew, Hangul, Lao, Thai, and Armenian, among others. Language families include Indo-European, Afro-Asiatic, Austronesian, Uralic, Sino-Tibetan, Dravidian, Atlantic-Congo, Turkic, Japonic, Koreanic, Kartvelian, Tai-Kadai, Austroasiatic, Mongolic-Khitan, and the Basque isolate (Vayani et al., 2024).

The benchmark spans 19 domains, split into 6 generic and 13 cultural domains (Vayani et al., 2024). The generic domains are indoor scenes, outdoor scenes, memes, paintings, food items, and sketches. The cultural domains are Architecture, Customs, Economy, Festivals, Food, Heritage, Lifestyle, Literature, Media, Music, Notable Key Figures, Religion, and Sports. These categories are designed to probe knowledge that is locally grounded rather than globally dominant. For example, the Literature category is described as distinctive relative to many prior VQA benchmarks because it requires knowledge of national authors, poets, and literary works.

The benchmark’s cultural branch is explicitly designed to avoid superficial cultural proxies. Questions are required to depend on the image, not merely on entity-name recognition, and annotators review whether a question is truly image-grounded (Vayani et al., 2024). This is important because a benchmark can claim cultural coverage while still rewarding only memorized textual facts; ALM-bench attempts to bind cultural knowledge to visual evidence.

The paper also stresses bias correction and cultural appropriateness. GPT-4o is instructed not to perpetuate stereotypes during question generation, and native speakers review images and questions for cultural relevance and appropriateness (Vayani et al., 2024). A plausible implication is that ALM-bench treats cultural benchmarking as a data governance problem as much as a modeling problem: representativeness, translation fidelity, and stereotype avoidance are all part of the benchmark definition.

4. Task formulation and evaluation protocol

Each ALM-bench sample consists conceptually of an image, a question in a target language, an answer, and metadata identifying language, country, domain, and question type (Vayani et al., 2024). The benchmark does not introduce a single formal dataset notation, but its task structure is explicit.

The four supported question types are MCQ, true/false, short VQA, and long VQA (Vayani et al., 2024). In MCQ, the model receives an image, question text, and four answer choices; the output is expected to be the chosen option content rather than only the letter label. In true/false, the model answers a statement in the target language. In short VQA, the model generates a short free-form answer, and in long VQA it produces a longer explanation-like response.

Prompting is standardized. The system prompts are written in English, because the paper notes prior evidence that English system prompts can improve performance, while the actual question is posed in the target language and the model is instructed to answer in that same language (Vayani et al., 2024). This is a deliberate asymmetry: instruction scaffolding is centralized in English, but the inference burden remains multilingual.

The evaluation protocol mixes exact-match-style scoring and LLM-based judgment. For MCQ and true/false, the primary metric is accuracy (Vayani et al., 2024). For short VQA, GPT-4o acts as an automatic judge and assigns a 0–10 correctness score based on the question, ground-truth answer, and model output. For long VQA, GPT-4o evaluates consistency, fluency, and relevance, each on a 0–10 scale (Vayani et al., 2024). The paper also reports a cross-check using LLaMA-3.1-8B-Instruct as an alternative judge on decision tasks, with similar outcomes.

Aggregation is reported at multiple levels: by language, by script, by family, by category, and globally (Vayani et al., 2024). “Overall accuracy” refers to the average across all decision-based questions, while category-wise and language-wise views average over the respective subsets. This multi-granular reporting is central to the benchmark’s purpose, because a single global score would obscure the exact disparities the benchmark is meant to reveal.

5. Empirical findings

The headline empirical result is that ALM-bench is difficult even for frontier proprietary models. The paper reports GPT-4o at 78.8% overall accuracy, while the best open-source model, GLM-4V-9B, reaches 51.9% (Vayani et al., 2024). This roughly 27 percentage point gap is presented as evidence that multilingual, culturally grounded multimodal reasoning remains far from solved.

The benchmark also reveals a systematic high-resource versus low-resource disparity. According to the reported analysis, GPT-4o experiences a 6% performance drop from high-resource to low-resource languages, while GLM-4V-9B and Qwen2-VL show drops greater than 8% (Vayani et al., 2024). Specific language-level examples are more severe: GPT-4o drops from 88.4% accuracy in English to 50.8% in Amharic, while GLM-4V-9B drops from 80.3% in English to 15.6% in Amharic (Vayani et al., 2024). These results suggest that multimodal transfer does not erase the underlying data imbalance across languages and scripts.

Question type is another major discriminator. All models perform better on MCQ and true/false than on short and long open-ended VQA (Vayani et al., 2024). Closed-source models reportedly perform better on long VQA than on short VQA, whereas open-source models do better on short VQA and struggle with long, fluent, accurate responses across many languages. This implies that long-form multilingual generation remains a distinct bottleneck, separate from recognition or retrieval.

Category-wise evaluation shows that cultural difficulty is uneven. GPT-4o reaches an overall score of 80.3% across the 13 cultural categories, but the paper reports lower performance on Notable Key Figures (72.7%) and Customs, compared with stronger results in areas such as Heritage (83.7%) (Vayani et al., 2024). The authors attribute part of this pattern to training data frequency: categories such as heritage or education may appear more often in web data, whereas local public figures and subtle customs are more weakly represented.

A particularly important result concerns multimodal grounding. When the image is removed and a base text-only model is evaluated on a subset of 50 languages, performance drops sharply: GPT-4o loses 27.3 absolute points, with large gains from visual context such as +38.7% for Sinhala, +50% for Sanskrit, and +40% for Dutch when the image is available (Vayani et al., 2024). This indicates that ALM-bench is not reducible to multilingual trivia; the image materially changes the task.

The benchmark also supports explicit error analysis. Annotators categorize errors into lack of knowledge, reasoning error, perceptual error, language error, translation error, and lack of cultural understanding (Vayani et al., 2024). The paper reports that knowledge gaps, reasoning errors, and cultural misunderstanding dominate, while language and script errors are especially visible for low-resource scripts. This suggests that poor performance is not solely a translation problem; it is jointly perceptual, linguistic, and cultural.

6. Position in the benchmark landscape and naming ambiguity

ALM-bench is positioned against both multilingual multimodal benchmarks and culturally focused VQA benchmarks. Compared with Henna, CulturalVQA, GD-VCR, MMMB, MMBench, EXAMS-V, M3Exam, MaXM, xGQA, M4U, MME, MaRVL, and CVQA, the paper characterizes ALM-bench as the broadest combination of language count, script diversity, cultural specificity, and question-type coverage (Vayani et al., 2024). In that comparison, ALM-bench is described as supporting 100 languages, 24 scripts, 19 domains, and four question types, with explicit native-speaker verification and bias correction.

An important source of confusion is that the phrase “ALM Bench” has been used informally for several unrelated benchmark efforts. In the literature provided here, SalamahBench evaluates the safety alignment of Arabic LLMs rather than multilingual multimodal cultural reasoning (Abdelnasser et al., 3 Feb 2026). RSA-Bench evaluates the robustness of Audio Large Models under complex acoustic ecologies rather than image–text cultural understanding (Zhang et al., 15 Jan 2026). A distinct ALM Bench is also introduced for text-conditioned crystal generation and optimization in atomistic modeling (Edamadaka et al., 19 Jun 2026). These benchmarks share an acronymic surface form but differ in modality, task, and evaluation philosophy. In that sense, ALM-bench in the strict sense refers specifically to the All Languages Matter Benchmark for culturally diverse multilingual large multimodal model evaluation (Vayani et al., 2024).

The benchmark is publicly available through its project website, and the paper presents it as a foundation rather than a finished endpoint (Vayani et al., 2024). Its limitations are explicitly noted: some domains such as Economy are harder to populate with culturally unique images, web-sourced data may underrepresent subcultures or minority groups, and cultural representation can never be exhaustive. This suggests that ALM-bench is best understood as an evolving benchmark infrastructure for global multimodal evaluation, not as a complete solution to multilingual and multicultural coverage.