Kaleidoscope Benchmark for Multilingual VLMs

• Kaleidoscope Benchmark is a multilingual, in-language exam suite designed to evaluate vision-language models using culturally authentic exam materials.
• It features 20,911 multiple-choice questions across 18 languages and 14 subjects, split into multimodal and text-only items to assess both visual and language reasoning.
• The evaluation protocol uses macro-averaged accuracy and format error rate to highlight performance disparities across language-resource tiers, visual complexities, and cultural contexts.

Searching arXiv for the benchmark paper and closely related multilingual VLM evaluation work. Kaleidoscope is a large-scale, in-language multimodal benchmark for evaluating vision-LLMs across diverse languages and visual inputs. It covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions, and was introduced to address the reliance of vision-language evaluation on English-language benchmarks and translated test items rather than culturally grounded, native-language assessments (Salazar et al., 9 Apr 2025).

1. Definition, scope, and naming

Kaleidoscope is framed as an exam benchmark for multilingual vision evaluation. Its central objective is to push vision-LLMs beyond an English-and-translation mindset and toward in-language, culturally grounded multimodal reasoning. The benchmark is therefore not merely multilingual in the sense of translation coverage; it is designed around original exam material in each language, with the stated aim of preserving cultural context such as local units, region-specific diagrams, and idiomatic phrasing (Salazar et al., 9 Apr 2025).

A common point of confusion is the name itself. “Kaleidoscope” also appears in the unrelated paper “Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps” (Dao et al., 2020), where it denotes a family of matrices for structured linear transformations rather than a benchmark for multilingual vision-language evaluation. In current usage, “Kaleidoscope Benchmark” refers specifically to the multilingual multimodal exam suite introduced for evaluating VLMs (Salazar et al., 9 Apr 2025).

The benchmark is presented as a response to two linked deficiencies in prior evaluation practice: English-centric testing and the use of machine-translated datasets. The former constrains coverage, while the latter may erase culturally specific cues and introduce “translationese” artifacts. This suggests that Kaleidoscope is intended not only as an accuracy benchmark but also as an instrument for detecting failures of linguistic, cultural, and visual grounding in deployed VLMs.

2. Coverage, composition, and modality structure

Kaleidoscope spans 18 languages drawn from eight language families and three resource tiers. The summary explicitly identifies high-resource examples such as English, Spanish, Portuguese, Hindi, and Persian; mid-resource examples such as Bengali, Lithuanian, and Ukrainian; and low-resource examples such as Telugu and Nepali (Salazar et al., 9 Apr 2025). This tiered design is important because the benchmark is meant to expose differential model behavior across resource conditions rather than report a single pooled score.

The subject coverage comprises 14 subjects clustered into six broad domains. These domains extend from Humanities & Social Sciences—History, Sociology, Economics, Language, Geography, and Social Sciences—through STEM—Mathematics, Physics, Chemistry, Biology, and Engineering—to Practical Skills—Medicine and Driving License—and General Reasoning (Salazar et al., 9 Apr 2025). The use of exam material from heterogeneous domains makes the benchmark sensitive to both disciplinary content knowledge and visual reasoning under language variation.

Each item is a four-way multiple-choice question with exactly four answer options. The collection contains 20,911 items in total. Of these, 55% (11,459) are multimodal, combining image and text, and 45% (9,452) are text-only and serve as a unimodal control. The multimodal subset includes tables, graphs, diagrams, photographs, formulas, maps, figures, and text-rich images (Salazar et al., 9 Apr 2025).

This composition has two methodological consequences. First, the text-only split allows direct comparison between language competence and joint vision-language competence. Second, the multimodal split is deliberately broad in visual type, so benchmark difficulty is not reducible to a single perceptual regime such as natural images. A plausible implication is that score differences across modalities partly reflect failures of structured visual parsing rather than generic image understanding alone.

3. Data collection, provenance, and quality control

The benchmark was built through an open-science collaboration involving over 100 volunteer researchers from 20 countries, working through the Cohere for AI community and allied networks. These contributors manually sourced exams in their original form from PDFs, government repositories, and textbooks, with an emphasis on materials authored by domain experts such as high-school or university instructors (Salazar et al., 9 Apr 2025). The collection protocol therefore prioritizes native provenance and educational authenticity over synthetic generation.

Contributors followed curation guidelines specifying which exam types to include, namely multiple-choice questions with four options, how to document licensing and provenance, and how to annotate whether an image was “essential” or merely “useful.” Text and images were extracted using PDF and web parsers together with OCR tools including Mathpix and Tesseract, after which heuristic rules and in-house prompts to GPT-4o and Claude 3.5 Sonnet were used to realign questions, answer choices, and figures (Salazar et al., 9 Apr 2025).

Quality control proceeded in three stages. The first stage used a dual annotator pass on raw collected exams for strict compliance with license and format. The second stage applied automated JSON schema and duplication checks. The third stage used final human review for each language to detect mislinked images, malformed formulas, and alignment errors. In addition, evaluation-phase quality control was performed: questions on which all models failed, or on which outputs were malformed, triggered secondary review, and flawed items were corrected or removed (Salazar et al., 9 Apr 2025).

This validation pipeline is notable because it treats benchmark integrity as an ongoing process rather than a one-time release condition. The explicit inclusion of language-specific final review is especially consequential in a multilingual setting, where parsing and OCR errors can be strongly script-dependent.

4. Evaluation protocol and formal metrics

Kaleidoscope is evaluated as a four-way multiple-choice question answering task. The primary metric is accuracy, augmented by a Format Error Rate to capture invalid or missing model outputs. To avoid domination by high-volume languages, results are also macro-averaged across languages so that each language carries equal weight (Salazar et al., 9 Apr 2025).

Let $N$ be the total number of questions in a given evaluation split, and let $y_i \in \{A, B, C, D\}$ be the correct answer for question $i$. Let $\hat{y}_i$ denote the model’s selected option if it is one of $\{A, B, C, D\}$, and let $\hat{y}_i = \perp$ if the model fails to produce a valid choice. Then the reported metrics are:

$$\mathrm{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat{y}_i = y_i]$$

$$\mathrm{FER} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat{y}_i = \perp]$$

If there are $L$ languages and $\mathrm{Acc}_l$ is the accuracy on language $y_i \in \{A, B, C, D\}$0, macro-average accuracy is defined as

$y_i \in \{A, B, C, D\}$1

The benchmark summary notes that $y_i \in \{A, B, C, D\}$2-score was not employed in this MCQ context, although per-option precision and recall could in principle be defined in the standard way and combined into $y_i \in \{A, B, C, D\}$3 (Salazar et al., 9 Apr 2025). The choice not to foreground $y_i \in \{A, B, C, D\}$4 is methodologically consistent with single-label, fixed-option multiple-choice evaluation, where accuracy directly reflects the task objective and FER captures output-format failures separately.

5. Baseline systems and empirical performance profile

The reported evaluation includes closed-source “leaderboard” models—Claude 3.5 Sonnet, Gemini 1.5 Pro, and GPT-4o—and open-weight VLMs including Aya-Vision, Molmo, Pangea, and Qwen2.5-VL models ranging from 3B to 72B (Salazar et al., 9 Apr 2025). Overall macro-averaged accuracy and FER for representative systems are summarized below.

Model	Macro-averaged accuracy	FER
Claude 3.5 Sonnet	62.9%	1.8%
Gemini 1.5 Pro	62.1%	1.6%
GPT-4o	58.3%	6.5%
Qwen2.5-VL-72B	52.9%	0.02%
Qwen2.5-VL-7B	39.6%	—

The aggregate numbers conceal marked stratification by language resource level, modality, domain, and script. On the multimodal split, closed models reach roughly 55–60% macro-averaged accuracy on high-resource languages such as English, Spanish, Portuguese, Hindi, and Persian, but only roughly 22–33% on low-resource languages such as Telugu and Nepali. The summary gives a concrete example: Gemini 1.5 achieves about 61.5% on English multimodal items versus about 22.8% on Nepali multimodal items (Salazar et al., 9 Apr 2025).

Modal differences are also pronounced. Closed models lose about 21.6 percentage points between text-only and multimodal settings; GPT-4o, for example, moves from 71.4% on text-only items to 49.8% on multimodal items. Open models show a smaller gap in at least one cited case, with Molmo-7B scoring about 35.1% on text and 31.4% on multimodal questions (Salazar et al., 9 Apr 2025). This suggests that stronger language competence does not automatically translate into correspondingly strong multimodal reasoning under multilingual conditions.

Domain-level disparities are similarly large. Across closed models on the multimodal subset, Humanities & Social Sciences average about 83.7% accuracy, whereas STEM subjects average about 59.2%. Sociology reaches about 93% for Claude 3.5 Sonnet and GPT-4o, while Mathematics remains below 50% (Salazar et al., 9 Apr 2025). Script effects are systematic as well: all models perform better on Latin-script languages than on non-Latin scripts, and closed models lose about 10–15 percentage points when moving from Latin to Devanagari, Cyrillic, Perso-Arabic, or Telugu scripts.

Taken together, these results indicate that multilingual VLM quality cannot be adequately summarized by a single global score. Performance depends strongly on whether the task involves low-resource languages, non-Latin scripts, structured visuals, or STEM-style problem solving.

6. Failure modes, limitations, and research implications

The benchmark identifies several recurrent failure modes. In visual-symbolic reasoning, models struggle with structured visuals such as diagrams, formulas, and tables; the summary reports 62.9% for Claude 3.5 on diagrams and 68.3% for Gemini on formulas (Salazar et al., 9 Apr 2025). In multilingual equity, low-resource languages remain a bottleneck, with the benchmark emphasizing that obtaining genuinely in-language STEM materials is difficult but necessary. In domain-specific reasoning, STEM problem solving involving multi-step arithmetic or physical reasoning continues to lag well behind Humanities performance.

Format robustness is treated as a first-class issue rather than a peripheral annoyance. GPT-4o’s multimodal regime reaches a FER of 10.5%, and Pangea-7B exhibits very large FER in some languages, including 45% in Telugu (Salazar et al., 9 Apr 2025). These results imply that answer-format compliance and instruction-following remain nontrivial parts of benchmark performance, especially in multilingual multimodal deployment settings.

The benchmark also foregrounds cultural adaptation. It notes that models often default to Western defaults, including pounds instead of grams and US-style traffic scenarios, even when the exam context is local (Salazar et al., 9 Apr 2025). This is significant because Kaleidoscope is explicitly constructed to test whether models preserve local conventions rather than merely approximate the surface form of the prompt.

The recommendations attached to the benchmark follow directly from these observations. They include expanding participatory data collection for underrepresented languages, incorporating structured visual backbones or neuro-symbolic modules for diagrams, formulae, and tables, designing targeted adapter or fine-tuning strategies for low-resource scripts and culturally specific imagery, and extending evaluation beyond raw accuracy to measures such as “cultural consistency” (Salazar et al., 9 Apr 2025). The benchmark is publicly available through the cited project links, positioning it as an open resource for multilingual and multicultural VLM evaluation.

A plausible implication is that Kaleidoscope functions both as a benchmark and as a diagnostic framework. Its significance lies not only in score ranking, but in its decomposition of multilingual multimodal capability into language-resource sensitivity, script robustness, visual-symbolic competence, and cultural grounding.