BLEnD-Vis: Benchmarking Cultural VLMs

• The paper introduces BLEnD-Vis, a multimodal benchmark that evaluates cultural knowledge in vision-language models using over 21,000 MCQs and nearly 5,000 images across 16 regions.
• It leverages linguistic rephrasing and visual cues to expose model fragility and reveal significant performance gaps, especially in low-resource regions.
• Evaluation metrics such as Δ_rephrase and Gain_visual offer actionable insights into improving cross-modal transfer and mitigating cultural bias in VLMs.

BLEnD-Vis is a multimodal, multicultural benchmark specifically designed to evaluate the robustness and transferability of culturally grounded knowledge in vision-LLMs (VLMs) under linguistic rephrasings and across visual modalities. Building upon the BLEnD dataset, BLEnD-Vis challenges VLMs to demonstrate not just static recall or isolated visual grounding, but genuine robustness in the integration and cross-modal transfer of everyday cultural knowledge across 16 culturally diverse regions, using over 21,000 multiple-choice question (MCQ) instances and 4,916 contextually generated images. This benchmark systematically reveals fragility in current VLMs’ representation and transfer of cultural knowledge, particularly highlighting weaknesses in low-resource regions and under linguistic or visual transformation (Tan et al., 13 Oct 2025).

1. Dataset Development and Structure

BLEnD-Vis is constructed through a multi-step pipeline that ensures rigorous, validated, and balanced coverage of tangible, everyday cultural knowledge.

1.1 Scope and Template Selection

• Regions Covered: 16 culturally diverse world regions.
• Question Templates: 313 visually representable ("tangible") MCQ templates were retained from 320 candidates after GPT-4o–based tangibility classification out of an initial pool of 500 English short-answer questions from BLEnD.
• Base Question Construction: Templates were consolidated into MCQ formats spanning all regions, grounding concepts in real-world, visually depictable cultural artifacts or practices.

1.2 Generation Pipeline

• MCQ Answer Consolidation: Approximately 306,000 BLEnD MCQ instances were parsed to associate each template-region pair with correct answers.
• Tangibility Filtering: Templates were filtered by GPT-4o for visual representability, resulting in 313 "tangible" instances.
• Rephrasing and Placeholders: Region → Entity templates were inverted to Entity → Region, and generic placeholders (e.g., “this food”, “this clothing”) introduced.
• Image Generation: Each unique answer–region pair was visualized using Gemini 2.5 Flash Image, producing 4,916 synthetic but contextually accurate images.
• Human Validation: Three annotators reviewed all rephrasings (12.2% required manual correction), and a stratified sample of 500 images (5.4% ‘Bad’ rate, inter-annotator κ ≈ 0.82).
• Parallel MCQ Instantiation: Each <template, region, answer, image> quadruple yielded up to 5 unique MCQs by sampling 3 semantic distractors from other regions.

Table: BLEnD-Vis MCQ Distribution by Topic (excerpt from Table 1)

Category	MCQ Count	Percentage
Education	1,765	8.10%
Family	2,312	10.61%
Food	6,681	30.67%
Total	21,782	100.00%

2. Task Formats and Evaluation Protocols

BLEnD-Vis offers three parallel MCQ formats, each probing different aspects of VLMs’ modal integration and cultural robustness. Evaluation is zero-shot for all tested models.

• Original MCQ (O): Text-only, Region → Entity (e.g., "What is a common snack in {region}?”).
• Rephrased MCQ (R): Text-only, Entity → Region (e.g., "In which country/region is {answer} a common snack?").
• VQA-Style MCQ (V): Image plus rephrased text (image depicting {answer} in regional context, with entity replaced by placeholder, options as regions).

Linguistic rephrasing was automated with GPT-4o, enforcing inversion and placeholder use, and human validated. VQA input simply prepends the generated image; no model-specific handling is required.

3. Evaluation Metrics

The primary performance metrics are:

• $$A_O = \frac{\#\text{correct on Original MCQ}}{21,782}$$
• $$A_R = \frac{\#\text{correct on Rephrased MCQ}}{21,782}$$
• $A_V = \frac{\#\text{correct on VQA MCQ}}{21,782}$

Comparative and consistency metrics:

• Δ_rephrase: $A_O - A_R$ (measures robustness to linguistic inversion)
• Gain_visual: $A_V - A_R$ (effect of visual cues)
• Cross-modal consistency: $\Delta_{\text{consistency}} = | A_R - A_V |$
• R–V Agree %: Proportion of instances where R and V answers match.
• R–V Correct %: Proportion where both R and V are correct.

Human validation uses Cohen's κ for inter-annotator agreement, e.g., for rephrasings, $\kappa \approx 0.76$ (“substantial” agreement).

4. Zero-Shot Model Performance and Analytic Results

Thirteen state-of-the-art VLMs were evaluated end-to-end as zero-shot solvers across all three formats.

Table: BLEnD-Vis Model Performance (excerpt from Table 2)

Model	A_O	A_R	A_V	R–V Agree	R–V Correct
GPT-4o	69.56	63.36	92.01	66.29	60.83
Qwen2.5-VL-32B	61.90	57.32	86.03	63.83	53.59
Kimi-VL-2.8B	57.13	52.22	83.21	61.59	48.51
Mean (overall)	53.97	52.03	69.82	62.53	42.19

4.1 Impact of Linguistic Rephrasing

• Mean Δ_rephrase: 1.94% absolute drop (53.97% → 52.03%), indicating brittleness of prompt generalization.

4.2 Visual Cue Effects

• Gain_visual: 17.79% absolute (52.03% → 69.82%), showing significant improvement with contextually relevant images.

4.3 Cross-Modal Consistency

• Mean R–V Agree: 62.53%
• Mean R–V Correct: 42.19%
• High agreement on incorrect answers in low-resource regions (e.g., North Korea: 8.13% Agree-Incorrect vs. US: 2.00%).

4.4 Regional and Topic Variation

• High-resource region mean VQA: US (74.40%), UK (72.36%), China (70.61%).
• Low-resource: North Korea (46.80%), Algeria (46.94%), Assam (47.22%).
• Entity → Region format increases region performance gap from ~22% (O) to ~42% (R/V).
• By topic (Table 4): "Work life" highest mean (68.18%), "Family" lowest (52.88%).
• VQA format consistently outperforms text-only; e.g., "Education" mean rises 44.57% → 70.67%.

4.5 Cross-Modal Transfer

• Fine-tuning on Rephrased; testing on VQA: mean gain +16.15%.
• Fine-tuning on VQA; testing on Rephrased: mean gain +3.98%.
• Asymmetry suggests richer visual knowledge is not readily transferable back to text-only queries.

5. Failure Modes, Bias, and Human Validation

Comprehensive analysis reveals:

• Rephrasing Fragility: Small Δ_rephrase masks substantial failures on individual instances; reliance on canonical, region → entity patterns limits generalization.
• Cross-Modal Mismatch: Even with high raw VQA accuracy, only 42% of items are answered correctly in both modalities.
• Systematic Regional Bias: Models often default to high-resource region answers for low-resource contexts, especially under rephrasal or cross-modal transfer.
• Qualitative Failure Modes: Examples include cases rescued by visual cues but misled by text, and vice versa, or systematic errors (Agree-Incorrect) on both modalities.
• Human Validation: High inter-annotator agreement (rephrasing κ≈0.76, image QA κ≈0.82); <1.7% absolute VQA accuracy drop when replacing synthetic with human-curated images, validating image quality.

6. Limitations and Prospects for Extension

Major constraints of BLEnD-Vis include:

• English-only setting—extending to multilingual and code-mixed contexts is a recommended future direction.
• Use of synthetic images, which may exhibit generator-specific bias; inclusion of real-world photographs and alternative image generators is suggested.
• Focus on tangible, discriminative cultural knowledge—abstract norms, open-ended dialogue, and generative explanation capabilities are not currently addressed.
• MCQ format prioritizes discriminative over generative reasoning; generative VQA and explanation prompts are not evaluated.
• Placeholder-based visual grounding tests only one paradigm; broader query-image structures could further expose modality integration failures.

7. Benchmark Artifacts and Interpretative Figures

• Table 1: MCQ instance distribution by topic and region, ensuring diversity.
• Table 2: Model-wise zero-shot accuracy and cross-modal agreement.
• Table 3: Per-region performance, highlighting resource-driven disparities.
• Table 4: Topic-level mean accuracies, illuminating domain difficulty.
• Table 5: Cross-modal transfer results, evidencing knowledge asymmetry.
• Figure 1: Schematic of data pipeline and evaluation protocol.
• Figure 2: Visualization of regional variation in VQA accuracy.

BLEnD-Vis establishes a rigorous framework for diagnostic evaluation of cultural robustness in vision-LLMs, systematically uncovering limitations in current systems and providing actionable insights for development toward more robust and culturally competent multimodal models (Tan et al., 13 Oct 2025).