RICE-VL: Cultural Competence in VLMs

• RICE-VL is a benchmark suite evaluating vision-language models’ cultural understanding across 11 ASEAN nations using tasks like CulturalVQA and Visual Grounding.
• It introduces the SEA-LAVE metric to measure semantic accuracy and cultural specificity, revealing biases and performance gaps in low-resource contexts.
• The dataset is meticulously curated with culturally emblematic imagery and expert annotations, capturing diverse domains such as architecture, traditional attire, and festivals.

RICE-VL is a benchmark suite designed to rigorously evaluate the cultural understanding capabilities of Vision-LLMs (VLMs) across the 11 nations of the Association of Southeast Asian Nations (ASEAN). Distinct from prior benchmarks, RICE-VL systematically measures both semantic and specifically cultural competence, revealing critical limitations in current state-of-the-art VLMs with respect to non-Western, culturally rich contexts. Its methodology incorporates finely calibrated evaluation protocols, a novel metric (SEA-LAVE), and meticulously curated data from diverse Southeast Asian domains (Pranav et al., 1 Dec 2025).

1. Dataset Construction and Curation

RICE-VL comprises two core tasks: Cultural Visual Question Answering (CulturalVQA) and Cultural Visual Grounding. The dataset spans Brunei, Cambodia, Indonesia, Laos, Malaysia, Myanmar, Philippines, Singapore, Thailand, Timor-Leste, and Vietnam.

• CulturalVQA: Over 28,000 human-curated samples across three formats—True/False, Fill-in-the-Blank, and open-ended questions. Domains include diverse cultural settings, practices, and symbolism.
• Visual Grounding: 1,000 image–bounding box pairs (990 verified samples), with culturally informed box annotations covering 14 high-level sub-categories:

| Sub-category | Focus | |-------------------------------|----------------------------------------| | Architecture & Heritage | Built environment, monuments | | Clothing & Attire | Traditional dress, textiles | | Dance & Music | Performances, instruments | | Drinks | Beverages, ceremonial drinks | | Festivals | Regional festivals, rituals | | Food & Desserts | Local dishes, culinary traditions | | Language Signs & Literature | Scripts, signage, literature | | Marriage Customs | Wedding rituals, attire | | Notable Key Figures | Prominent cultural/historical figures | | Painting | Artistic works | | Religious Practices | Ritual items, ceremonies | | Places of Worship | Religious sites | | Traditional Games | Region-specific games | | Transport | Culturally unique transport modes |

Curatorial procedures involved targeted web-scraping of culturally emblematic imagery and stratified sampling across countries and subdomains. Six Southeast Asia–based annotators performed 720 hours of annotation, followed by multi-stage expert review, ensuring cultural fidelity and inter-annotator consistency. Bounding boxes were designed to isolate the specific cultural marker (e.g., Batik motif on textiles), discarding irrelevant background (Pranav et al., 1 Dec 2025).

2. The SEA-LAVE Metric

SEA-LAVE (Southeast Asia Linguistic Agreement with Visual Evidence) provides a composite measure of VLM performance, emphasizing not only textual accuracy but also cultural specificity and precise country identification. This metric generalizes the LAVE protocol for the Southeast Asian context.

The SEA-LAVE score for a question–response pair is defined as: $$\text{SEA-LAVE} = \frac{\mathrm{TU} + \mathrm{CU} + (\mathrm{CI}/2)}{3}$$ where each variable is assigned as:

• $\mathrm{TU}, \mathrm{CU} \in \{0,1\}$: Text Understanding (semantic match), Cultural Understanding (presence of tradition-specific information)
• $\mathrm{CI} \in \{0,1,2\}$: Country Identification (2 for correct SEA country, 1 for any other SEA country, 0 otherwise)

Thus, all components are normalized to $[0,1]$, with SEA-LAVE per sample also in $[0,1]$. For aggregate reporting, sample-level SEA-LAVE values are averaged over task, country, and prompt variant, yielding a comprehensive model-level performance profile. (Pranav et al., 1 Dec 2025)

3. Model Evaluation Protocol and Results

The evaluation framework encompasses both open-source (OS) and closed-source (CS) multimodal LLMs:

	Qwen-VL 2.5 (7B)	Ovis 2 (8B)	LLaMA 3.2 (11B)	Ola (7B)	GPT-4O (≈175B)	Claude-3-Opus (≈100B)
Type	OS	OS	OS	OS	CS	CS

CulturalVQA SEA-LAVE Results: Average performance across countries under two prompt regimes:

• Global prompt: General instruction to answer.
• SEA-specific prompt: Explicitly states the context as Southeast Asian.

Selected results:

	Claude-3-Opus	GPT-4O	LLaMA 3.2	Ola	Ovis 2	Qwen-VL 2.5
Brunei	0.58	0.55	0.50	0.33	0.53	0.44
Indonesia	0.73	0.74	0.66	0.54	0.64	0.62
Timor-Leste	0.40	0.26	0.21	0.17	0.19	0.20

SEA-specific prompt effects: For Ola on Thailand, SEA-LAVE improved from 0.59 (global) to 0.87 (SEA-specific), confirming geographic/priming cues facilitate cultural alignment.

Key empirical observations:

• Closed-source models outperform open-source by 5–15 SEA-LAVE points, especially in data-rich ASEAN countries.
• Performance drastically degrades in low-resource contexts (Timor-Leste, Brunei, Laos), where even GPT-4O and Claude-3-Opus attain only 0.30–0.50.
• Cultural accuracy is lower in abstract/symbolic domains (Religious Practices, Notable Key Figures) than in concrete categories (Food & Desserts, Clothing).

These results identify not only a Western-centric bias, but also domain-targeted gaps in current VLM generalization (Pranav et al., 1 Dec 2025).

4. Visual Grounding Task Performance

Visual grounding evaluation employs the standard Intersection-over-Union (IoU) metric: $$\mathrm{IoU} = \frac{|R_{\text{pred}} \cap R_{\text{gt}}|}{|R_{\text{pred}} \cup R_{\text{gt}}|}$$ with a threshold of $\mathrm{IoU} \geq 0.5$ for a correct match; mean IoU characterizes category and country-level results.

Model	Brunei	Cambodia	Indonesia	Laos	Malaysia	Myanmar	Vietnam
Qwen2.5 VL 3B	0.546	0.317	0.551	0.535	0.548	0.458	0.440
Qwen2.5 VL 7B	0.531	0.312	0.494	0.506	0.510	0.412	0.390
Paligemma2 10B	0.408	0.222	0.520	0.434	0.475	0.393	0.282
Kosmos2	0.421	0.264	0.523	0.449	0.492	0.369	0.327
GroundingDino	0.380	0.271	0.452	0.488	0.438	0.469	0.343

Sub-category insights:

• Highest mean IoUs: Clothing & Attire, Festivals, Transport; attributable to large and visually distinctive objects.
• Lowest mean IoUs: Notable Key Figures, Painting, Religious Practices; these involve finer, more abstract spatial cues and often symbol-dense compositions.

Concrete successes include Qwen2.5-VL accurately localizing Batik motifs (Malaysia/Indonesia) and Thai “chada” headdress; failures often involve confusion between visually similar foods (e.g., generic bread vs. “kaya toast”) or mis-localization in visually cluttered religious scenes.

5. Limitations, Cultural Biases, and Recommendations

Systematic analysis of benchmark results exposes persistent Western-centric bias, with all six models underperforming on Southeast Asian-specific data relative to their Western-targeted evaluations. Notable error modes include:

• Poor generalization to under-represented ASEAN nations (lowest SEA-LAVE in Timor-Leste, Brunei, Laos)
• Difficulty with abstract or symbolic cultural constructs (Marriage Customs, Language Signs & Literature, Religious Practices), evidenced by misattribution to visually but not semantically similar instances

Recommended strategies for advancing model inclusivity:

1. Increase inclusion of SEA-centric image–text pairs in training sets, with native language and dialectal coverage.
2. Employ human-in-the-loop expert annotation to resolve cultural subtleties, capturing minority and indigenous category variance.
3. Apply prompt engineering and regional priming to enhance geographical/cultural conditioning at inference time.
4. Expand to higher-order cultural reasoning (e.g., folklore retelling, context-rich cross-modal dialogue), moving beyond surface-level matching.

6. Future Directions and Expansions

RICE-VL’s framework highlights the necessity for broader, multi-lingual benchmarks and more generalizable evaluation tools:

• Integration of question prompts and answer keys in local ASEAN languages (e.g., Bahasa, Khmer, Lao) to explicitly assess vision–language interplay in non-English settings.
• Enlargement of the visual grounding corpus, targeting minority/indigenous subcultures to further mitigate geographic and cultural coverage gaps.
• Extension to “distilled” VLMs and low-compute architectures suitable for resource-constrained, on-device deployments.
• Inclusion of dynamic tasks (e.g., dance ritual videos) to facilitate temporal visual grounding, aligning with real-world multimodal reasoning needs.

RICE-VL, in conjunction with SEA-LAVE, establishes a gold standard for quantifying VLMs’ cultural competence, driving the development of equitable multimodal AI adapted to the full diversity of global cultures (Pranav et al., 1 Dec 2025).