---
title: Seeing Culture Benchmark (SCB)
url: https://www.emergentmind.com/topics/seeing-culture-benchmark-scb
type: topic
---

# Seeing Culture Benchmark (SCB)

The **Seeing Culture Benchmark (SCB)** is a benchmark for testing whether vision-language models can do more than recognize culturally marked objects: it requires them to reason on culturally rich images and then ground that reasoning spatially in the image. Introduced in "Seeing Culture: A Benchmark for Visual Reasoning and Grounding" [2509.16517], SCB is organized as a two-stage task in which a model first selects the correct visual option in multiple-choice visual question answering and then, only after a correct choice, segments the relevant cultural artifact as evidence. The benchmark contains **1,065 images**, **138 cultural artifacts**, **five categories**, **seven Southeast Asian countries**, and **3,178 questions**, of which **1,093 are unique and meticulously curated by human annotators** [2509.16517].

## 1. Scope and research objective

SCB is motivated by the claim that existing cultural benchmarks often measure cultural understanding in a relatively shallow way. The paper states that prior resources may use **text-only answer choices**, **simpler images without much distractor content**, **AI-generated questions**, or may stop at answering a question without asking the model to justify its choice by locating the relevant cultural evidence [2509.16517]. SCB is designed to test cultural reasoning rather than simple object naming, and to couple that reasoning with grounding.

A central design decision is the focus on **underrepresented Southeast Asian cultures**. The benchmark covers **Cambodia, Myanmar, Indonesia, Vietnam, the Philippines, Malaysia, and Thailand**, and its five cultural categories are **music, game, dance, celebration, and wedding** [2509.16517]. The benchmark’s taxonomy is hierarchical: **country/category/concept**.

| Component | Value |
|---|---|
| Images | 1,065 |
| Cultural artifacts or concepts | 138 |
| Categories | music, game, dance, celebration, wedding |
| Countries | Cambodia, Myanmar, Indonesia, Vietnam, the Philippines, Malaysia, Thailand |
| Questions | 3,178 |
| Unique human-curated questions | 1,093 |

This organization makes SCB a benchmark for **cross-modal cultural reasoning** rather than a standard VQA dataset. A recurring misconception in cultural VLM evaluation is that correct answer selection is sufficient evidence of understanding. SCB is explicitly constructed against that assumption: the model must also identify the supporting artifact in the image [2509.16517].

## 2. Two-stage task formulation

SCB is built as a **two-stage pipeline**. In **stage one**, the model performs multiple-choice visual question answering: it receives a question plus **four visual options** and must choose the correct image. In **stage two**, the model must **segment the relevant cultural artifact in the image as evidence** for its earlier answer. Progression to stage two occurs only after a correct answer in stage one [2509.16517].

The visual options in stage one are organized into **three types**. **Type 1** is **within culture** or **same-country**: all answer options come from the same country as the correct answer and the same cultural category. **Type 2** is **across culture** or **different-country**: all options are from different countries than the correct answer, but still within the same category. **Type 3** is **mixed culture**: it combines same-country and different-country distractors in one question, using a rule-based swapping procedure [2509.16517]. The paper is explicit that all options for each question type come from a **single category only**, so the task cannot be solved by recognizing only the broad class of artifact.

The second stage uses **semantic segmentation using polygons rather than bounding boxes**, because the goal is to capture intricate cultural details more precisely [2509.16517]. This choice is significant because many cultural artifacts are not well represented by coarse rectangular localization. The paper’s formulation therefore links **visual comparison**, **cultural interpretation**, and **fine-grained spatial evidence** within a single evaluation pipeline.

SCB’s question design reinforces this objective. Annotators write **unique, non-templated questions** that **do not directly name the artifact**. Instead, the questions ask about **symbols, meanings, or cultural significance**. This shifts the task away from label lookup and toward culturally situated inference [2509.16517].

## 3. Dataset construction and annotation

The concept inventory was created in a human-verified pipeline. The authors first used **ChatGPT** to suggest candidate cultural concepts for each country and category, and then **verified and refined** them with **native speakers and local cultural experts**. The paper reports **two rounds of verification**, involving **two to three respondents from each country**, and retains only concepts that reach **unanimous agreement** [2509.16517]. The resulting representation is not limited to isolated objects; some concepts involve **multiple characters or objects**, such as dances consisting of several performers.

Image collection followed a similarly selective process. The authors crawled **Google Images** using the selected concepts, initially gathering **about 150 images per concept**. Human annotators then filtered images by several criteria: the image had to be **relevant to the concept**, depict a **real-world scene**, **avoid duplicates**, **not obscure the artifact too much**, and include **distracting objects or scenes that can confuse shallow matching** while still leaving the target artifact identifiable [2509.16517]. The paper states that this procedure reduces **roughly 20,000 crawled images down to about 4,000 before final curation**, and that **32 images are incorporated from SEA-VL**.

For grounding, annotators used the **Make Sense** tool to produce **polygon masks** for the cultural artifact or its associated characters or objects [2509.16517]. For question formulation, annotators also provided **rationales** using online sources or their cultural knowledge, and they were instructed **not to use AI-generated sources** when composing the questions.

The benchmark’s multiple-choice generation is carefully constrained. For **Type 1**, distractors are sampled from the **same country and category** as the correct answer. For **Type 2**, distractors are sampled from **different countries but the same category**. For **Type 3**, the benchmark mixes same-country and different-country options by swapping choices from Type 1 and Type 2, while limiting repeated use of questions and choice combinations [2509.16517]. The paper also mentions an **avoid list** to prevent overlap between culturally similar items across countries.

## 4. Evaluation protocol and empirical results

SCB is evaluated in a **zero-shot** setting on a range of open-source and closed-source vision-language models. The open-source models include **InstructBLIP, Idefics2, Llama-3.2, LLaVA-OneVision, MiniCPM-2.6, InternVL2.5-4B, and Qwen2.5-VL-7B**. The closed-source models are **GPT-4.1, Gemini-2.5-Pro, and GPT-o3** [2509.16517].

In stage one, the metric is **accuracy**, as in standard multiple-choice VQA. In stage two, the paper evaluates grounding using **mean IoU**, comparing predicted regions to the polygon masks, but **at the bounding-box level** because current VLMs with segmentation capability are constrained to bounding-box-style grounding [2509.16517]. The paper defines overlap as IoU and reports mean IoU.

The main result is a large gap between **visual reasoning** and **spatial grounding**. Among open-source models, **Qwen2.5-VL-7B** is the strongest with **53.78% accuracy** and **47.20 mean IoU**. **InternVL2.5-4B** reaches **31.34% accuracy overall with 28.56 mean IoU**, and **MiniCPM-2.6** reaches **32.13% accuracy overall**. At the lower end, **InstructBLIP** is near chance at **10.86% overall**, while **Idefics2** reaches **12.21% accuracy overall with very low IoU** [2509.16517].

The closed-source models perform substantially better on the reasoning stage: **GPT-4.1** achieves **81.97% overall accuracy**, **Gemini-2.5-Pro** reaches **82.88%**, and **GPT-o3** reaches **85.15%**, the best result in the benchmark. Their grounding scores, however, are much lower relative to answer accuracy: **GPT-4.1** has **13.74 mean IoU**, **Gemini-2.5-Pro** has **16.22**, and **GPT-o3** has **31.78** [2509.16517]. This disparity is one of the benchmark’s defining empirical findings.

Performance also varies systematically by question type. VLMs perform **worst on Type 1**, **best on Type 2**, and **in between on Type 3** [2509.16517]. This indicates that same-country, same-category discrimination is substantially harder than choosing among cross-country alternatives. By category, models generally do **best on dance** and **worst on celebration**. The paper further reports that direct grounding of cultural objects yields better IoU than culturally reasoned grounding, with an **average drop of about 16% in mean IoU** when grounding is tied to reasoning rather than simple object localization [2509.16517].

The distribution of question types reflects the benchmark’s construction: **834 Type 1 questions**, **840 Type 2 questions**, and **1,504 Type 3 questions**. The paper explains that **259 Type 1 questions** and **253 Type 2 questions** are excluded from analysis, and that Type 3 includes **871 unique questions** plus **633 repeated instances** generated under the benchmark’s swapping procedure [2509.16517].

## 5. Position within cultural and multimodal benchmarking

SCB is situated among a rapidly expanding set of cultural benchmarks, but its contribution is unusually specific. The paper contrasts it with datasets such as **CulturalVQA, SEA-VQA, CVQA, CROPE, and parts of CultureVerse or GIMMICK**, which focus on cultural VQA but usually with **text answers** and often **AI-generated or partially AI-generated questions**. It also contrasts SCB with **GlobalRG** and parts of **CVLUE**, which include grounding but use **bounding boxes** and **simpler prompts**, and do not couple grounding with a preceding reasoning stage [2509.16517].

A useful point of comparison is **CROPE**, which evaluates culture-specific concepts and in-context adaptation through a **binary visual question answering benchmark** with **1,060 examples** from **Indonesian, Swahili, Tamil, Turkish, and Chinese** sources, and shows that models struggle to use textual and visual context to bind culture-specific concepts to depictions [2410.15453]. SCB differs by requiring **visual-option selection** among culturally similar alternatives and by making **grounding contingent on correct reasoning**.

Later multimodal cultural benchmarks broaden the design space rather than supersede SCB. **C\(^3\)B** uses **comics** as the visual medium and introduces **2,220 images** and **18,789 QA pairs** across **Extraction@Culture, Conflict@Culture, and Generation@Culture**, emphasizing progressed difficulty and multilingual generation [2510.00041]. **DRISHTIKON** shifts attention to Indian culture, with **15 languages**, **all states and union territories**, and **over 64,000 aligned text-image pairs** [2509.19274]. These developments suggest that SCB occupies a distinctive position: it is not a multilingual benchmark, a text-only cultural QA resource, or a synthetic comic-based benchmark, but a benchmark for **cultural reasoning plus grounding** in real-world images.

## 6. Interpretation, misconceptions, and significance

SCB’s central finding is that **selecting the correct cultural image does not mean the model truly grounded its reasoning in the right cultural evidence** [2509.16517]. The paper’s qualitative analysis includes cases in which all evaluated models fail to choose the correct visual option among same-country alternatives, and cases in which **GPT-o3** answers the multiple-choice question correctly but grounds to the wrong object in the image. The benchmark therefore distinguishes between **answer-level success** and **evidence-level success**.

This distinction is important in the broader methodological debate about cultural evaluation. "Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens" argues that many cultural benchmarks reduce culture to **static facts** or **homogeneous national values**, and identifies recurring problems such as **conflating nation-states with culture**, **assuming cultural consensus**, and **stripping context from cultural scenarios** [2510.05931]. SCB addresses part of this critique by using **contextual questions about symbols, meanings, and cultural significance**, **human-verified concepts**, and **real-world scenes with distractors** [2509.16517]. A plausible implication is that SCB moves cultural evaluation closer to situated visual reasoning than to factual recall alone.

At the same time, SCB remains explicitly organized by **country/category/concept**, and its coverage is limited to **seven Southeast Asian countries** and **five categories** [2509.16517]. This focus is deliberate rather than incidental: the paper argues that Southeast Asia is **diverse and culturally rich, yet not well covered in existing benchmarks**, and that building SCB around these cultures helps expand the evaluation landscape and reduce the tendency of VLMs to perform well only on more heavily represented cultures [2509.16517].

SCB’s significance, accordingly, lies less in scale than in task formulation. It forces a model to connect **visual recognition**, **cultural knowledge**, and **spatial evidence** in a single benchmark. Within the literature on multimodal cultural evaluation, its contribution is to show that cultural reasoning and cultural grounding are separable capabilities, and that current models remain substantially weaker on their conjunction than on either component in isolation [2509.16517].

Source: https://www.emergentmind.com/topics/seeing-culture-benchmark-scb