Seeing Culture Benchmark (SCB)

• Seeing Culture Benchmark (SCB) is a structured test integrating visual reasoning with spatial grounding, focusing on Southeast Asian cultural artifacts.
• It employs a two-stage task where models first select the correct visual response and then segment the relevant cultural evidence.
• SCB reveals key performance gaps between reasoning accuracy and precise artifact localization, informing improvements in vision-language models.

The Seeing Culture Benchmark (SCB) is a benchmark for testing whether vision-LLMs can do more than recognize culturally marked objects: it requires them to reason on culturally rich images and then ground that reasoning spatially in the image. Introduced in "Seeing Culture: A Benchmark for Visual Reasoning and Grounding" (Satar et al., 20 Sep 2025), SCB is organized as a two-stage task in which a model first selects the correct visual option in multiple-choice visual question answering and then, only after a correct choice, segments the relevant cultural artifact as evidence. The benchmark contains 1,065 images, 138 cultural artifacts, five categories, seven Southeast Asian countries, and 3,178 questions, of which 1,093 are unique and meticulously curated by human annotators (Satar et al., 20 Sep 2025).

1. Scope and research objective

SCB is motivated by the claim that existing cultural benchmarks often measure cultural understanding in a relatively shallow way. The paper states that prior resources may use text-only answer choices, simpler images without much distractor content, AI-generated questions, or may stop at answering a question without asking the model to justify its choice by locating the relevant cultural evidence (Satar et al., 20 Sep 2025). SCB is designed to test cultural reasoning rather than simple object naming, and to couple that reasoning with grounding.

A central design decision is the focus on underrepresented Southeast Asian cultures. The benchmark covers Cambodia, Myanmar, Indonesia, Vietnam, the Philippines, Malaysia, and Thailand, and its five cultural categories are music, game, dance, celebration, and wedding (Satar et al., 20 Sep 2025). The benchmark’s taxonomy is hierarchical: country/category/concept.

Component	Value
Images	1,065
Cultural artifacts or concepts	138
Categories	music, game, dance, celebration, wedding
Countries	Cambodia, Myanmar, Indonesia, Vietnam, the Philippines, Malaysia, Thailand
Questions	3,178
Unique human-curated questions	1,093

This organization makes SCB a benchmark for cross-modal cultural reasoning rather than a standard VQA dataset. A recurring misconception in cultural VLM evaluation is that correct answer selection is sufficient evidence of understanding. SCB is explicitly constructed against that assumption: the model must also identify the supporting artifact in the image (Satar et al., 20 Sep 2025).

2. Two-stage task formulation

SCB is built as a two-stage pipeline. In stage one, the model performs multiple-choice visual question answering: it receives a question plus four visual options and must choose the correct image. In stage two, the model must segment the relevant cultural artifact in the image as evidence for its earlier answer. Progression to stage two occurs only after a correct answer in stage one (Satar et al., 20 Sep 2025).

The visual options in stage one are organized into three types. Type 1 is within culture or same-country: all answer options come from the same country as the correct answer and the same cultural category. Type 2 is across culture or different-country: all options are from different countries than the correct answer, but still within the same category. Type 3 is mixed culture: it combines same-country and different-country distractors in one question, using a rule-based swapping procedure (Satar et al., 20 Sep 2025). The paper is explicit that all options for each question type come from a single category only, so the task cannot be solved by recognizing only the broad class of artifact.

The second stage uses semantic segmentation using polygons rather than bounding boxes, because the goal is to capture intricate cultural details more precisely (Satar et al., 20 Sep 2025). This choice is significant because many cultural artifacts are not well represented by coarse rectangular localization. The paper’s formulation therefore links visual comparison, cultural interpretation, and fine-grained spatial evidence within a single evaluation pipeline.

SCB’s question design reinforces this objective. Annotators write unique, non-templated questions that do not directly name the artifact. Instead, the questions ask about symbols, meanings, or cultural significance. This shifts the task away from label lookup and toward culturally situated inference (Satar et al., 20 Sep 2025).

3. Dataset construction and annotation

The concept inventory was created in a human-verified pipeline. The authors first used ChatGPT to suggest candidate cultural concepts for each country and category, and then verified and refined them with native speakers and local cultural experts. The paper reports two rounds of verification, involving two to three respondents from each country, and retains only concepts that reach unanimous agreement (Satar et al., 20 Sep 2025). The resulting representation is not limited to isolated objects; some concepts involve multiple characters or objects, such as dances consisting of several performers.

Image collection followed a similarly selective process. The authors crawled Google Images using the selected concepts, initially gathering about 150 images per concept. Human annotators then filtered images by several criteria: the image had to be relevant to the concept, depict a real-world scene, avoid duplicates, not obscure the artifact too much, and include distracting objects or scenes that can confuse shallow matching while still leaving the target artifact identifiable (Satar et al., 20 Sep 2025). The paper states that this procedure reduces roughly 20,000 crawled images down to about 4,000 before final curation, and that 32 images are incorporated from SEA-VL.

For grounding, annotators used the Make Sense tool to produce polygon masks for the cultural artifact or its associated characters or objects (Satar et al., 20 Sep 2025). For question formulation, annotators also provided rationales using online sources or their cultural knowledge, and they were instructed not to use AI-generated sources when composing the questions.

The benchmark’s multiple-choice generation is carefully constrained. For Type 1, distractors are sampled from the same country and category as the correct answer. For Type 2, distractors are sampled from different countries but the same category. For Type 3, the benchmark mixes same-country and different-country options by swapping choices from Type 1 and Type 2, while limiting repeated use of questions and choice combinations (Satar et al., 20 Sep 2025). The paper also mentions an avoid list to prevent overlap between culturally similar items across countries.

4. Evaluation protocol and empirical results

SCB is evaluated in a zero-shot setting on a range of open-source and closed-source vision-LLMs. The open-source models include InstructBLIP, Idefics2, Llama-3.2, LLaVA-OneVision, MiniCPM-2.6, InternVL2.5-4B, and Qwen2.5-VL-7B. The closed-source models are GPT-4.1, Gemini-2.5-Pro, and GPT-o3 (Satar et al., 20 Sep 2025).

In stage one, the metric is accuracy, as in standard multiple-choice VQA. In stage two, the paper evaluates grounding using mean IoU, comparing predicted regions to the polygon masks, but at the bounding-box level because current VLMs with segmentation capability are constrained to bounding-box-style grounding (Satar et al., 20 Sep 2025). The paper defines overlap as IoU and reports mean IoU.

The main result is a large gap between visual reasoning and spatial grounding. Among open-source models, Qwen2.5-VL-7B is the strongest with 53.78% accuracy and 47.20 mean IoU. InternVL2.5-4B reaches 31.34% accuracy overall with 28.56 mean IoU, and MiniCPM-2.6 reaches 32.13% accuracy overall. At the lower end, InstructBLIP is near chance at 10.86% overall, while Idefics2 reaches 12.21% accuracy overall with very low IoU (Satar et al., 20 Sep 2025).

The closed-source models perform substantially better on the reasoning stage: GPT-4.1 achieves 81.97% overall accuracy, Gemini-2.5-Pro reaches 82.88%, and GPT-o3 reaches 85.15%, the best result in the benchmark. Their grounding scores, however, are much lower relative to answer accuracy: GPT-4.1 has 13.74 mean IoU, Gemini-2.5-Pro has 16.22, and GPT-o3 has 31.78 (Satar et al., 20 Sep 2025). This disparity is one of the benchmark’s defining empirical findings.

Performance also varies systematically by question type. VLMs perform worst on Type 1, best on Type 2, and in between on Type 3 (Satar et al., 20 Sep 2025). This indicates that same-country, same-category discrimination is substantially harder than choosing among cross-country alternatives. By category, models generally do best on dance and worst on celebration. The paper further reports that direct grounding of cultural objects yields better IoU than culturally reasoned grounding, with an average drop of about 16% in mean IoU when grounding is tied to reasoning rather than simple object localization (Satar et al., 20 Sep 2025).

The distribution of question types reflects the benchmark’s construction: 834 Type 1 questions, 840 Type 2 questions, and 1,504 Type 3 questions. The paper explains that 259 Type 1 questions and 253 Type 2 questions are excluded from analysis, and that Type 3 includes 871 unique questions plus 633 repeated instances generated under the benchmark’s swapping procedure (Satar et al., 20 Sep 2025).

5. Position within cultural and multimodal benchmarking

SCB is situated among a rapidly expanding set of cultural benchmarks, but its contribution is unusually specific. The paper contrasts it with datasets such as CulturalVQA, SEA-VQA, CVQA, CROPE, and parts of CultureVerse or GIMMICK, which focus on cultural VQA but usually with text answers and often AI-generated or partially AI-generated questions. It also contrasts SCB with GlobalRG and parts of CVLUE, which include grounding but use bounding boxes and simpler prompts, and do not couple grounding with a preceding reasoning stage (Satar et al., 20 Sep 2025).

A useful point of comparison is CROPE, which evaluates culture-specific concepts and in-context adaptation through a binary visual question answering benchmark with 1,060 examples from Indonesian, Swahili, Tamil, Turkish, and Chinese sources, and shows that models struggle to use textual and visual context to bind culture-specific concepts to depictions (Nikandrou et al., 2024). SCB differs by requiring visual-option selection among culturally similar alternatives and by making grounding contingent on correct reasoning.

Later multimodal cultural benchmarks broaden the design space rather than supersede SCB. C$^3$B uses comics as the visual medium and introduces 2,220 images and 18,789 QA pairs across Extraction@Culture, Conflict@Culture, and Generation@Culture, emphasizing progressed difficulty and multilingual generation (Song et al., 27 Sep 2025). DRISHTIKON shifts attention to Indian culture, with 15 languages, all states and union territories, and over 64,000 aligned text-image pairs (Maji et al., 23 Sep 2025). These developments suggest that SCB occupies a distinctive position: it is not a multilingual benchmark, a text-only cultural QA resource, or a synthetic comic-based benchmark, but a benchmark for cultural reasoning plus grounding in real-world images.

6. Interpretation, misconceptions, and significance

SCB’s central finding is that selecting the correct cultural image does not mean the model truly grounded its reasoning in the right cultural evidence (Satar et al., 20 Sep 2025). The paper’s qualitative analysis includes cases in which all evaluated models fail to choose the correct visual option among same-country alternatives, and cases in which GPT-o3 answers the multiple-choice question correctly but grounds to the wrong object in the image. The benchmark therefore distinguishes between answer-level success and evidence-level success.

This distinction is important in the broader methodological debate about cultural evaluation. "Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens" argues that many cultural benchmarks reduce culture to static facts or homogeneous national values, and identifies recurring problems such as conflating nation-states with culture, assuming cultural consensus, and stripping context from cultural scenarios (AlKhamissi et al., 7 Oct 2025). SCB addresses part of this critique by using contextual questions about symbols, meanings, and cultural significance, human-verified concepts, and real-world scenes with distractors (Satar et al., 20 Sep 2025). A plausible implication is that SCB moves cultural evaluation closer to situated visual reasoning than to factual recall alone.

At the same time, SCB remains explicitly organized by country/category/concept, and its coverage is limited to seven Southeast Asian countries and five categories (Satar et al., 20 Sep 2025). This focus is deliberate rather than incidental: the paper argues that Southeast Asia is diverse and culturally rich, yet not well covered in existing benchmarks, and that building SCB around these cultures helps expand the evaluation landscape and reduce the tendency of VLMs to perform well only on more heavily represented cultures (Satar et al., 20 Sep 2025).

SCB’s significance, accordingly, lies less in scale than in task formulation. It forces a model to connect visual recognition, cultural knowledge, and spatial evidence in a single benchmark. Within the literature on multimodal cultural evaluation, its contribution is to show that cultural reasoning and cultural grounding are separable capabilities, and that current models remain substantially weaker on their conjunction than on either component in isolation (Satar et al., 20 Sep 2025).