BharatBBQ: Multilingual Bias Benchmark

Updated 8 July 2026

BharatBBQ is a culturally localized question-answering benchmark that measures social biases in the Indian context by spanning eight Indian languages and 13 social categories.
It employs a rigorous methodology using crowdsourced stereotype inventories, machine translation with back-translation validation, and culturally tailored template designs.
The benchmark evaluates bias through ambiguous and disambiguated contexts with metrics like BS and SBS, revealing that Indian languages exhibit higher bias compared to English.

Searching arXiv for BharatBBQ and closely related benchmark papers to ground the article in current literature. BharatBBQ is a multilingual, culturally adapted bias benchmark for question answering that was created specifically to measure social biases in LLMs in the Indian context. It follows the question-answering setup of BBQ, but it is not a translated replica of a U.S.-centric resource; it is a culturally re-authored benchmark whose social ontology, target groups, templates, and linguistic cues are localized to Indian social realities. BharatBBQ covers English plus seven Indian languages—Hindi, Marathi, Bengali, Tamil, Telugu, Odia, and Assamese—spans 13 social categories including three intersectional groups, and contains 323 templates and 49,108 examples per language, for 392,864 examples across eight languages (Tomar et al., 9 Aug 2025).

1. Conceptual basis and localization

BharatBBQ was introduced in response to a specific limitation of prior bias benchmarks for QA: existing resources, especially BBQ, were anchored in Western or U.S. social realities and therefore missed many identities, stereotypes, and linguistic cues salient in India. BBQ’s core methodology—ambiguous versus disambiguated contexts, negative versus non-negative questions, and an explicit unknown option—was retained as the structural basis, but BharatBBQ re-engineered the benchmark around Indian stratification and multilinguality rather than U.S. English-speaking categories (Parrish et al., 2021).

The motivating claim is that India is both socially stratified and deeply multilingual. Models may encode stereotypes about caste, region, religion, nationality, gendered social roles, or Northeastern identity, and these stereotypes are expressed differently across languages through local idioms, honorifics, kinship terms, and culturally specific naming conventions. On this account, a benchmark that merely translates a U.S.-centric dataset would be insufficient.

This localization is substantive rather than cosmetic. The authors remove U.S.-specific categories and templates that do not fit Indian social life, culturally transform templates whose scenarios can be localized, modify target groups to reflect Indian stereotypes, and create many entirely new templates and categories. Antisemitism-focused templates are removed because they are regarded as having negligible footprint in India, and U.S. race-centered Black/white dynamics are omitted because Indian stratification operates differently. The benchmark also substitutes culturally recognizable entities—for example, replacing an American retail chain with Big Bazaar—and introduces India-specific categories such as caste and region, together with intersectional categories such as Religion $\times$ Gender, Region $\times$ Gender, and Age $\times$ Gender (Tomar et al., 9 Aug 2025).

BharatBBQ covers 13 social categories. Ten are single-axis categories—Gender Identity, Age, Religion, Disability Status, Caste, Region, Sexual Orientation, Socio-economic Status, Physical Appearance, and Nationality—and three are explicitly intersectional—Religion $\times$ Gender, Age $\times$ Gender, and Region $\times$ Gender. The dataset statistics reported for one language are as follows (Tomar et al., 9 Aug 2025).

Category	Examples
Age	6,656
Disability Status	5,296
Gender Identity	6,536
Sexual Orientation	904
Socio-Economic	2,336
Physical Appearance	5,980
Religion	4,800
Nationality	3,264
Caste	3,864
Region	3,144
Religion $\times$ Gender	1,504
Region $\times$ Gender	1,944
Age $\times$ Gender	2,880

Each item is a multiple-choice QA example with a context, a question, and three answer options corresponding to the stereotyped group, the non-stereotyped group, and an unknown option. The design varies both context type and question polarity. In an ambiguous context, the context is intentionally under-informative, so the correct answer is always the unknown option. In a disambiguated context, additional information makes one of the two social groups clearly correct. A negative question asks in the direction of the stereotype, while a non-negative question asks in the opposite direction.

Disambiguated items are further partitioned by pairing type. If the disambiguated context makes the stereotyped group the correct answer for the stereotype-consistent question, the item is a Negative Pairing ( $\mathit{NP}$ ). If the context instead assigns the stereotyped property to the non-stereotyped group, it is a Non-Negative Pairing ( $\times$ 0). The expected-answer structure is systematic: ambiguous contexts require “Unknown” for both question polarities; in disambiguated contexts, NP maps the negative question to the stereotyped group and the non-negative question to the non-stereotyped group, whereas NNP reverses that mapping (Tomar et al., 9 Aug 2025).

The category design is explicitly tied to Indian stereotypes. For caste, surnames such as Goyal and Agarwal are used to reflect stereotypes associated with Baniya identity. For region, the paper gives stereotypes such as “South Indians do not prefer to speak in Hindi.” Region $\times$ 1 Gender includes Northeastern women being exoticized or treated as outsiders, North Indian men being caricatured as loud or aggressive, and Tamil women being portrayed as overly traditional. Religion $\times$ 2 Gender includes Muslim women being veiled and submissive, Sikh women being assumed conservative, and Hindu women being framed as passionate devotees. Age $\times$ 3 Gender includes teenage girls depicted as carefree consumers, middle-aged men as dominant breadwinners, and older men as authoritative patriarchs.

The benchmark also treats names as socially informative. In categories such as Religion, Caste, Gender Identity, and Religion $\times$ 4 Gender, Indian first names and surnames themselves can signal social identity. Because caste is often inferable from surname and religion or gender from first name, BharatBBQ includes both examples with proper nouns and variants with common nouns naming the demographic explicitly.

3. Construction pipeline and validation

The stereotype inventory was built through a multi-stage process. The authors first circulated an open-ended Google form across academic and non-academic forums in India and received 241 responses from across Indian regions. Participants reported stereotypes either as tuples such as “<lower caste, poor>” or as free text such as “Chinese products are of low quality.” The open format was chosen to reduce annotator cognitive load and reduce bias introduced by structured questionnaires. Only stereotypes mentioned by at least three respondents were retained (Tomar et al., 9 Aug 2025).

This crowdsourced inventory was supplemented with prior literature and existing resources, including SeeGULL, multilingual SeeGULL, Indian-BHED, and IndiBias. Each stereotype was then manually validated for prevalence through online search, and only stereotypes backed by trustworthy news or research articles were retained. The resulting inventory contains 307 validated stereotype concepts across the 13 categories.

Template construction followed four transformation strategies. “Sample Removed” denotes a BBQ template dropped because it did not fit Indian culture. “Culturally Transformed” localizes a scenario while preserving structural logic. “Target Group Modified” changes the relevant social groups to fit Indian stereotypes. “Newly Created” denotes templates written manually in BBQ-like format using placeholders such as [NAME1] and [NAME2]. Some stereotypes were given multiple scenario variants to test robustness across contexts. The template breakdown is as follows (Tomar et al., 9 Aug 2025).

Template status	Count
Removed	16
Target-modified or culturally transformed	20
Simply transferred	164
Newly created	139

Newly created templates were validated by two independent annotators familiar with Indian social structure. They checked whether the template captured the intended stereotype, whether the negative and non-negative questions were correctly designed, and whether the placeholders represented the right groups. Inter-annotator agreement was strong, with Cohen’s $\times$ 5. Only templates approved by both annotators on all criteria were included; disagreements were resolved by discussion or by discarding the template.

4. Multilingual expansion and corpus scale

BharatBBQ was first authored in English and then translated into Hindi, Marathi, Bengali, Tamil, Telugu, Odia, and Assamese using IndicTransv2. To verify semantic fidelity, the translated examples were back-translated into English and cosine similarity was computed between the original English and the back-translated English using modernBERT embeddings. Translations with cosine similarity above 0.75 were retained, and translations below 0.75 were manually corrected. The paper states that the 0.75 threshold was chosen after manual inspection (Tomar et al., 9 Aug 2025).

The human verification protocol is described in two partially different ways. In the main multilingual extension section, two language-specific annotators for each language assessed a sample of contexts and questions for fluency and adequacy. In the later “Translation Annotation Task” section, one annotator for each of the seven Indian languages was employed for the verification task for low-similarity examples. Taken together, the intended pipeline appears to be machine translation, back-translation similarity filtering, manual correction for low-similarity cases, and separate human quality evaluation on sampled translations.

The human translation study sampled 100 contexts and 100 questions per target language, for 200 annotated sentences per language, and annotators rated adequacy and fluency on 1–5 scales. Reported averages are high across languages. Tamil is near-perfect, often 5.0 in fluency and around 4.94–5.0 in adequacy. Hindi combined scores are around 4.80/4.75 for fluency and 4.83/4.81 for adequacy across two annotators. Assamese is around 4.76/4.80 fluency and 4.85/4.84 adequacy, while Marathi is lower but still high at around 4.55/4.73 fluency and 4.49/4.57 adequacy.

In one language, the benchmark contains 323 templates and 49,108 examples. It also reports 11,600 examples with proper nouns in one language. Because the dataset is replicated across eight languages with the same structure, the total becomes $\times$ 6 examples, with $\times$ 7 proper-noun examples. The paper further notes that BharatBBQ is often larger than BBQ in comparable categories and attributes this to systematic inclusion of both question polarities for both context types, more culturally grounded target-group combinations, and multiple lexical variants of the unknown answer such as “Unknown,” “Not enough information,” and “Cannot be determined” (Tomar et al., 9 Aug 2025).

5. Evaluation protocol and metrics

The evaluation covers five multilingual decoder-based LM families: Llama-3.1-8B-Instruct, Gemma-2-9B-it, Phi-3.5-mini-instruct, Bloomz-7b1, and Sarvam-2b-v0.5. Each model is evaluated on all eight languages under four prompting conditions: target-language instruction zero-shot, target-language instruction two-shot, English-instruction zero-shot, and English-instruction two-shot. The prompt follows an ARC-style multiple-choice format with context, question, and three options, instructing the model to return only the correct answer. Predictions are not extracted by free-form generation heuristics; instead, each answer option is scored independently by average log-likelihood as a continuation of the prompt, and the highest-probability option is selected (Tomar et al., 9 Aug 2025).

The benchmark defines two accuracy measures:

$\times$ 8

$\times$ 9

Here, $\times$ 0 measures abstention correctness in ambiguous contexts, and $\times$ 1 measures standard correctness in disambiguated contexts over both pairing types.

It also defines two bias measures:

$\times$ 2

$\times$ 3

In ambiguous contexts, $\times$ 4 compares stereotyped versus non-stereotyped selections when the model should have answered unknown. In disambiguated contexts, $\times$ 5 measures whether the model is more accurate when the correct answer aligns with stereotype than when it conflicts with stereotype.

BharatBBQ’s novel metric is the Stereotypical Bias Score (SBS), intended to separate stereotype-driven behavior from generic lexical preferences:

$\times$ 6

$\times$ 7

$\times$ 8 counts stereotype-consistent response patterns in ambiguous contexts, and $\times$ 9 isolates stereotype-respecting but incorrect behavior in NNP disambiguated cases. The paper argues that SBS improves on prior BBQ-style bias metrics because a model that systematically prefers one token or one group name could have a modest BS while still exhibiting highly stereotype-congruent behavior (Tomar et al., 9 Aug 2025).

6. Empirical findings, comparison, and limitations

The reported findings are consistent across model families, languages, and categories: bias persists across all evaluated models, and Indian languages often exhibit stronger bias than English. Gemma is the strongest overall on accuracy and is also the least biased on average, with the lowest BS and SBS in both zero-shot and few-shot settings. Bloomz is often the most biased, especially in disambiguated contexts, where it shows very high BS and high SBS. Llama shows substantial bias, particularly in English and in several categories. Sarvam has low accuracy on average for both ambiguous and disambiguated contexts and high SBS, and the paper notes that Sarvam appears nearly unbiased on the original BBQ in some ambiguous settings but not on BharatBBQ, which the authors speculate may reflect benchmark contamination or exposure effects on BBQ (Tomar et al., 9 Aug 2025).

Instruction language matters less than data language. Models generally perform best on English examples with English instructions, but accuracy is nearly invariant between English and target-language instructions in both zero-shot and two-shot settings. Few-shot prompting helps somewhat: SBS generally decreases in few-shot compared to zero-shot across models and languages, suggesting that in-context examples can mitigate reliance on stereotypes, though they do not remove it.

At the language level, Bengali and Tamil stand out in aggregate plots as among the more biased Indian languages, and the paper explicitly states that Indian languages have higher BS than English overall. It provides approximate examples: for Gemma in zero-shot ambiguous contexts, BS rises from about 0.025 in English to about 0.15 in Bengali; for Bloomz, BS rises from about 0.1 in English to about 0.25 in Telugu. The broader interpretation offered is that limited pretraining data and weaker contextual understanding in Indian languages make models more likely to default to stereotypes, especially in disambiguated settings.

At the category level, disability status is the clearest high-bias category across models and languages. Sexual orientation is another strongly biased category, especially in Telugu and in English for disambiguated contexts, and physical appearance is also high-bias, especially in Telugu. Religion illustrates the value of SBS: average BS in disambiguated contexts may appear low while SBS remains substantial, indicating stereotype alignment that ordinary bias score can miss.

Direct comparison with the English subset of BBQ shows that bias and stereotypical bias are generally higher on BharatBBQ than on BBQ, particularly in disambiguated contexts. The paper gives concrete examples: Llama’s ambiguous-context BS for Disability Status rises from 0.11 on BBQ to 0.33 on BharatBBQ in zero-shot, and Gemma’s ambiguous-context SBS for Age rises from 0.25 on BBQ to 0.47 on BharatBBQ. This comparison is central to the benchmark’s argument that culturally grounded evaluation surfaces failures that a Western-centric benchmark can miss (Parrish et al., 2021).

The proper-noun analysis finds that proper nouns and common nouns produce similar bias patterns on average, but some categories become more biased with proper names, especially religion for Gemma, Llama, and Bloomz. Model scaling is not monotonic: larger Gemma variants show increases in BS and SBS, while larger Llama and Sarvam variants show modest bias reductions. The paper therefore concludes that fairness effects of scale are architecture-specific rather than uniform.

The limitations are explicit. BharatBBQ is broad but not exhaustive; it does not capture all biases in Indian society, all demographic groups, all dialects, or all linguistic varieties. It evaluates five LM families and is an evaluation benchmark rather than a mitigation method. The paper also does not investigate the causes of why Indian-language bias is often higher than English bias. Ethically, it stresses that the dataset is intended for bias measurement rather than stereotype propagation, that sensitive categories such as caste and religion are included only for research purposes, and that proper names are used to capture linguistic and cultural nuance rather than to target real individuals.

Subsequent work has already treated BharatBBQ as a methodological base for further India-grounded bias auditing. “ImplicitBBQ: Benchmarking Implicit Bias in LLMs through Characteristic Based Cues” explicitly states that it is built from BBQ and BharatBBQ, with BharatBBQ used for the caste dimension, and extends the explicit-label QA paradigm to characteristic-based implicit cues such as “wears Janeu” or “prays namaz” (Vedula et al., 2 Apr 2026). This situates BharatBBQ within a growing family of culturally localized and technically controlled bias benchmarks whose central claim is that fairness evaluation must be both linguistically and socially grounded.