Bias Benchmark for Question Answering

• The paper introduces the BBQ framework that uses ambiguous versus disambiguated contexts with uncertainty choices to detect when QA systems resort to stereotypes over evidence-based answers.
• It systematically compares error rates across models, revealing that many systems frequently select stereotype-aligned responses in ambiguous scenarios.
• Cultural adaptations to other languages incorporate localized stereotypes, validating the framework's robustness and adaptability across diverse linguistic contexts.

A bias benchmark for question answering is an evaluation framework that measures whether a QA system follows the evidence supplied by a context or instead defaults to socially or structurally biased shortcuts. In contemporary NLP, the canonical example is BBQ, a hand-built multiple-choice benchmark with an explicit “unknown/cannot be determined” option, ambiguous and disambiguated contexts, and stereotype-targeted social categories; later work has extended this formulation to Korean, German, Hindi and other Indian languages, Urdu, Filipino, Spanish, and Catalan, while related research has broadened the notion of QA bias benchmarking to include demographic counterfactuals and protocol artifacts such as answer-position and option-selection shortcuts (Parrish et al., 2021, Jin et al., 2023, Ruiz-Fernández et al., 15 Jul 2025).

1. Origins in BBQ and the output-centered view of bias

The foundational formulation is "BBQ: A Hand-Built Bias Benchmark for Question Answering" (Parrish et al., 2021). BBQ was introduced to study how social biases appear in QA outputs rather than only in latent preference scores. Its design targets representational harms that arise when systems reinforce group subordination through stereotypical associations. In contrast to evaluations that compare only likelihoods over options, BBQ always provides a correct option, including an explicit uncertainty option, so that the benchmark can distinguish between appropriate abstention and stereotype-driven guessing.

BBQ is hand-built from attested stereotypes and covers nine social dimensions relevant for U.S. English-speaking contexts: age, disability status, gender identity, nationality, physical appearance, race/ethnicity, religion, socio-economic status, and sexual orientation, together with intersectional subsets for race by gender and race by SES (Parrish et al., 2021). The dataset contains 58,492 unique examples. Its central methodological contribution is the pairing of under-informative and adequately informative contexts. This made it possible to ask two different questions at once: whether a model stereotypes when the context is insufficient, and whether the same stereotype can override an evidence-supported answer when the context is sufficient.

This original architecture has become the template for a large family of culturally adapted benchmarks. A recent example is "EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering," which introduces Spanish and Catalan datasets adapted to the social context of Spain and designed to assess social bias across 10 categories in a multiple-choice QA setting (Ruiz-Fernández et al., 15 Jul 2025).

2. Core dataset mechanics and scoring logic

In the standard BBQ-style design, each instance contains a context, a question, and three answer options: two identity-bearing candidates and an explicit “unknown/cannot be determined” choice. The benchmark is deliberately balanced through ambiguous versus disambiguated contexts, negative versus non-negative questions, target versus non-target groups, and permutations of answer order (Parrish et al., 2021). In the original BBQ, each template yields a cluster of four items formed by crossing negative versus non-negative questions with ambiguous versus disambiguated contexts, and the uncertain answer is randomly sampled from 10 synonymous strings such as “Unknown” and “Cannot be determined” (Parrish et al., 2021).

The ambiguous condition is the benchmark’s central control. Here, the context does not support either identity-bearing answer, so the only correct output is the uncertainty option. A non-unknown choice is therefore interpreted as evidence that the model is filling the informational gap with a stereotype. In the disambiguated condition, the context explicitly specifies which identity-bearing answer is correct, so the evaluation shifts from abstention to evidence adherence (Parrish et al., 2021).

Later adaptations preserved this structure but made the scoring more explicit. KoBBQ reports separate accuracy for ambiguous and disambiguated settings and introduces directional diff-bias scores for each context type (Jin et al., 2023). BharatBBQ goes further by distinguishing generic preference from stereotype-structured preference through Accuracy, Bias Score, and Stereotypical Bias Score, computed separately for ambiguous and disambiguated contexts, with disambiguated items additionally partitioned into Negative Pairing and Non-Negative Pairing cases (Tomar et al., 9 Aug 2025). This suggests an increasingly fine-grained view of QA bias: not only whether a model answers incorrectly, but also whether its errors preserve the internal logic of a stereotype.

Quality control has also been part of the benchmark tradition from the outset. In BBQ, templates were edited until 4 out of 5 crowd annotators agreed, human majority-vote accuracy reached 99.7%, and Krippendorff’s $\alpha$ was reported as $0.883$ (Parrish et al., 2021). These values were used to argue that the task items are intelligible to humans even when they probe socially sensitive or pragmatically delicate stereotypes.

3. Empirical regularities in benchmarked models

The first large empirical result is that modern QA systems often fail precisely where uncertainty should dominate. In BBQ, when models answered ambiguous items incorrectly, the proportion of errors aligning with the targeted social bias was 56% for RoBERTa-Base, 59% for RoBERTa-Large, 62% for DeBERTaV3-Base, 68% for DeBERTaV3-Large, 76% for UnifiedQA in RACE format, and 77% for UnifiedQA in ARC format (Parrish et al., 2021). The same study found that models were more accurate in disambiguated settings, but still showed up to 3.4 percentage points higher accuracy when the correct answer aligned with a social bias than when it conflicted with it, with the difference widening to over 5 points on gender items for most models (Parrish et al., 2021).

This general pattern recurs in later multilingual benchmarks. The EsBBQ/CaBBQ abstract reports that models tend to fail to choose the correct answer in ambiguous scenarios and that high QA accuracy often correlates with greater reliance on social biases (Ruiz-Fernández et al., 15 Jul 2025). BharatBBQ reports persistent biases across languages and social categories and often amplified biases in Indian languages compared to English (Tomar et al., 9 Aug 2025). PakBBQ reports an average accuracy gain of 12% with disambiguation, stronger counter-bias behaviors in Urdu than in English, and framing effects that reduce stereotypical responses when questions are posed negatively (Hashmat et al., 13 Aug 2025).

Recent work also shows that the exact direction of bias can depend on linguistic realization. GG-BBQ, which adapts the gender identity subset of BBQ to German, reports that all evaluated models exhibit bias, both along and against existing social stereotypes (Satheesh et al., 22 Jul 2025). FilBBQ, a Filipino adaptation focused on gender and sexual orientation, reports sexist and homophobic biases relating to emotion, domesticity, stereotyped queer interests, and polygamy, and further shows that measured bias scores vary across random seeds (Gamboa et al., 16 Feb 2026).

A plausible implication is that BBQ-style evaluation has exposed two intertwined properties of current QA systems: weak abstention under ambiguity and residual stereotype preference even when the context is informative. Different language ecologies, identity signals, and prompt framings change the surface form of this behavior, but not its general evaluative relevance.

4. Cultural adaptation and multilingual expansion

The benchmark family has expanded by replacing literal translation with culturally grounded reconstruction.

Benchmark	Scope	Salient property
KoBBQ (Jin et al., 2023)	Korean; 268 templates; 76,048 samples; 12 categories	Uses Simply-Transferred, Target-Modified, and Sample-Removed classes plus four Korean-specific categories
EsBBQ / CaBBQ (Ruiz-Fernández et al., 15 Jul 2025)	Spanish and Catalan; 10 categories	Adapts BBQ to Spanish and Catalan and to the social context of Spain
BharatBBQ (Tomar et al., 9 Aug 2025)	8 languages; 49,108 examples per language; 392,864 total; 13 categories including 3 intersectional groups	Targets India-specific categories such as caste and region
PakBBQ (Hashmat et al., 13 Aug 2025)	English and Urdu; 217 templates; 17,180 QA pairs; 8 categories	Adds regional affiliation and language formality for Pakistan
GG-BBQ (Satheesh et al., 22 Jul 2025)	German gender identity subset	Separates group-term and proper-name subsets and emphasizes manual post-editing
FilBBQ (Gamboa et al., 16 Feb 2026)	Filipino; 123 templates; 10,576 prompts	Focuses on gender and sexual orientation with 52 new Filipino-specific templates

KoBBQ is especially important methodologically because it formalizes a reusable adaptation pipeline: templates are partitioned into Simply-Transferred, Target-Modified, and Sample-Removed groups, then supplemented by newly created categories specific to Korean culture, including domestic area of origin, family structure, political orientation, and educational background (Jin et al., 2023). A large-scale survey was used to validate stereotypes and target groups before final inclusion. This is a strong statement that cultural adaptation is not equivalent to translation.

German and Urdu adaptations further show why localization requires grammatical and sociolinguistic intervention. GG-BBQ argues that manual revision is crucial when translating from English into a language with grammatical gender, because articles, pronouns, adjectival agreement, and gendered occupational nouns can leak answer cues or distort the intended stereotype probe (Satheesh et al., 22 Jul 2025). PakBBQ, by contrast, emphasizes regionally salient targets such as religious sects, provincial identities, and language formality, and reports that translating fully instantiated items rather than fragile templates better preserves placeholders and semantics (Hashmat et al., 13 Aug 2025).

BharatBBQ and FilBBQ illustrate two additional directions. BharatBBQ scales the family to eight languages and introduces India-specific axes such as caste and region, including three intersectional groups (Tomar et al., 9 Aug 2025). FilBBQ narrows the category set to gender and sexual orientation but adds Filipino-specific stereotypes and a robust multi-seed evaluation protocol (Gamboa et al., 16 Feb 2026). Taken together, these variants show that the BBQ framework is stable at the level of task design while remaining highly plastic at the level of cultural content.

5. Reliability, prompt sensitivity, and non-social QA artifacts

A major methodological development is the recognition that benchmark scores themselves can be unstable. "On Measuring Social Biases in Prompt-Based Multi-Task Learning" showed that T0 acts more biased in question-answer form than in premise-hypothesis form on semantically equivalent material derived from BBQ and BBNLI, indicating that measured bias depends not only on content but also on input encoding (Akyürek et al., 2022). FilBBQ extends this critique by running 50 distinct random seeds per model and averaging the resulting template-level bias scores, explicitly documenting that a single template can exhibit seed-level score variation from $-1$ to $1$ (Gamboa et al., 16 Feb 2026).

Parallel work has expanded “bias benchmark” to cover evaluation artifacts beyond social stereotyping. In video QA, pretrained LLMs were shown to answer 37.33% of MovieQA questions and 48.91% of TVQA questions correctly without any multimodal context, far above the 20% random baseline for 5-choice questions, revealing strong QA-only and answer-only artifacts (Yang et al., 2020). In multiple-choice visual QA, Easy-Options Bias allows models to choose the correct option from vision and answer options alone; across six benchmarks and four VLM series, mean $\mathrm{Acc}(V{+}O)$ was 51.57%, only 9.54% below mean $\mathrm{Acc}(V{+}Q{+}O)=61.11\%$, and GroundAttack was proposed to generate harder negatives (Zhang et al., 19 Aug 2025).

Other studies isolate position and label artifacts. In binary QA, positional bias was found to be nearly absent under low uncertainty but to grow exponentially as uncertainty increases (Labruna et al., 30 Jun 2025). In MCQ evaluation, ABCD identifies label-position-few-shot-prompt bias and proposes a Matched-and-Dashed protocol that reduces mean accuracy variance by approximately $3\times$ with only a minimal drop in mean performance (Nowak et al., 19 Feb 2026). These results do not replace social-bias benchmarks, but they clarify that a QA system can appear biased either because it stereotypes social groups or because the benchmark protocol itself contains exploitable shortcuts.

6. Limits of the paradigm and broader significance

Bias benchmarking for QA is constrained by the representativeness of its own data. "Social Bias in Popular Question-Answering Benchmarks" analyzed 30 benchmark papers and 20 datasets and found that most papers provided insufficient information about the stakeholders involved in benchmark creation, that just one benchmark paper explicitly reported measures taken to address social representation issues, and that the datasets exhibited gender, religion, and geographic biases (Kraft et al., 21 May 2025). This means that a benchmark can audit model bias while still embedding representational skew in its own source material.

The same caution appears in domain-specific QA benchmarks. BioPulse-QA, a biomedical benchmark that operationalizes bias through age and gender counterfactual swaps, reports negligible differences under those swaps but also notes that the release is limited to age and gender and does not report subgroup disparity aggregates or statistical significance tests (Bhattarai et al., 19 Jan 2026). PediatricsMQA uses age-stratified accuracy to expose performance variability across seven developmental stages in pediatric text and vision QA, but the paper does not define a bespoke bias index or report statistical significance for age-cohort disparities (Bahaj et al., 22 Aug 2025). These examples show that “bias benchmark” can refer either to explicit social-bias measurement or to structured subgroup performance analysis, and the distinction matters analytically.

The benchmark family has also become a substrate for mitigation. BMBI uses BBQ as its evaluation base, tracks bias influence across instances, introduces a new bias evaluation metric, and reports significant bias reduction across all 9 BBQ categories while maintaining comparable QA accuracy (Ma et al., 2023). This suggests that BBQ-style benchmarks are no longer only diagnostic instruments; they are also becoming optimization targets for debiasing methods.

In aggregate, bias benchmarks for question answering now form a layered research area. At the core lies the original BBQ insight that the right control condition for QA bias is an ambiguous context with an explicit uncertainty option. Around that core, later work has added cultural localization, finer-grained scoring, seed-robust evaluation, artifact diagnostics, and mitigation-oriented training. The result is not a single dataset but a benchmark paradigm for testing whether QA behavior is grounded in evidence rather than in stereotype, prompt form, answer order, or other spurious cues (Parrish et al., 2021, Ruiz-Fernández et al., 15 Jul 2025).