Bias Benchmark for QA (BBQ)

• Bias Benchmark for QA (BBQ) is a curated evaluation suite that uses controlled multiple-choice questions to measure stereotype-driven responses in language models.
• It employs manipulations of context informativeness and question polarity to clearly separate bias-driven errors from fact-based answers.
• International adaptations extend BBQ’s methodology to various languages and cultural contexts, revealing discrepancies in model bias and performance.

The Bias Benchmark for Question Answering (BBQ) is an extensively curated and adopted evaluation suite for probing social biases in LLMs via multiple-choice question answering. BBQ and its international adaptations provide a structured, empirically anchored methodology for measuring, comparing, and ultimately mitigating stereotype-driven behaviors across diverse linguistic and cultural contexts. The benchmark’s design is grounded in controlled manipulations of evidence within the test items, allowing precise separation between bias-driven and information-driven responses.

1. Origin and Conceptual Structure

BBQ was first introduced by Parrish et al. as a large-scale multiple-choice QA dataset targeting social biases against members of protected classes in applied LLM outputs (Parrish et al., 2021). It consists of hand-constructed templates, each mapped to attested social stereotypes and spanning nine primary social categories relevant to U.S. English contexts: Age, Disability Status, Gender Identity, Nationality, Physical Appearance, Race/Ethnicity, Religion, Sexual Orientation, and Socio-Economic Status, plus intersectional subsets. Each template is instantiated into four QA items via two orthogonal manipulations:

• Context Informativeness:
  • Ambiguous: No evidence to favor either group; the only factually correct answer is “unknown.”
  • Disambiguated: Additional evidence identifies the correct group.
• Question Polarity:
  • Negative: Asks about stereotypical or negatively colored attributes.
  • Non-Negative: Probes the complement or positive attribute.

Answer choices for each item are: Target entity (stereotyped group), Non-Target entity, and an “unknown” or “cannot be determined” response.

This factorial template expansion allows direct measurement of models’ reliance on stereotypes when information is missing versus when it is explicitly present. BBQ’s annotation pipeline ensures high inter-annotator agreement and documents each stereotype with literature sources.

2. Evaluation Metrics and Formulas

BBQ adopts a rigorous metric suite designed to quantify both accuracy and the alignment of model errors with stereotypes. The canonical metrics, as implemented in the original benchmark and subsequent work (Parrish et al., 2021, Rani et al., 28 Sep 2025, Pelosio et al., 22 Jul 2025), include:

• Accuracy:

$$\mathrm{Accuracy} = \frac{\# \text{correct outputs}}{\# \text{examples}}$$

• Bias Alignment Drop (ΔAcc) (for disambiguated contexts):

$$\Delta \mathrm{Acc} = \mathrm{Acc}_\text{aligned} - \mathrm{Acc}_\text{conflicting}$$

where “aligned” cases have correct answers matching the stereotype; “conflicting” are the converse.

• Bias Score (s$_\text{Dis}$):

$$s_\text{Dis} = 2 \cdot \left( \frac{n_\text{biased ans}}{n_\text{non-unknown outputs}} \right) - 1$$

This metric ranges from –1 (always anti-stereotype) to +1 (always pro-stereotype).

• Bias Score in Ambiguous Contexts (s$_\text{Amb}$):

$$s_\text{Amb} = (1 - \mathrm{Accuracy}) \cdot s_\text{Dis}$$

Here, errors are weighted by their alignment to stereotypes only when a model fails to abstain.

Adaptations and extensions for multilingual or region-specific BBQ variants employ analogous metrics, sometimes adjusting for category-comparability or changing how "biased" vs. "counter-biased" choices are counted (Neplenbroek et al., 11 Jun 2024, Tomar et al., 9 Aug 2025, Pelosio et al., 22 Jul 2025, Jin et al., 2023). Additional measures such as error retention ratios (how much error persists after replacing explicit demographic cues with proxies like names (Pelosio et al., 22 Jul 2025)) and difference-bias scores (for nuanced context breakdowns (Satheesh et al., 22 Jul 2025)) further enrich the analytic landscape.

3. International Adaptations and Cultural Extensions

BBQ’s methodology has seen significant global adaptation to enable fair cross-lingual and cross-cultural model evaluation:

• Multilingual Extensions: Datasets such as MBBQ for Dutch, Spanish, and Turkish (Neplenbroek et al., 11 Jun 2024), BharatBBQ for eight Indian languages (Tomar et al., 9 Aug 2025), EsBBQ/CaBBQ for Spanish and Catalan (Ruiz-Fernández et al., 15 Jul 2025), GG-BBQ for German (Satheesh et al., 22 Jul 2025), JBBQ for Japanese (Yanaka et al., 4 Jun 2024), KoBBQ for Korean (Jin et al., 2023), PBBQ for Persian (Farsi et al., 22 Oct 2025), and PakBBQ for English/Urdu in Pakistan (Hashmat et al., 13 Aug 2025) retain the two-axis (ambiguity × polarity) design but tailor templates, target/non-target demography, and stereotypes to local social realities via expert review, large-scale surveys, and template partitioning (e.g., ST/TM/SR in KoBBQ). These resources introduce new bias axes (e.g., caste, region, political and domestic origin) and validate stereotype relevance among local populations.
• Proper-name and Accentual Proxies: Name-based BBQ (e.g., substituting explicit group labels with culturally salient names (Pelosio et al., 22 Jul 2025)) and spoken-language variants such as VoiceBBQ (evaluating both content and acoustic-induced bias (Choi et al., 25 Sep 2025)) probe models’ robustness to less explicit demographic cues.
• Template Adaptation Protocols: Frameworks for localizing BBQ (simply-transferred, target-modified, newly added, or removed templates) ensure that datasets are both linguistically and sociologically faithful to the target culture (Jin et al., 2023, Tomar et al., 9 Aug 2025, Hashmat et al., 13 Aug 2025).

These international resources enable robust comparison of bias behaviors across model families and deployment settings, controlling for both linguistic competence and cultural stereotype transfer.

4. Empirical Insights and Comparative Analyses

BBQ reveals several systematic patterns in LLM behaviors:

• Bias Prevalence in Ambiguous Contexts: Across models and deployment contexts, the default tendency when evidence is missing is to rely on stereotypes. In ambiguous contexts, errors are overwhelmingly stereotype-aligned (up to 77% in the original paper (Parrish et al., 2021)). Disambiguated contexts improve accuracy, yet the gap in performance when correct answers conflict with stereotypes (ΔAcc up to 5% for gender) remains substantial.
• Cross-Lingual and Cultural Gaps: Multilingual evaluations show that models often exhibit stronger or different bias profiles in non-English languages or when exposed to regionally specific stereotypes. For instance, Spanish prompts elicited the most persistent bias in MBBQ (Neplenbroek et al., 11 Jun 2024), and Indian languages in BharatBBQ showed amplified stereotype reliance compared to English (Tomar et al., 9 Aug 2025). Name-based proxies reduce, but do not erase, stereotype-driven errors (Pelosio et al., 22 Jul 2025).
• Model Size and Bias: Larger models typically achieve higher accuracy and lower bias scores, while smaller models both underperform in general QA and exhibit more error-aligned bias. However, “unbiased” performance is often fragile—minimal contextual perturbations or template rephrasings can “jailbreak” supposed alignment (Miandoab et al., 27 Oct 2025).
• Modality and Acoustic Effects: In spoken settings, both content and acoustic cues (e.g., speaker gender, accent) can induce or amplify bias (Choi et al., 25 Sep 2025). Some architectures are more robust to acoustic levers than others.
• Open-Ended Evaluations and Format Effects: BBQ-style multiple-choice metrics do not always reliably transfer to open-ended (generative) QA or fill-in-the-blank settings; models often produce more biased outputs in these flexible scenarios, emphasizing the need to evaluate both MCQ and generative formats (Liu et al., 9 Dec 2024, Jin et al., 10 Mar 2025).

The empirical finding that improved QA accuracy often correlates with greater bias reliance, especially in ambiguous contexts, persists across LLM architectures, training regimes, and international benchmarks (Ruiz-Fernández et al., 15 Jul 2025). This suggests that unchecked accuracy gains via standard fine-tuning may inadvertently reinforce social stereotypes rather than mitigate them.

5. Bias Mitigation Techniques and Robustness

BBQ underpins several state-of-the-art bias mitigation approaches targeting QA models:

• Debiasing via Influence and Multi-Task Learning (BMBI): The BMBI algorithm tracks the influence of training examples on bias-relevant predictions and incorporates this as an explicit loss term. It uses small reference sets per bias axis and a refined, probabilistic bias metric to optimize toward both accuracy and bias reduction (Ma et al., 2023).
• Parameter- and Data-Efficient Adapter Fusion (Open-DeBias): Open-DeBias introduces lightweight adapters per bias category, combined with fusion layers, to achieve near-zero bias and high ambiguous-context accuracy using only a small fraction of the original training data. Notably, adapters trained in English transfer zero-shot to Korean BBQ with high efficacy (Rani et al., 28 Sep 2025).
• Prompt Engineering and Format Manipulation: Prompt design, including bias warnings, chain-of-thought reasoning, and negative question framings, can measurably reduce bias but do not suffice for full mitigation (Yanaka et al., 4 Jun 2024, Hashmat et al., 13 Aug 2025). Some models show marked sensitivity to prompt framing, particularly in low-resource or non-English settings.
• Robustness Under Perturbation: Studies show that high abstention or neutrality in BBQ can be brittle. When scenarios are minimally perturbed while preserving semantic content, decisiveness and thus bias increase significantly, exposing a lack of deep uncertainty calibration (Miandoab et al., 27 Oct 2025).

Effective mitigation is not universal; models that perform well on BBQ may fail under less-constrained, generative, or open-ended settings—suggesting that current bias alignment strategies may only be superficial.

6. Open Research Problems and Future Directions

BBQ and its international adaptations have defined a rigorous paradigm for social bias assessment in LLMs, but several challenges remain:

• Coverage and Generalization: The domain of BBQ—while broad—cannot capture the entirety of culturally specific biases, emergent stereotypes, or all intersectional axes. Expanding to include domain-specific, low-resource languages, category intersections, and real-life deployment scenarios is ongoing (Jin et al., 10 Mar 2025, Farsi et al., 22 Oct 2025).
• Metric Harmonization and Calibration: Some BBQ metrics may not scale directly across open-ended or generative QA; harmonizing bias scores across different answer formats and model types is required for meaningful cross-benchmark comparison.
• Debiasing Trade-offs: Modifications that reduce bias (e.g., through adapter fusion, prompt framing) must preserve underlying QA competence, uncertainty calibration, and general-domain NLU performance (Rani et al., 28 Sep 2025).
• Acoustic and Multimodal Bias: As LLMs become integrated into speech and multimodal platforms, benchmarks such as VoiceBBQ identify a need for joint mitigation of content and acoustics-induced social bias (Choi et al., 25 Sep 2025).
• Cultural Sensitivity and Local Validation: Successful adaptation to new cultural settings requires survey-driven target selection, stereotype validation, and domain-expert involvement; simple translation of stereotypes or scenarios risks misalignment or erasure of meaningful bias dimensions (Jin et al., 2023, Tomar et al., 9 Aug 2025, Farsi et al., 22 Oct 2025).
• Robustness to Rephrasing and Input Perturbation: Ensuring that bias-mitigation approaches are not brittle to small changes in context or question phrasing remains an open challenge. Current models frequently overfit to the specific format of BBQ rather than internalizing causal or counterfactual reasoning about bias (Miandoab et al., 27 Oct 2025).

BBQ and its descendants serve as critical infrastructure for both measuring and guiding mitigation work in the service of fair, trustworthy QA systems. Their evolution reflects the ongoing need for scale, nuance, cultural sensitivity, and adversarial robustness in social bias benchmarking for language technologies.