---
title: 'PakBBQ: Pakistani Bias Benchmark'
url: https://www.emergentmind.com/topics/pakbbq
type: topic
---

# PakBBQ: Pakistani Bias Benchmark

Searching arXiv for the specified paper to ground the article and confirm bibliographic details.
PakBBQ is a culturally and regionally adapted extension of the original Bias Benchmark for Question Answering (BBQ) dataset, designed to evaluate fairness in Large Language Models (LLMs) under Pakistani socio-cultural and linguistic conditions. It targets a gap left by predominantly Western-centric training and evaluation regimes by introducing a benchmark in both English and Urdu that reflects bias dimensions salient in Pakistan, including religion, regional affiliation, and language formality. The benchmark comprises 214 templates and 17,180 QA pairs across eight categories, and it evaluates multilingual LLMs under ambiguous versus disambiguated contexts as well as negative versus non-negative question framings. Its central empirical findings are an average accuracy gain of approximately 12 percentage points under disambiguation, stronger counter-bias behavior in Urdu than in English, and framing effects that reduce stereotypical responses when questions are posed negatively [2508.10186].

## 1. Provenance and adaptation framework

PakBBQ begins from the original U.S.-centric BBQ dataset and adapts it to the Pakistani setting through four template transformation categories inspired by KoBBQ [2508.10186]. The first category, **Directly Translated (DT)**, covers templates whose social context applies globally, such as age-related assumptions or generic gender stereotypes; these templates were translated into Roman-Urdu or Urdu, or retained in English. The second, **Target Modified (TM)**, replaces U.S.-specific institutions or groups with Pakistani analogs, such as substituting high-school cliques or fraternities with university societies or local biradaris. The third, **Newly Added (NA)**, consists of templates crafted specifically for Pakistan-relevant biases, including sectarian identities such as Barelvi versus Deobandi, ethnic and regional groups such as Punjabi, Sindhi, and Baloch, religious minorities such as Ahmadis and Hindus, and Urdu formality registers involving honorifics and register choice. The fourth, **Simply Removed (SR)**, excludes templates deemed irrelevant to Pakistan, including examples such as Native American tribes [2508.10186].

The final template pool contains 214 templates after filtering by a cultural-relevance score of at least 1.5 and Fleiss’ $\kappa \ge 0.2$ [2508.10186]. The adaptation scheme is important because it formalizes cultural transfer as a selective rather than literal process. A plausible implication is that PakBBQ treats benchmark localization as a structured redesign problem: some constructs are portable, some require analog substitution, some require new category invention, and some must be eliminated entirely.

## 2. Dataset composition and bias dimensions

PakBBQ contains 17,180 QA pairs split exactly evenly across English and Urdu, with 8,590 examples in each language [2508.10186]. Each template was instantiated with multiple name and scenario permutations, two context conditions, two framings, and three cyclic answer-choice permutations. The dataset covers eight bias dimensions, each represented by roughly 25–30 templates:

1. Age  
2. Disability Status  
3. Physical Appearance  
4. Gender Identity  
5. Socio-Economic Status  
6. Religion  
7. Regional Affiliation  
8. Language Formality

These dimensions are explicitly grounded in Pakistan-relevant social distinctions. Age includes assumptions about the elderly or children; Disability Status covers physical and mental disability stereotypes; Physical Appearance includes examples such as skin tone biases; Gender Identity includes non-binary identities; Socio-Economic Status captures rich-versus-poor assumptions; Religion includes majority-versus-minority sects; Regional Affiliation captures provincial and ethnic differences; and Language Formality concerns honorifics and register in Urdu [2508.10186].

The overall dataset breakdown is concise:

| Component | Quantity |
|---|---:|
| Templates | 214 |
| English QA pairs | 8,590 |
| Urdu QA pairs | 8,590 |
| Grand total QA pairs | 17,180 |

The adaptation-type distribution is approximately 32% DT, approximately 36% TM, and approximately 32% NA, with SR removed prior to instantiation [2508.10186]. QA pairs per bias dimension are approximately 2,147 when summed across both contexts and framings. This distribution indicates that Pakistan-specific additions are not marginal supplements; they occupy approximately the same proportion of the template pool as directly translated material. This suggests that substantial benchmark validity depends on newly authored local content rather than on translation alone.

## 3. Experimental protocol and prompting design

PakBBQ evaluates a representative slice of multilingual LLMs in a zero-shot QA setting [2508.10186]. The models listed are DeepSeek-V3 (671 B parameters), GPT-4.1 Nano, GPT-4.1 Mini, GPT-4.1, Gemini 2.0 Flash Lite, and Gemini 2.0 Flash. Prompts were written in English and Urdu, and the benchmark used cyclic permutation of the three answer choices to mitigate position bias. Majority voting across the three permutations produced the final label [2508.10186].

The protocol distinguishes two context conditions. In the **Ambiguous Context** setting, the scenario is under-specified, so models must rely on prior associations. In the **Disambiguated Context** setting, the correct answer is explicitly given in the context. The benchmark also distinguishes **Negative Framing**, in which questions are phrased to highlight “avoid this stereotype,” from **Non-Negative Framing** [2508.10186].

This design isolates several distinct mechanisms that can affect measured bias. Ambiguity tests default stereotype reliance; disambiguation tests whether the model can override prior associations when evidence is explicit; framing tests whether surface question form modulates response behavior. Because the evaluation uses the same underlying templates across these manipulations, PakBBQ functions not only as a static benchmark but also as a controlled perturbation framework for bias-sensitive QA behavior.

## 4. Evaluation metrics and interpretive semantics

PakBBQ reports overall accuracy, context-conditioned accuracy, and BBQ-style bias scores [2508.10186]. Overall accuracy is defined as

$$
\mathrm{Acc} = \frac{\#\;\text{correctly answered examples}}{\#\;\text{total examples}}.
$$

Context-conditioned accuracy is tracked separately as $\mathrm{Acc}_{\mathrm{AMB}}$ under ambiguous context and $\mathrm{Acc}_{\mathrm{DIS}}$ under disambiguated context. The benchmark adopts the BBQ bias score for disambiguated settings as

$$
s_{\mathrm{DIS}} = 2\left(\frac{n_{\mathrm{biased\_ans}}}{n_{\mathrm{non\_UNKNOWN}}}\right) - 1,
$$

and for ambiguous settings as

$$
s_{\mathrm{AMB}} = (1 - \mathrm{Acc}) \times s_{\mathrm{DIS}}.
$$

The score ranges over $[-1,1]$, where positive values indicate bias toward stereotype and negative values indicate counter-bias [2508.10186]. For newly added templates, inter-annotator agreement was controlled by requiring Fleiss’ $\kappa \ge 0.2$ for template inclusion.

These metrics encode a specific interpretation of model behavior. Accuracy measures whether the model selects the benchmark-labeled answer, whereas bias scores characterize the directionality of incorrect or stereotype-sensitive choices. The distinction matters because a model can improve in accuracy under disambiguation while also exhibiting different bias-score behavior across languages and categories. PakBBQ therefore separates epistemic performance from stereotype alignment rather than collapsing them into a single scalar criterion.

## 5. Empirical results

The headline quantitative result is that disambiguation improves performance substantially: the average gain is approximately 12 percentage points, defined as $\Delta \mathrm{Acc} = \mathrm{Acc}_{\mathrm{DIS}} - \mathrm{Acc}_{\mathrm{AMB}}$ [2508.10186]. One example reported is GPT-4.1 Mini in Urdu, which rises from 64% to 87%, a gain of 23 percentage points. This indicates that explicit context can materially reduce reliance on prior associations.

Across languages, English accuracy exceeds Urdu accuracy on average. The reported overall accuracies are as follows [2508.10186]:

| Model | ENG Overall | UR Overall |
|---|---:|---:|
| GPT-4.1 Nano | 0.80 | 0.72 |
| GPT-4.1 Mini | 0.82 | 0.75 |
| GPT-4.1 | 0.82 | 0.75 |
| DeepSeek-V3 | 0.85 | 0.67 |
| Gemini 2.0 Flash-Lite | 0.88 | 0.69 |
| Gemini 2.0 Flash | 0.84 | 0.81 |
| Average | 0.83 | 0.73 |

English models cluster around 82–88% accuracy, whereas Urdu models lag by 7–17 percentage points, with an average gap of approximately 10 percentage points. Urdu models also exhibit higher variance, with $\sigma \approx 0.11$ versus 0.07 in English [2508.10186]. At the same time, Urdu shows consistently stronger counter-bias behaviors than English. The coexistence of lower accuracy and stronger counter-bias suggests that bias behavior and task performance do not move monotonically together.

Question framing also affects performance. All models perform better when questions are negatively framed, with average gains of 3–5 percentage points [2508.10186]. The reported results are:

| Model | ENG Neg | ENG Non-Neg | UR Neg | UR Non-Neg |
|---|---:|---:|---:|---:|
| GPT-4.1 Nano | 0.83 | 0.78 | 0.73 | 0.71 |
| GPT-4.1 Mini | 0.83 | 0.82 | 0.77 | 0.74 |
| DeepSeek-V3 | 0.85 | 0.84 | 0.69 | 0.64 |
| Gemini 2.0 Flash-Lite | 0.89 | 0.87 | 0.71 | 0.66 |
| Gemini 2.0 Flash | 0.85 | 0.82 | 0.81 | 0.81 |

The paper characterizes this as a “framing effect,” suggesting that negative prompts can curb stereotypical default answers [2508.10186]. In bias-score terms, the latest Gemini models achieve $s_{\mathrm{DIS}} \approx -1$ across all categories in disambiguated contexts, indicating strong counter-bias. In ambiguous contexts, bias scores are closer to 0 or slightly negative, with Urdu showing more counter-bias overall. Language Formality and Religion often exhibit the strongest counter-bias in both languages when disambiguated [2508.10186].

A further result concerns the newly added templates. PakBBQ reports substantial performance drops on NA templates, with accuracies ranging from 0.49 to 0.68 [2508.10186]. This supports the claim that Western benchmarks miss Pakistan-specific axes such as sectarian identity, biradari, and formality register.

## 6. Significance, limitations, and projected extensions

PakBBQ is presented as the first QA bias benchmark adapted to the Pakistani socio-cultural and linguistic landscape, with 17,180 QA pairs, 214 templates, eight bias axes, and bilingual coverage in English and Urdu [2508.10186]. Its significance lies in demonstrating that benchmark validity is culturally contingent. Western benchmarks such as BBQ do not capture sectarian, biradari, or Urdu formality phenomena, and PakBBQ shows that performance can degrade substantially on newly added local templates. This suggests that “one-size-fits-all” evaluations risk overlooking harmful stereotypes in under-represented regions.

The benchmark also identifies two lightweight bias-mitigation techniques: **explicit disambiguation** and **negative question framing** [2508.10186]. Providing clear context is reported to cure automatic stereotype reliance, while rephrasing a question as “Which of these is *not* stereotypically X?” reduces biased responses by 3–5 percentage points on average. These strategies are described as model-agnostic, low-cost, and especially beneficial for low-resource languages such as Urdu.

The reported limitations are specific. PakBBQ focuses only on Urdu and English, leaving major regional languages such as Punjabi, Sindhi, Pashto, and Balochi unaddressed. It uses zero-shot prompting with a fixed “You are a Pakistani person” system prompt, which may not perfectly capture all registers. It also notes potential translation noise in the Urdu version, with “dark-skinned” given as an example [2508.10186]. These limitations indicate that the benchmark is a substantial but incomplete account of linguistic and cultural bias in Pakistan.

The future directions stated are to extend the benchmark to additional Pakistani languages and intersectional identities, investigate fine-tuning under PakBBQ supervision for further bias reduction, and explore cultural adaptation of additional genres such as dialogue and summarization, together with deeper anthropological prompting [2508.10186]. By open-sourcing the resource, the authors position PakBBQ as a blueprint for culturally sensitive bias-benchmark construction in other under-represented regions.

Source: https://www.emergentmind.com/topics/pakbbq