PakBBQ: Pakistani Bias Benchmark

• PakBBQ is a specialized QA bias benchmark tailored to Pakistani socio-cultural contexts, offering bilingual evaluation across eight distinct bias dimensions.
• It applies a four-category adaptation framework—direct translation, target modification, newly added templates, and removals—to ensure cultural relevance and robust dataset construction.
• Empirical findings indicate a significant 12 percentage point accuracy gain with disambiguation and effective bias reduction via negative question framing across multiple LLMs.

Searching arXiv for the specified paper to ground the article and confirm bibliographic details. PakBBQ is a culturally and regionally adapted extension of the original Bias Benchmark for Question Answering (BBQ) dataset, designed to evaluate fairness in LLMs under Pakistani socio-cultural and linguistic conditions. It targets a gap left by predominantly Western-centric training and evaluation regimes by introducing a benchmark in both English and Urdu that reflects bias dimensions salient in Pakistan, including religion, regional affiliation, and language formality. The benchmark comprises 214 templates and 17,180 QA pairs across eight categories, and it evaluates multilingual LLMs under ambiguous versus disambiguated contexts as well as negative versus non-negative question framings. Its central empirical findings are an average accuracy gain of approximately 12 percentage points under disambiguation, stronger counter-bias behavior in Urdu than in English, and framing effects that reduce stereotypical responses when questions are posed negatively (Hashmat et al., 13 Aug 2025).

1. Provenance and adaptation framework

PakBBQ begins from the original U.S.-centric BBQ dataset and adapts it to the Pakistani setting through four template transformation categories inspired by KoBBQ (Hashmat et al., 13 Aug 2025). The first category, Directly Translated (DT), covers templates whose social context applies globally, such as age-related assumptions or generic gender stereotypes; these templates were translated into Roman-Urdu or Urdu, or retained in English. The second, Target Modified (TM), replaces U.S.-specific institutions or groups with Pakistani analogs, such as substituting high-school cliques or fraternities with university societies or local biradaris. The third, Newly Added (NA), consists of templates crafted specifically for Pakistan-relevant biases, including sectarian identities such as Barelvi versus Deobandi, ethnic and regional groups such as Punjabi, Sindhi, and Baloch, religious minorities such as Ahmadis and Hindus, and Urdu formality registers involving honorifics and register choice. The fourth, Simply Removed (SR), excludes templates deemed irrelevant to Pakistan, including examples such as Native American tribes (Hashmat et al., 13 Aug 2025).

The final template pool contains 214 templates after filtering by a cultural-relevance score of at least 1.5 and Fleiss’ $\kappa \ge 0.2$ (Hashmat et al., 13 Aug 2025). The adaptation scheme is important because it formalizes cultural transfer as a selective rather than literal process. A plausible implication is that PakBBQ treats benchmark localization as a structured redesign problem: some constructs are portable, some require analog substitution, some require new category invention, and some must be eliminated entirely.

2. Dataset composition and bias dimensions

PakBBQ contains 17,180 QA pairs split exactly evenly across English and Urdu, with 8,590 examples in each language (Hashmat et al., 13 Aug 2025). Each template was instantiated with multiple name and scenario permutations, two context conditions, two framings, and three cyclic answer-choice permutations. The dataset covers eight bias dimensions, each represented by roughly 25–30 templates:

1. Age
2. Disability Status
3. Physical Appearance
4. Gender Identity
5. Socio-Economic Status
6. Religion
7. Regional Affiliation
8. Language Formality

These dimensions are explicitly grounded in Pakistan-relevant social distinctions. Age includes assumptions about the elderly or children; Disability Status covers physical and mental disability stereotypes; Physical Appearance includes examples such as skin tone biases; Gender Identity includes non-binary identities; Socio-Economic Status captures rich-versus-poor assumptions; Religion includes majority-versus-minority sects; Regional Affiliation captures provincial and ethnic differences; and Language Formality concerns honorifics and register in Urdu (Hashmat et al., 13 Aug 2025).

The overall dataset breakdown is concise:

Component	Quantity
Templates	214
English QA pairs	8,590
Urdu QA pairs	8,590
Grand total QA pairs	17,180

The adaptation-type distribution is approximately 32% DT, approximately 36% TM, and approximately 32% NA, with SR removed prior to instantiation (Hashmat et al., 13 Aug 2025). QA pairs per bias dimension are approximately 2,147 when summed across both contexts and framings. This distribution indicates that Pakistan-specific additions are not marginal supplements; they occupy approximately the same proportion of the template pool as directly translated material. This suggests that substantial benchmark validity depends on newly authored local content rather than on translation alone.

3. Experimental protocol and prompting design

PakBBQ evaluates a representative slice of multilingual LLMs in a zero-shot QA setting (Hashmat et al., 13 Aug 2025). The models listed are DeepSeek-V3 (671 B parameters), GPT-4.1 Nano, GPT-4.1 Mini, GPT-4.1, Gemini 2.0 Flash Lite, and Gemini 2.0 Flash. Prompts were written in English and Urdu, and the benchmark used cyclic permutation of the three answer choices to mitigate position bias. Majority voting across the three permutations produced the final label (Hashmat et al., 13 Aug 2025).

The protocol distinguishes two context conditions. In the Ambiguous Context setting, the scenario is under-specified, so models must rely on prior associations. In the Disambiguated Context setting, the correct answer is explicitly given in the context. The benchmark also distinguishes Negative Framing, in which questions are phrased to highlight “avoid this stereotype,” from Non-Negative Framing (Hashmat et al., 13 Aug 2025).

This design isolates several distinct mechanisms that can affect measured bias. Ambiguity tests default stereotype reliance; disambiguation tests whether the model can override prior associations when evidence is explicit; framing tests whether surface question form modulates response behavior. Because the evaluation uses the same underlying templates across these manipulations, PakBBQ functions not only as a static benchmark but also as a controlled perturbation framework for bias-sensitive QA behavior.

4. Evaluation metrics and interpretive semantics

PakBBQ reports overall accuracy, context-conditioned accuracy, and BBQ-style bias scores (Hashmat et al., 13 Aug 2025). Overall accuracy is defined as

$$\mathrm{Acc} = \frac{\#\;\text{correctly answered examples}}{\#\;\text{total examples}}.$$

Context-conditioned accuracy is tracked separately as $\mathrm{Acc}_{\mathrm{AMB}}$ under ambiguous context and $\mathrm{Acc}_{\mathrm{DIS}}$ under disambiguated context. The benchmark adopts the BBQ bias score for disambiguated settings as

$$s_{\mathrm{DIS}} = 2\left(\frac{n_{\mathrm{biased\_ans}}}{n_{\mathrm{non\_UNKNOWN}}}\right) - 1,$$

and for ambiguous settings as

$$s_{\mathrm{AMB}} = (1 - \mathrm{Acc}) \times s_{\mathrm{DIS}}.$$

The score ranges over $[-1,1]$, where positive values indicate bias toward stereotype and negative values indicate counter-bias (Hashmat et al., 13 Aug 2025). For newly added templates, inter-annotator agreement was controlled by requiring Fleiss’ $\kappa \ge 0.2$ for template inclusion.

These metrics encode a specific interpretation of model behavior. Accuracy measures whether the model selects the benchmark-labeled answer, whereas bias scores characterize the directionality of incorrect or stereotype-sensitive choices. The distinction matters because a model can improve in accuracy under disambiguation while also exhibiting different bias-score behavior across languages and categories. PakBBQ therefore separates epistemic performance from stereotype alignment rather than collapsing them into a single scalar criterion.

5. Empirical results

The headline quantitative result is that disambiguation improves performance substantially: the average gain is approximately 12 percentage points, defined as $$\Delta \mathrm{Acc} = \mathrm{Acc}_{\mathrm{DIS}} - \mathrm{Acc}_{\mathrm{AMB}}$$ (Hashmat et al., 13 Aug 2025). One example reported is GPT-4.1 Mini in Urdu, which rises from 64% to 87%, a gain of 23 percentage points. This indicates that explicit context can materially reduce reliance on prior associations.

Across languages, English accuracy exceeds Urdu accuracy on average. The reported overall accuracies are as follows (Hashmat et al., 13 Aug 2025):

Model	ENG Overall	UR Overall
GPT-4.1 Nano	0.80	0.72
GPT-4.1 Mini	0.82	0.75
GPT-4.1	0.82	0.75
DeepSeek-V3	0.85	0.67
Gemini 2.0 Flash-Lite	0.88	0.69
Gemini 2.0 Flash	0.84	0.81
Average	0.83	0.73

English models cluster around 82–88% accuracy, whereas Urdu models lag by 7–17 percentage points, with an average gap of approximately 10 percentage points. Urdu models also exhibit higher variance, with $\sigma \approx 0.11$ versus 0.07 in English (Hashmat et al., 13 Aug 2025). At the same time, Urdu shows consistently stronger counter-bias behaviors than English. The coexistence of lower accuracy and stronger counter-bias suggests that bias behavior and task performance do not move monotonically together.

Question framing also affects performance. All models perform better when questions are negatively framed, with average gains of 3–5 percentage points (Hashmat et al., 13 Aug 2025). The reported results are:

Model	ENG Neg	ENG Non-Neg	UR Neg	UR Non-Neg
GPT-4.1 Nano	0.83	0.78	0.73	0.71
GPT-4.1 Mini	0.83	0.82	0.77	0.74
DeepSeek-V3	0.85	0.84	0.69	0.64
Gemini 2.0 Flash-Lite	0.89	0.87	0.71	0.66
Gemini 2.0 Flash	0.85	0.82	0.81	0.81

The paper characterizes this as a “framing effect,” suggesting that negative prompts can curb stereotypical default answers (Hashmat et al., 13 Aug 2025). In bias-score terms, the latest Gemini models achieve $$\mathrm{Acc} = \frac{\#\;\text{correctly answered examples}}{\#\;\text{total examples}}.$$0 across all categories in disambiguated contexts, indicating strong counter-bias. In ambiguous contexts, bias scores are closer to 0 or slightly negative, with Urdu showing more counter-bias overall. Language Formality and Religion often exhibit the strongest counter-bias in both languages when disambiguated (Hashmat et al., 13 Aug 2025).

A further result concerns the newly added templates. PakBBQ reports substantial performance drops on NA templates, with accuracies ranging from 0.49 to 0.68 (Hashmat et al., 13 Aug 2025). This supports the claim that Western benchmarks miss Pakistan-specific axes such as sectarian identity, biradari, and formality register.

6. Significance, limitations, and projected extensions

PakBBQ is presented as the first QA bias benchmark adapted to the Pakistani socio-cultural and linguistic landscape, with 17,180 QA pairs, 214 templates, eight bias axes, and bilingual coverage in English and Urdu (Hashmat et al., 13 Aug 2025). Its significance lies in demonstrating that benchmark validity is culturally contingent. Western benchmarks such as BBQ do not capture sectarian, biradari, or Urdu formality phenomena, and PakBBQ shows that performance can degrade substantially on newly added local templates. This suggests that “one-size-fits-all” evaluations risk overlooking harmful stereotypes in under-represented regions.

The benchmark also identifies two lightweight bias-mitigation techniques: explicit disambiguation and negative question framing (Hashmat et al., 13 Aug 2025). Providing clear context is reported to cure automatic stereotype reliance, while rephrasing a question as “Which of these is not stereotypically X?” reduces biased responses by 3–5 percentage points on average. These strategies are described as model-agnostic, low-cost, and especially beneficial for low-resource languages such as Urdu.

The reported limitations are specific. PakBBQ focuses only on Urdu and English, leaving major regional languages such as Punjabi, Sindhi, Pashto, and Balochi unaddressed. It uses zero-shot prompting with a fixed “You are a Pakistani person” system prompt, which may not perfectly capture all registers. It also notes potential translation noise in the Urdu version, with “dark-skinned” given as an example (Hashmat et al., 13 Aug 2025). These limitations indicate that the benchmark is a substantial but incomplete account of linguistic and cultural bias in Pakistan.

The future directions stated are to extend the benchmark to additional Pakistani languages and intersectional identities, investigate fine-tuning under PakBBQ supervision for further bias reduction, and explore cultural adaptation of additional genres such as dialogue and summarization, together with deeper anthropological prompting (Hashmat et al., 13 Aug 2025). By open-sourcing the resource, the authors position PakBBQ as a blueprint for culturally sensitive bias-benchmark construction in other under-represented regions.