EsBBQ: Spanish Bias Benchmark for QA
- EsBBQ is a Spanish bias benchmark that adapts the BBQ framework to assess social bias across 10 categories in Spain’s context.
- It employs a multiple-choice QA format to differentiate between ambiguity and evidence-based responses, revealing bias even with high accuracy.
- Empirical findings indicate that higher QA performance can correlate with increased reliance on social biases, emphasizing the need for nuanced bias evaluation.
EsBBQ is the Spanish Bias Benchmark for Question Answering introduced together with CaBBQ, the Catalan counterpart, as a pair of parallel datasets for social-bias evaluation in LLMs. It is based on the original BBQ benchmark, but is adapted to the Spanish language and to the social context of Spain. The benchmark uses a multiple-choice QA setting and assesses social bias across 10 categories. Its stated motivation is the scarcity of resources for social-bias evaluation in languages other than English and for social contexts outside the United States. Reported results indicate that models tend to fail to choose the correct answer in ambiguous scenarios, and that high QA accuracy often correlates with greater reliance on social biases (Ruiz-Fernández et al., 15 Jul 2025).
1. Definition and scope
EsBBQ is defined in the literature as part of a paired release, “EsBBQ and CaBBQ,” where the two resources are presented as the Spanish and Catalan Bias Benchmarks for Question Answering (Ruiz-Fernández et al., 15 Jul 2025). The benchmark is explicitly framed as a resource for evaluating social bias in LLMs rather than as a general-purpose QA dataset. Its target is not factual recall alone, but model behavior when answering socially sensitive questions under controlled conditions.
The benchmark’s scope is delimited by three design decisions stated in the paper. First, it is a multiple-choice QA benchmark. Second, it is parallel with CaBBQ. Third, it is adapted not only to a language, but also to a social context, namely Spain (Ruiz-Fernández et al., 15 Jul 2025). This places EsBBQ within the broader class of BBQ-derived benchmarks that preserve the original task structure while relocating the benchmark linguistically and culturally.
2. Relation to BBQ
EsBBQ is described as being “based on the original BBQ” (Ruiz-Fernández et al., 15 Jul 2025). In the original BBQ formulation, the benchmark is a hand-built multiple-choice QA resource for measuring social bias in model outputs, with examples organized around ambiguous and disambiguated contexts, an explicit uncertainty option, and paired negative and non-negative questions (Parrish et al., 2021). BBQ was designed for U.S. English-speaking contexts and covers attested harmful stereotypes across multiple social dimensions (Parrish et al., 2021).
Because EsBBQ is stated to be based on BBQ, a plausible interpretation is that it inherits the core evaluative logic of BBQ while relocating that logic to Spanish and to Spain-specific social meanings. The original BBQ’s central distinction is between under-informative contexts, where the correct answer is an uncertainty option, and adequately informative contexts, where the model must follow evidence rather than stereotype (Parrish et al., 2021). This suggests that EsBBQ should be understood not as a loose inspiration from BBQ, but as part of the same benchmark family.
A common misunderstanding is to treat BBQ-style adaptations as translations only. That is not how EsBBQ is characterized. The paper states that the benchmark is adapted to “the social context of Spain,” which implies a cultural recontextualization in addition to language transfer (Ruiz-Fernández et al., 15 Jul 2025).
3. Task structure and localization
The paper states that EsBBQ and CaBBQ are “designed to assess social bias across 10 categories using a multiple-choice QA setting” (Ruiz-Fernández et al., 15 Jul 2025). It does not, in the provided material, enumerate those 10 categories individually, but it does specify the task format and the adaptation target. The multiple-choice structure matters because BBQ-style evaluation operationalizes bias through answer selection under controlled ambiguity.
Localization is central to EsBBQ’s identity. The original BBQ was explicitly built for U.S. English-speaking contexts (Parrish et al., 2021), whereas EsBBQ is adapted to Spain (Ruiz-Fernández et al., 15 Jul 2025). In later BBQ-family work on implicit bias, EsBBQ is cited as one of the “numerous benchmark extensions” inspired by BBQ, alongside other culturally or linguistically adapted descendants (Vedula et al., 2 Apr 2026). That positioning is significant: EsBBQ is part of a benchmark lineage in which cross-linguistic adaptation is treated as a first-class research problem rather than a secondary implementation detail.
This cultural localization has methodological consequences. Stereotypes, protected-group salience, pragmatic judgments of ambiguity, and socially legible descriptors are not invariant across societies. The fact that EsBBQ is defined relative to Spain therefore matters as much as the fact that it is written in Spanish.
4. Reported empirical findings
The EsBBQ/CaBBQ paper reports evaluation results “on different LLMs, factoring in model family, size and variant” (Ruiz-Fernández et al., 15 Jul 2025). Within the provided material, the main empirical conclusions are twofold. First, “models tend to fail to choose the correct answer in ambiguous scenarios.” Second, “high QA accuracy often correlates with greater reliance on social biases” (Ruiz-Fernández et al., 15 Jul 2025).
The second point is especially important because it cuts against a frequent assumption in benchmark interpretation. In BBQ-style settings, accuracy is not a sufficient fairness proxy. The original BBQ already distinguished between correctness under ambiguity and correctness under disambiguation, and introduced bias scores precisely because aggregate accuracy can obscure stereotype-consistent behavior (Parrish et al., 2021). EsBBQ sharpens that concern by reporting that stronger QA performance may coincide with stronger bias reliance (Ruiz-Fernández et al., 15 Jul 2025).
This result aligns with a broader pattern in the BBQ literature. An open-ended extension of BBQ reports that a model can answer the multiple-choice version correctly while producing a biased answer in fill-in-the-blank or short-answer form (Liu et al., 2024). A generation-based adaptation likewise finds that QA-based BBQ results do not positively correlate with generation-based neutrality or bias scores (Jin et al., 10 Mar 2025). Taken together, these results suggest that EsBBQ’s multiple-choice findings should be interpreted as a controlled diagnostic signal, not as an exhaustive measure of downstream fairness.
5. Position within the broader BBQ ecosystem
The benchmark family surrounding EsBBQ now includes several orthogonal extensions of the original BBQ paradigm.
| Benchmark | Main adaptation axis | Relation to EsBBQ |
|---|---|---|
| BBQ | Original U.S. English QA benchmark | Foundational source design (Parrish et al., 2021) |
| EsBBQ / CaBBQ | Spanish/Catalan adaptation to Spain | Direct subject (Ruiz-Fernández et al., 15 Jul 2025) |
| Open-BBQ | Open-ended fill-in-the-blank and short-answer formats | Extends BBQ beyond multiple choice (Liu et al., 2024) |
| BBG | Story-generation adaptation of BBQ-style evaluation | Shifts from QA to generation (Jin et al., 10 Mar 2025) |
| ImplicitBBQ | Implicit-cue bias evaluation | Tests bias when identity is not explicit (Wagh et al., 7 Dec 2025) |
Within this ecology, EsBBQ represents the language-and-context adaptation axis. Other variants stress different failure modes. Open-BBQ targets open-ended generation rather than closed-form answer selection (Liu et al., 2024). BBG tests long-form continuations and shows that generation-based bias rankings can diverge from QA-based rankings (Jin et al., 10 Mar 2025). ImplicitBBQ replaces explicit identity labels with implicit social cues and shows that models may appear fairer when protected attributes are named explicitly than when they are only indirectly signaled (Wagh et al., 7 Dec 2025). A later characteristic-based ImplicitBBQ explicitly cites EsBBQ as part of the broader BBQ extension lineage (Vedula et al., 2 Apr 2026).
This comparison clarifies what EsBBQ does and does not claim to do. It extends BBQ into Spanish and Spain-specific social settings, but it remains within the multiple-choice QA paradigm. It is therefore complementary to, rather than interchangeable with, open-ended, generation-based, or implicit-cue variants.
6. Interpretive significance
EsBBQ is significant because it addresses a structural gap in bias evaluation: most benchmark development had centered on English and on U.S.-specific social contexts, despite the fact that LLM deployment is multilingual and socially situated (Ruiz-Fernández et al., 15 Jul 2025). By adapting the BBQ framework to Spanish and to Spain, EsBBQ provides a way to test whether benchmarked bias phenomena persist under different linguistic and sociocultural conditions.
A second contribution is methodological rather than geographic. The benchmark supports direct comparison across model families, sizes, and variants within a controlled QA format (Ruiz-Fernández et al., 15 Jul 2025). That matters for bias analysis because it separates the problem of cross-model comparison from the problem of cross-language adaptation.
A third point concerns interpretation. A recurrent misconception in fairness evaluation is that better benchmark accuracy entails less biased behavior. EsBBQ explicitly reports the opposite tendency in at least part of its evaluation: high QA accuracy often correlates with greater reliance on social biases (Ruiz-Fernández et al., 15 Jul 2025). Related BBQ-family work reinforces the same caution from other directions, showing that explicit multiple-choice evaluation can miss bias visible in free-form generation or in implicitly cued settings (Liu et al., 2024, Wagh et al., 7 Dec 2025).
A plausible implication is that EsBBQ should be read as a foundational Spanish-language component of a larger evaluation stack rather than as a complete fairness audit by itself. In the current BBQ research landscape, the most natural extensions would be implicit-cue and generation-oriented Spanish variants, because those are precisely the dimensions along which recent BBQ-family work has revealed additional failure modes (Jin et al., 10 Mar 2025, Vedula et al., 2 Apr 2026).