CaBBQ: Catalan Bias Benchmark for QA

• CaBBQ is a benchmark designed to evaluate social biases in question answering by adapting the original BBQ dataset to Catalan and Spanish contexts.
• The benchmark quantifies biases across 10 social categories using ambiguous scenarios to reveal when models rely on stereotypes over evidence.
• Empirical findings show that higher QA accuracy can coincide with greater bias reliance, underlining the need for fairness beyond mere performance metrics.

Searching arXiv for the specified paper and closely related BBQ-style benchmark papers to ground the article in current literature. CaBBQ is the Catalan Bias Benchmark for Question Answering, introduced together with EsBBQ in the paper "EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering" (Ruiz-Fernández et al., 15 Jul 2025). It is described as a parallel dataset based on the original BBQ and designed to assess social bias across 10 categories in a multiple-choice QA setting, adapted both to the Catalan language and to the social context of Spain. Its motivation is the previously noted lack of resources for social bias evaluation in languages other than English and for social contexts outside of the United States. The reported evaluation spans different LLMs while factoring in model family, size, and variant; the principal findings are that models tend to fail to choose the correct answer in ambiguous scenarios, and that high QA accuracy often correlates with greater reliance on social biases (Ruiz-Fernández et al., 15 Jul 2025).

1. Provenance, naming, and scope

CaBBQ denotes the Catalan member of a paired benchmark release, EsBBQ and CaBBQ, targeting bias evaluation in question answering for Spanish and Catalan respectively (Ruiz-Fernández et al., 15 Jul 2025). In this usage, the suffix "Ca" refers to Catalan, while "BBQ" refers to the benchmark lineage originating in Bias Benchmark for Question Answering. The benchmark is therefore situated within fairness evaluation for LLMs rather than within unrelated literatures that reuse the acronym "BBQ," such as "Beyond Bare Queries" for open-vocabulary 3D object grounding (Linok et al., 2024) or "BBQ-mIS" for a hybrid quantum-classical graph coloring algorithm (Vercellino et al., 5 May 2026).

The benchmark’s stated scope is specifically sociolinguistic and sociocultural adaptation. It is not presented merely as a translation exercise, but as an adaptation to the social context of Spain. This distinction matters because bias benchmarks are sensitive to the demographic categories, stereotypes, and contextual assumptions embedded in their prompts. CaBBQ is therefore best understood as a localized fairness benchmark within a broader QA-based bias-evaluation tradition.

2. Position within the BBQ benchmark lineage

CaBBQ is explicitly described as being based on the original BBQ (Ruiz-Fernández et al., 15 Jul 2025). In related BBQ-derived work, the underlying paradigm is a multiple-choice QA framework for evaluating whether a model relies on stereotypes rather than evidence, particularly when answering ambiguous social questions (Ranjan et al., 23 Mar 2026). Closely related extensions such as ImplicitBBQ preserve BBQ’s multiple-choice structure and its emphasis on ambiguity while modifying how protected attributes are cued (Wagh et al., 7 Dec 2025). This situates CaBBQ within a family of benchmarks that treat bias assessment as a controlled inference problem rather than as unrestricted text generation.

Because CaBBQ is based on original BBQ and retains a multiple-choice QA setting, a plausible implication is that it inherits the central BBQ evaluative logic: bias is revealed when model behavior departs from evidential sufficiency under socially charged conditions. The abstract-level description confirms that ambiguous scenarios are a salient part of the evaluation, since one of the main reported findings is that models tend to fail to choose the correct answer in such cases (Ruiz-Fernández et al., 15 Jul 2025). In that respect, CaBBQ aligns with the broader BBQ research program, where ambiguity is not a nuisance variable but a diagnostic condition.

3. Linguistic and contextual adaptation to Catalan and Spain

The defining contribution of CaBBQ is its adaptation to Catalan and to the social context of Spain (Ruiz-Fernández et al., 15 Jul 2025). The benchmark responds to two gaps identified in prior work: the scarcity of social bias evaluation resources outside English and the scarcity of benchmarks reflecting social contexts outside the United States. CaBBQ addresses both simultaneously. Its paired release with EsBBQ also indicates a parallel design across Spanish and Catalan, which suggests a controlled multilingual setup rather than two independently assembled resources.

This localization has methodological significance. Bias benchmarks are often shaped by nationally specific institutions, demographic taxonomies, linguistic registers, and stereotype repertoires. A benchmark adapted to Spain is not interchangeable with a benchmark grounded in U.S. social categories, even if both derive from the same template family. Relative to ImplicitBBQ, which extends BBQ through implicitly cued protected attributes in an English-only and U.S.-centric setting across 6 categories, CaBBQ targets a different axis of benchmark generalization: language and sociocultural transfer rather than cue explicitness (Wagh et al., 7 Dec 2025). These are orthogonal interventions, and together they illustrate that fairness evaluation can vary along both representational and geopolitical dimensions.

4. Benchmark design as reported

The reported abstract specifies three design properties of CaBBQ. First, it is a dataset for question answering rather than free-form generation. Second, it uses a multiple-choice QA setting. Third, it assesses social bias across 10 categories (Ruiz-Fernández et al., 15 Jul 2025). The paper also states that EsBBQ and CaBBQ are parallel datasets, which indicates that their construction was coordinated across languages rather than merely analogous in theme.

The publicly described evaluation protocol examines different LLMs while factoring in model family, size, and variant (Ruiz-Fernández et al., 15 Jul 2025). That phrasing indicates that the benchmark was used not only for overall model ranking but also for systematic comparison across architectural lineages and scale or release differences within those lineages. A plausible implication is that CaBBQ is intended to support comparative bias profiling across multilingual or instruction-tuned model ecosystems, especially where Catalan performance would otherwise be under-measured.

At the same time, the abstract-level description does not enumerate the 10 category labels, the detailed construction procedure, or the quantitative result tables. For encyclopedia purposes, this limits any more granular reconstruction of annotation workflow, prompt templates, or per-category outcomes. The benchmark’s documented identity is therefore clearest at the level of objective, task format, and high-level findings.

5. Empirical findings and what they imply

Two empirical conclusions are explicitly reported. The first is that models tend to fail to choose the correct answer in ambiguous scenarios. The second is that high QA accuracy often correlates with greater reliance on social biases (Ruiz-Fernández et al., 15 Jul 2025). Taken together, these results place CaBBQ among fairness benchmarks that treat accuracy and bias as partially decoupled dimensions rather than assuming that better task performance automatically indicates less harmful behavior.

The second finding is especially important because it counters a common simplification in benchmark interpretation. If higher QA accuracy can coincide with greater reliance on social biases, then raw correctness is an incomplete proxy for fairness. This is consistent with broader BBQ-oriented evaluation practice, where debiasing studies separately report standard task accuracy and a bias-sensitive metric such as Bias Score, defined as the relative preference for stereotype-congruent over stereotype-incongruent answers (Ranjan et al., 23 Mar 2026). Although CaBBQ’s abstract does not state that it uses Bias Score, the reported correlation between high QA accuracy and greater reliance on social biases points to the same conceptual distinction: a model may answer more items correctly overall while still exhibiting more biased decision rules on socially sensitive cases.

The emphasis on ambiguous scenarios also suggests that CaBBQ is designed to probe calibration under underdetermination. In the BBQ lineage, ambiguity is where stereotype-driven completion is most readily exposed, because the task is structured so that social priors should not supply the missing evidence. CaBBQ’s reported results therefore indicate that this failure mode persists in Catalan- and Spain-adapted settings rather than being confined to English-language benchmarks.

6. Relation to adjacent fairness benchmarks and downstream use

CaBBQ belongs to a growing body of work extending BBQ-style evaluation beyond its original setting. ImplicitBBQ modifies cue presentation by replacing explicit protected-attribute mentions with names, occupations, clothing, relationship references, religious or cultural practices, and other naturalistic descriptors, showing that a model can appear fair on explicit prompts yet still reveal implicit bias when identity is only implied (Wagh et al., 7 Dec 2025). CaBBQ addresses a different generalization problem: whether bias evaluation remains valid when transferred into Catalan and into the social context of Spain. In methodological terms, ImplicitBBQ and CaBBQ can be read as complementary benchmark evolutions, one along the axis of cue explicitness and the other along the axis of language and national context.

The continuing importance of BBQ-style benchmarks is also evident in downstream debiasing research. CatRAG evaluates debiasing on BBQ across gender, nationality, race, and race × gender subsets and uses both Accuracy and Bias Score to quantify fairness–utility trade-offs (Ranjan et al., 23 Mar 2026). This broader ecosystem underscores why localized resources such as CaBBQ matter: debiasing methods, fairness audits, and model comparisons require benchmark coverage that extends beyond English and beyond U.S.-centric social assumptions. CaBBQ contributes that coverage for Catalan, which is otherwise underrepresented in benchmark infrastructures for LLM bias assessment.

A separate misconception concerns the acronym itself. In contemporary arXiv literature, "BBQ" is polysemous and appears in unrelated NLP, robotics, and quantum-computing contexts. Within fairness evaluation, however, CaBBQ specifically refers to the Catalan Bias Benchmark for Question Answering, not to CatRAG, ImplicitBBQ, "Beyond Bare Queries," or "BBQ-mIS."

7. Significance and present evidential limits

CaBBQ’s significance lies in the conjunction of three properties: it is BBQ-derived, Catalan-language, and Spain-adapted (Ruiz-Fernández et al., 15 Jul 2025). That combination directly addresses an identifiable deficiency in LLM fairness evaluation, namely the concentration of benchmark resources in English and in U.S.-specific social settings. For research on multilingual LLMs, Catalan is a particularly informative case because it tests fairness evaluation outside the major-language defaults that dominate benchmark design.

The benchmark also crystallizes a substantive point about model assessment. The reported correlation between high QA accuracy and greater reliance on social biases indicates that performance optimization and fairness mitigation cannot be treated as interchangeable objectives. This suggests that multilingual evaluation suites should not collapse bias analysis into generic QA accuracy, especially in socially ambiguous settings.

The present public description, however, supports a bounded level of specificity. The abstract establishes the benchmark’s purpose, inheritance from BBQ, adaptation to Catalan and Spain, coverage of 10 categories, and its headline empirical findings. It does not, in the available description, enumerate the individual categories, provide construction and validation procedures, or report per-model and per-category numbers. Consequently, CaBBQ is most precisely characterized at this stage as a localized BBQ-style benchmark whose main documented contribution is to extend social-bias QA evaluation into Catalan and Spanish, and whose main documented result is that ambiguity remains difficult for LLMs and that higher QA accuracy can coincide with greater bias reliance (Ruiz-Fernández et al., 15 Jul 2025).