---
title: 'WinoIdentity: Intersectional Bias Benchmark'
url: https://www.emergentmind.com/topics/winoidentity
type: topic
---

# WinoIdentity: Intersectional Bias Benchmark

Searching arXiv for the exact term “WinoIdentity” and related benchmark paper to ground the article in current arXiv records.
WinoIdentity is a benchmark for studying **intersectional social bias in large language models (LLMs)** through **coreference resolution**, with a particular emphasis on **model confidence/uncertainty** rather than accuracy alone. It was introduced as an extension of **WinoBias**, augmenting occupation-based Winograd-style sentences with demographic identity markers so that the task probes **intersectional identity reasoning** rather than only binary gender stereotypes [2508.07111]. The benchmark is motivated by two claims: that **single-axis bias evaluation is insufficient**, and that bias should not only be measured by errors, but also by uncertainty, especially when the relevant harm is **omission due to underrepresentation** [2508.07111].

## 1. Conceptual framing

WinoIdentity is situated within fairness evaluation for LLMs used in decision-support settings such as **hiring and admissions**. Its core premise is that existing bias benchmarks often evaluate one social axis at a time, whereas real-world harms arise when “multiple axes of discrimination intersect” and create distinct patterns of disadvantage [2508.07111]. In that sense, WinoIdentity operationalizes **intersectional bias** as disparities across groups formed by combining binary gender with other demographic attributes such as race, disability, nationality, or socio-economic status.

The benchmark also reorients evaluation away from accuracy alone. The authors explicitly center **representational harms**, especially **omission due to underrepresentation**, and argue that uncertainty is a finer-grained signal than accuracy for detecting such harms [2508.07111]. In this framing, a model may not overtly misclassify a marginalized identity, yet still behave unfairly if it is systematically less confident, more brittle, or more inconsistent for that group.

A further conceptual distinction in the work is between **value alignment** and **validity**. The former concerns whether a model treats groups fairly; the latter concerns whether the model is genuinely performing the intended reasoning task rather than relying on memorization or shallow pattern-matching [2508.07111]. The paper argues that these are independent dimensions and that a system may fail on both simultaneously.

## 2. Relation to WinoBias and benchmark construction

WinoIdentity extends **WinoBias**, a benchmark originally built to test **gender bias in coreference resolution** using occupation-based Winograd-style sentences. In these prompts, a model must decide which of two occupations a pronoun refers to. WinoIdentity preserves this occupational and stereotype-sensitive structure while adding demographic markers to occupations, thereby transforming a gender-bias probe into an intersectional fairness benchmark [2508.07111].

WinoBias contains **3,150 sentences** total, corresponding to **1,575 unique base prompts**, each appearing with **male and female pronouns**. It includes **783 unique Type-1 sentences**, which are **syntactically ambiguous** and require semantic or world knowledge, and **792 unique Type-2 sentences**, which are **syntactically unambiguous** and solvable by syntax alone [2508.07111]. WinoIdentity inherits both the Type-1/Type-2 distinction and the stereotypical versus anti-stereotypical occupational settings from WinoBias.

The benchmark extends WinoBias with **25 demographic markers** across **10 demographic attributes**, each intersected with **binary gendered pronouns**, using **3 augmentation types**. This yields **245,700 prompts** and evaluates **50 distinct bias patterns** [2508.07111]. The total is given by

$$1575 \times 2 \times (25+1) \times 3 = 245{,}700.$$

The benchmark includes two unaugmented gender-only prompts per base sentence as a baseline, one with a feminine pronoun and one with a masculine pronoun. For an attribute \(G\) with marker set \(Y_G\), the intersectional groups are

$$(masc, fem) \times Y_G.$$

This produces, for example, four groups for age and eight groups for race [2508.07111].

## 3. Demographic schema and augmentation strategies

The benchmark uses demographic markers drawn from the “Wheel of Power and Privilege,” chosen to be “well represented in the US context.” The attributes and markers listed in the paper are summarized below [2508.07111].

| Attribute | Markers |
|---|---|
| age | young / old |
| body type | thin / fat |
| disability | neurotypical (NT), able-bodied / neurodivergent (ND), disabled |
| gender identity | cisgender / transgender |
| language | English-speaking / non-English-speaking |
| nationality | American / immigrant |
| sexual orientation | heterosexual / gay |
| socio-economic status | rich / poor |
| race | White / Black / Asian / Hispanic |
| religion | Christian / Muslim / Jewish |

The table is explicitly framed in terms of **hegemonic/privileged** markers versus **disadvantaged** markers, although the paper also notes one caveat for age: privilege and disadvantage are context-specific, since old may be disadvantaged in hiring while young may be disadvantaged in lending [2508.07111].

WinoIdentity applies three augmentation strategies. In **Referent augmentation (R-Aug)**, the demographic marker is prepended to the referent occupation only. In **Non-referent augmentation (NR-Aug)**, the marker is prepended to the non-referent occupation only. In **Contrastive augmentation (C-Aug)**, both occupations are marked: the referent receives the marker being tested, while the non-referent receives a contrastive marker [2508.07111]. For attributes with multiple contrastive out-groups, such as race or religion, the paper states that “we marginalize over all of them” [2508.07111].

The construction preserves the WinoBias distinction between **stereotypical contexts**, such as masculine pronouns with male-dominated occupations, and **anti-stereotypical contexts**, such as feminine subgroups in historically male-dominated occupations. The paper highlights cases including **transgender women** and **gay women** in male-dominated occupations, as well as masculine pronouns in female-dominated occupations [2508.07111]. These settings are central to the empirical analysis.

## 4. Evaluation methodology and confidence-based scoring

WinoIdentity evaluates five causal LLMs: **mistral-7B-instruct-v0.2**, **mixtral-8x7B-instruct**, **llama3-70b-instruct**, **pythia-12B**, and **falcon-40B-instruct** [2508.07111]. The model is prompted with the sentence ending in:

> “The pronoun [pronoun] refers to the”

The two candidate occupations are then scored as candidate next tokens or sequences. Formally, the model is treated as a causal LM \(f_\theta\) with next-token distribution

$$P(w \mid L) = \text{softmax}(f_\theta(L))_w.$$

For each prompt, the probability of the **referent occupation** and the **non-referent occupation** is computed. If an occupation is multi-token, the model evaluation sums the log probabilities of the individual tokens to obtain the overall next-word probability [2508.07111]. The experiments use **greedy decoding** for deterministic reproducibility.

The per-example confidence score is **Coreference Confidence**:

$$CC(L_i) = P(referent \mid L_i) - P(non\text{-}referent \mid L_i).$$

A value close to \(1\) indicates that the model is correct and confident; a value around \(0\) indicates uncertainty; a negative value indicates that the non-referent was scored above the referent [2508.07111]. This confidence-based formulation is central to the benchmark’s emphasis on uncertainty as a fairness signal.

The paper also evaluates a **Chain of Thought (CoT)** variant on Mistral. The procedure is to ask the model to reason step by step, append that reasoning as assistant output, and then re-prompt with “Based on the reasoning above, the pronoun "him" refers to the” [2508.07111]. The reported result is that CoT often reduces disparities, but usually also lowers overall confidence.

## 5. Coreference Confidence Disparity

The principal fairness metric introduced with WinoIdentity is **Coreference Confidence Disparity**. For demographic attribute \(G\), over base prompts \(L = (L_1, \dots, L_n)\), it is defined as

$$\Delta CC_G(L) = max_{g \in \{ (masc, fem)\times Y_G\} } \sum_{i=1}^n CC(L_i^g) - min_{g \in \{(masc, fem)\times Y_G\} } \sum_{i=1}^n CC(L_i^g).$$

This is the gap between the subgroup with the highest aggregate confidence and the subgroup with the lowest aggregate confidence [2508.07111]. The interpretation given in the paper is explicit: **lower \(\Delta CC_G(L)\)** means fairer behavior, and **higher \(\Delta CC_G(L)\)** means larger confidence disparities across groups.

This is a worst-group disparity measure. Rather than averaging across subgroups, it asks how far apart the best-treated and worst-treated intersectional groups are. The paper relates this uncertainty-based evaluation to fairness notions that complement classical measures such as **Equalized Odds** and **Statistical Parity** [2508.07111].

An analogous **accuracy disparity** baseline is defined in the appendix by replacing \(CC(L_i)\) with

$$Acc(L_i)=\mathbb{1}[CC(L_i)>0],$$

and then applying the same max-min subgroup gap [2508.07111]. The authors’ methodological claim is that confidence disparity often reveals bias that accuracy alone obscures.

## 6. Empirical findings and interpretive claims

The benchmark’s central empirical conclusion is that current LLMs are “still far from achieving intersectionally fair coreference resolution” [2508.07111]. The paper reports **confidence disparities as high as 40%** along several demographic axes, including **body type**, **sexual orientation**, and **socio-economic status** [2508.07111].

Under **R-Aug**, Mistral exhibits disparities between **20% and 40% on 7 out of 10** attributes, whereas Pythia has disparities below **20%** across all attributes but also poor overall performance [2508.07111]. Concrete Type-2 examples reported in the paper include the following values:

| Model | Attribute | Value |
|---|---|---|
| Mistral | socio-economic status | 0.400 |
| Mistral | body type | 0.392 |
| Mistral | disability | 0.382 |
| Llama3 | socio-economic status | 0.382 |
| Llama3 | sexual orientation | 0.346 |
| Mixtral | race | 0.329 |
| Falcon | disability | 0.251 |

The paper identifies the most concerning disparity dimensions as **body type**, **disability**, **gender identity**, **sexual orientation**, **socio-economic status**, and **race**, while noting that the smallest disparities appear along **age**, **nationality**, and **religion** [2508.07111].

One of the main substantive findings is that models are **most uncertain about doubly-disadvantaged identities in anti-stereotypical settings**, such as assigning **transgender women** to historically male-dominated occupations [2508.07111]. In Mistral’s occupation-level analysis for male-dominated occupations like **mechanic** and **construction worker**, masculine pronouns are more confident and correct, feminine pronouns are less confident and sometimes wrong, and marginalized feminine subgroups do worse still. The paper gives a concrete example for *mechanic*: average coreference confidence for **fem** is **-0.065**, compared to **-0.11** for **transgender_fem**, and **-0.24** for **gay_fem** [2508.07111].

A notable and somewhat unexpected result is that coreference confidence decreases even when adding privileged or “unmarked” identities such as **White**, **cisgender**, and **heterosexual** [2508.07111]. The authors had expected these markers to behave like near-defaults in Western language use, but instead report that confidence “consistently decrease[s] after referent augmentation, even for hegemonic markers, albeit not as much as for non-hegemonic identities” [2508.07111]. This suggests that LLMs have difficulty handling identity-marked variants more generally, not only marginalized ones.

The paper interprets these patterns as evidence that strong base performance on WinoBias may reflect **memorization** more than robust language reasoning. The argument is based on four observations: performance degrades after semantically irrelevant identity insertions; confidence drops even for hegemonic markers; non-demographic modifiers also reduce performance somewhat; and non-referent augmentations affect confidence differently depending on ambiguity and position [2508.07111]. The authors therefore state that “the recent impressive performance of LLMs is more likely due to memorization than logical reasoning” [2508.07111].

## 7. Scope, limitations, and relation to adjacent work

The paper identifies several limitations. First, the identity schema is **US-centric**, since the chosen identities were selected to be “well represented in the US context” [2508.07111]. Second, the benchmark uses **binary gender pronouns only**, so it does not fully capture non-binary or neopronoun configurations. Third, the study covers **50 identities** over **1,575 WinoBias sentences**, which is large but still only a small subset of the broader intersectional space [2508.07111].

A further limitation is **combinatorial explosion**. As more attributes and markers are added, the cost of exhaustive evaluation grows rapidly. The paper explicitly suggests future work using subsampling techniques while preserving statistical guarantees [2508.07111]. Because WinoIdentity is a templated augmentation benchmark, it also inherits the standard caveats of template-based fairness evaluation, including sensitivity to prompt wording, occupation framing, and assumptions about which groups are hegemonic versus disadvantaged.

The authors additionally emphasize that confidence disparity is not identical to real-world harm. The ethics discussion states that fairness is “a complex, non-technical concept,” and that stakeholders rather than ML researchers alone must decide what disparity thresholds are unacceptable [2508.07111]. This suggests that WinoIdentity should be understood as an auditing instrument rather than a complete social theory of harm.

Within the Winograd-style benchmark family, WinoIdentity is most directly connected to **WinoBias**, from which it inherits its occupation-based pronoun resolution structure [2508.07111]. A plausible implication is that WinoIdentity occupies a more specialized niche than generic bias benchmarks: it is not merely a stereotype dataset, but a controlled probe of whether identity-marked but semantically irrelevant modifications perturb reasoning confidence across intersectional groups. The paper proposes extending this framework to datasets such as **StereoSet** and **WinoQueer** to obtain a broader picture of “socially-salient brittleness” [2508.07111].

Overall, WinoIdentity is best characterized as a **large-scale, templated, intersectional fairness benchmark for LLM coreference resolution** that measures not only whether a model selects the correct antecedent, but how confidently and consistently it does so across demographic groups [2508.07111]. Its principal contribution is the combination of demographic augmentation with a confidence-based disparity metric, and its principal empirical result is that even recent LLMs exhibit substantial subgroup disparities, especially for **doubly-disadvantaged identities in anti-stereotypical occupational settings** [2508.07111].

Source: https://www.emergentmind.com/topics/winoidentity