WinoIdentity: Intersectional Bias Benchmark

• WinoIdentity is a large-scale, templated benchmark that assesses intersectional fairness in LLM coreference resolution via demographic identity markers and confidence-based scoring.
• It extends WinoBias by incorporating 25 demographic markers across 10 attributes, enabling analysis of multi-axial biases beyond traditional binary gender evaluations.
• Empirical findings reveal confidence disparities up to 40% across intersectional groups, highlighting significant challenges in LLM reasoning under identity-marked prompts.

Searching arXiv for the exact term “WinoIdentity” and related benchmark paper to ground the article in current arXiv records. WinoIdentity is a benchmark for studying intersectional social bias in LLMs through coreference resolution, with a particular emphasis on model confidence/uncertainty rather than accuracy alone. It was introduced as an extension of WinoBias, augmenting occupation-based Winograd-style sentences with demographic identity markers so that the task probes intersectional identity reasoning rather than only binary gender stereotypes (Khan et al., 9 Aug 2025). The benchmark is motivated by two claims: that single-axis bias evaluation is insufficient, and that bias should not only be measured by errors, but also by uncertainty, especially when the relevant harm is omission due to underrepresentation (Khan et al., 9 Aug 2025).

1. Conceptual framing

WinoIdentity is situated within fairness evaluation for LLMs used in decision-support settings such as hiring and admissions. Its core premise is that existing bias benchmarks often evaluate one social axis at a time, whereas real-world harms arise when “multiple axes of discrimination intersect” and create distinct patterns of disadvantage (Khan et al., 9 Aug 2025). In that sense, WinoIdentity operationalizes intersectional bias as disparities across groups formed by combining binary gender with other demographic attributes such as race, disability, nationality, or socio-economic status.

The benchmark also reorients evaluation away from accuracy alone. The authors explicitly center representational harms, especially omission due to underrepresentation, and argue that uncertainty is a finer-grained signal than accuracy for detecting such harms (Khan et al., 9 Aug 2025). In this framing, a model may not overtly misclassify a marginalized identity, yet still behave unfairly if it is systematically less confident, more brittle, or more inconsistent for that group.

A further conceptual distinction in the work is between value alignment and validity. The former concerns whether a model treats groups fairly; the latter concerns whether the model is genuinely performing the intended reasoning task rather than relying on memorization or shallow pattern-matching (Khan et al., 9 Aug 2025). The paper argues that these are independent dimensions and that a system may fail on both simultaneously.

2. Relation to WinoBias and benchmark construction

WinoIdentity extends WinoBias, a benchmark originally built to test gender bias in coreference resolution using occupation-based Winograd-style sentences. In these prompts, a model must decide which of two occupations a pronoun refers to. WinoIdentity preserves this occupational and stereotype-sensitive structure while adding demographic markers to occupations, thereby transforming a gender-bias probe into an intersectional fairness benchmark (Khan et al., 9 Aug 2025).

WinoBias contains 3,150 sentences total, corresponding to 1,575 unique base prompts, each appearing with male and female pronouns. It includes 783 unique Type-1 sentences, which are syntactically ambiguous and require semantic or world knowledge, and 792 unique Type-2 sentences, which are syntactically unambiguous and solvable by syntax alone (Khan et al., 9 Aug 2025). WinoIdentity inherits both the Type-1/Type-2 distinction and the stereotypical versus anti-stereotypical occupational settings from WinoBias.

The benchmark extends WinoBias with 25 demographic markers across 10 demographic attributes, each intersected with binary gendered pronouns, using 3 augmentation types. This yields 245,700 prompts and evaluates 50 distinct bias patterns (Khan et al., 9 Aug 2025). The total is given by

$1575 \times 2 \times (25+1) \times 3 = 245{,}700.$

The benchmark includes two unaugmented gender-only prompts per base sentence as a baseline, one with a feminine pronoun and one with a masculine pronoun. For an attribute $G$ with marker set $Y_G$, the intersectional groups are

$(masc, fem) \times Y_G.$

This produces, for example, four groups for age and eight groups for race (Khan et al., 9 Aug 2025).

3. Demographic schema and augmentation strategies

The benchmark uses demographic markers drawn from the “Wheel of Power and Privilege,” chosen to be “well represented in the US context.” The attributes and markers listed in the paper are summarized below (Khan et al., 9 Aug 2025).

Attribute	Markers
age	young / old
body type	thin / fat
disability	neurotypical (NT), able-bodied / neurodivergent (ND), disabled
gender identity	cisgender / transgender
language	English-speaking / non-English-speaking
nationality	American / immigrant
sexual orientation	heterosexual / gay
socio-economic status	rich / poor
race	White / Black / Asian / Hispanic
religion	Christian / Muslim / Jewish

The table is explicitly framed in terms of hegemonic/privileged markers versus disadvantaged markers, although the paper also notes one caveat for age: privilege and disadvantage are context-specific, since old may be disadvantaged in hiring while young may be disadvantaged in lending (Khan et al., 9 Aug 2025).

WinoIdentity applies three augmentation strategies. In Referent augmentation (R-Aug), the demographic marker is prepended to the referent occupation only. In Non-referent augmentation (NR-Aug), the marker is prepended to the non-referent occupation only. In Contrastive augmentation (C-Aug), both occupations are marked: the referent receives the marker being tested, while the non-referent receives a contrastive marker (Khan et al., 9 Aug 2025). For attributes with multiple contrastive out-groups, such as race or religion, the paper states that “we marginalize over all of them” (Khan et al., 9 Aug 2025).

The construction preserves the WinoBias distinction between stereotypical contexts, such as masculine pronouns with male-dominated occupations, and anti-stereotypical contexts, such as feminine subgroups in historically male-dominated occupations. The paper highlights cases including transgender women and gay women in male-dominated occupations, as well as masculine pronouns in female-dominated occupations (Khan et al., 9 Aug 2025). These settings are central to the empirical analysis.

4. Evaluation methodology and confidence-based scoring

WinoIdentity evaluates five causal LLMs: mistral-7B-instruct-v0.2, mixtral-8x7B-instruct, llama3-70b-instruct, pythia-12B, and falcon-40B-instruct (Khan et al., 9 Aug 2025). The model is prompted with the sentence ending in:

“The pronoun [pronoun] refers to the”

The two candidate occupations are then scored as candidate next tokens or sequences. Formally, the model is treated as a causal LM $f_\theta$ with next-token distribution

$P(w \mid L) = \text{softmax}(f_\theta(L))_w.$

For each prompt, the probability of the referent occupation and the non-referent occupation is computed. If an occupation is multi-token, the model evaluation sums the log probabilities of the individual tokens to obtain the overall next-word probability (Khan et al., 9 Aug 2025). The experiments use greedy decoding for deterministic reproducibility.

The per-example confidence score is Coreference Confidence:

$$CC(L_i) = P(referent \mid L_i) - P(non\text{-}referent \mid L_i).$$

A value close to $1$ indicates that the model is correct and confident; a value around $0$ indicates uncertainty; a negative value indicates that the non-referent was scored above the referent (Khan et al., 9 Aug 2025). This confidence-based formulation is central to the benchmark’s emphasis on uncertainty as a fairness signal.

The paper also evaluates a Chain of Thought (CoT) variant on Mistral. The procedure is to ask the model to reason step by step, append that reasoning as assistant output, and then re-prompt with “Based on the reasoning above, the pronoun "him" refers to the” (Khan et al., 9 Aug 2025). The reported result is that CoT often reduces disparities, but usually also lowers overall confidence.

5. Coreference Confidence Disparity

The principal fairness metric introduced with WinoIdentity is Coreference Confidence Disparity. For demographic attribute $G$, over base prompts $G$0, it is defined as

$G$1

This is the gap between the subgroup with the highest aggregate confidence and the subgroup with the lowest aggregate confidence (Khan et al., 9 Aug 2025). The interpretation given in the paper is explicit: lower $G$2 means fairer behavior, and higher $G$3 means larger confidence disparities across groups.

This is a worst-group disparity measure. Rather than averaging across subgroups, it asks how far apart the best-treated and worst-treated intersectional groups are. The paper relates this uncertainty-based evaluation to fairness notions that complement classical measures such as Equalized Odds and Statistical Parity (Khan et al., 9 Aug 2025).

An analogous accuracy disparity baseline is defined in the appendix by replacing $G$4 with

$G$5

and then applying the same max-min subgroup gap (Khan et al., 9 Aug 2025). The authors’ methodological claim is that confidence disparity often reveals bias that accuracy alone obscures.

6. Empirical findings and interpretive claims

The benchmark’s central empirical conclusion is that current LLMs are “still far from achieving intersectionally fair coreference resolution” (Khan et al., 9 Aug 2025). The paper reports confidence disparities as high as 40% along several demographic axes, including body type, sexual orientation, and socio-economic status (Khan et al., 9 Aug 2025).

Under R-Aug, Mistral exhibits disparities between 20% and 40% on 7 out of 10 attributes, whereas Pythia has disparities below 20% across all attributes but also poor overall performance (Khan et al., 9 Aug 2025). Concrete Type-2 examples reported in the paper include the following values:

Model	Attribute	Value
Mistral	socio-economic status	0.400
Mistral	body type	0.392
Mistral	disability	0.382
Llama3	socio-economic status	0.382
Llama3	sexual orientation	0.346
Mixtral	race	0.329
Falcon	disability	0.251

The paper identifies the most concerning disparity dimensions as body type, disability, gender identity, sexual orientation, socio-economic status, and race, while noting that the smallest disparities appear along age, nationality, and religion (Khan et al., 9 Aug 2025).

One of the main substantive findings is that models are most uncertain about doubly-disadvantaged identities in anti-stereotypical settings, such as assigning transgender women to historically male-dominated occupations (Khan et al., 9 Aug 2025). In Mistral’s occupation-level analysis for male-dominated occupations like mechanic and construction worker, masculine pronouns are more confident and correct, feminine pronouns are less confident and sometimes wrong, and marginalized feminine subgroups do worse still. The paper gives a concrete example for mechanic: average coreference confidence for fem is -0.065, compared to -0.11 for transgender_fem, and -0.24 for gay_fem (Khan et al., 9 Aug 2025).

A notable and somewhat unexpected result is that coreference confidence decreases even when adding privileged or “unmarked” identities such as White, cisgender, and heterosexual (Khan et al., 9 Aug 2025). The authors had expected these markers to behave like near-defaults in Western language use, but instead report that confidence “consistently decrease[s] after referent augmentation, even for hegemonic markers, albeit not as much as for non-hegemonic identities” (Khan et al., 9 Aug 2025). This suggests that LLMs have difficulty handling identity-marked variants more generally, not only marginalized ones.

The paper interprets these patterns as evidence that strong base performance on WinoBias may reflect memorization more than robust language reasoning. The argument is based on four observations: performance degrades after semantically irrelevant identity insertions; confidence drops even for hegemonic markers; non-demographic modifiers also reduce performance somewhat; and non-referent augmentations affect confidence differently depending on ambiguity and position (Khan et al., 9 Aug 2025). The authors therefore state that “the recent impressive performance of LLMs is more likely due to memorization than logical reasoning” (Khan et al., 9 Aug 2025).

7. Scope, limitations, and relation to adjacent work

The paper identifies several limitations. First, the identity schema is US-centric, since the chosen identities were selected to be “well represented in the US context” (Khan et al., 9 Aug 2025). Second, the benchmark uses binary gender pronouns only, so it does not fully capture non-binary or neopronoun configurations. Third, the study covers 50 identities over 1,575 WinoBias sentences, which is large but still only a small subset of the broader intersectional space (Khan et al., 9 Aug 2025).

A further limitation is combinatorial explosion. As more attributes and markers are added, the cost of exhaustive evaluation grows rapidly. The paper explicitly suggests future work using subsampling techniques while preserving statistical guarantees (Khan et al., 9 Aug 2025). Because WinoIdentity is a templated augmentation benchmark, it also inherits the standard caveats of template-based fairness evaluation, including sensitivity to prompt wording, occupation framing, and assumptions about which groups are hegemonic versus disadvantaged.

The authors additionally emphasize that confidence disparity is not identical to real-world harm. The ethics discussion states that fairness is “a complex, non-technical concept,” and that stakeholders rather than ML researchers alone must decide what disparity thresholds are unacceptable (Khan et al., 9 Aug 2025). This suggests that WinoIdentity should be understood as an auditing instrument rather than a complete social theory of harm.

Within the Winograd-style benchmark family, WinoIdentity is most directly connected to WinoBias, from which it inherits its occupation-based pronoun resolution structure (Khan et al., 9 Aug 2025). A plausible implication is that WinoIdentity occupies a more specialized niche than generic bias benchmarks: it is not merely a stereotype dataset, but a controlled probe of whether identity-marked but semantically irrelevant modifications perturb reasoning confidence across intersectional groups. The paper proposes extending this framework to datasets such as StereoSet and WinoQueer to obtain a broader picture of “socially-salient brittleness” (Khan et al., 9 Aug 2025).

Overall, WinoIdentity is best characterized as a large-scale, templated, intersectional fairness benchmark for LLM coreference resolution that measures not only whether a model selects the correct antecedent, but how confidently and consistently it does so across demographic groups (Khan et al., 9 Aug 2025). Its principal contribution is the combination of demographic augmentation with a confidence-based disparity metric, and its principal empirical result is that even recent LLMs exhibit substantial subgroup disparities, especially for doubly-disadvantaged identities in anti-stereotypical occupational settings (Khan et al., 9 Aug 2025).