WinoBias: Gender Bias in Coreference Benchmark

• WinoBias is a benchmark for evaluating gender bias in coreference resolution using Winograd-schema style sentences designed to neutralize gender cues.
• It comprises 3,160 balanced instances split into pro- and anti-stereotypical cases, ensuring parity in evaluating male and female pronoun resolution.
• The dataset facilitates practical debiasing strategies such as data augmentation and embedding neutralization to mitigate stereotype-driven errors in NLP models.

Searching arXiv for WinoBias and closely related benchmark papers. WinoBias is a benchmark for evaluating gender bias in coreference resolution. It was introduced as a corpus of Winograd-schema style sentences with entities corresponding to people referred by their occupation, and its design makes the gender of the pronoun irrelevant to making the correct co-reference decision; instead, correct resolution should follow global context rather than gender cues (Zhao et al., 2018). In the literature summarized here, WinoBias functions both as a diagnostic dataset for stereotype-sensitive pronoun resolution and as a reusable fairness probe that has been extended, repurposed, and embedded into broader evaluation pipelines for machine translation, language modeling, representation debiasing, prompting, and intersectional bias analysis (Zhao et al., 2018).

1. Origin, construction, and task design

WinoBias was introduced in "Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods" (Zhao et al., 2018). The corpus contains Winograd-schema style sentences with entities corresponding to people referred by their occupation, such as "the nurse," "the doctor," and "the carpenter" (Zhao et al., 2018). The dataset uses a vocabulary of 40 distinct jobs, encompassing both male- and female-dominated professions according to U.S. Department of Labor statistics, and each occupation is labeled as male- or female-stereotypical based on the percentage of women in the profession (Zhao et al., 2018).

The benchmark is organized around two prototypical templates. Type 1 requires semantic reasoning and is described as a setting where world knowledge is needed and no syntactic cues are available. Type 2 provides syntactic cues and can often be disambiguated syntactically (Zhao et al., 2018). The corpus contains 3,160 sentences, balanced across dev/test, equally distributed for male and female pronouns, and with equal numbers of pro- and anti-stereotypical linkage cases (Zhao et al., 2018).

A pro-stereotypical case is one in which the gendered pronoun links to an occupation matching societal stereotype, while an anti-stereotypical case links opposite to the stereotype (Zhao et al., 2018). The stated purpose is to create sentences in which the pronoun can refer to either gender based on global context, not just gender cues (Zhao et al., 2018). This framing made WinoBias a canonical occupational coreference benchmark for testing whether systems exploit stereotypes rather than resolving reference from the sentence itself.

2. Evaluation protocol and bias criterion

The original evaluation uses F1 score for each setting and defines bias as the absolute difference in F1 between pro- and anti-stereotypical cases (Zhao et al., 2018). A system is considered free of gender bias if its accuracy is the same for pro- and anti-stereotypical cases in both Type 1 and Type 2 sentence templates (Zhao et al., 2018). This yields a simple operational criterion: parity between pro- and anti-stereotypical resolution performance.

Subsequent work reused WinoBias with alternative metrics. "Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned LLMs" (Manela et al., 2021) proposed two WinoBias-specific quantities. Stereotype measures whether the model performs better when the correct pronoun matches the stereotype, and skew captures overall preference for one gender regardless of stereotype. The paper defines

$$\text{Stereotype}^g = \text{F1}^g_\text{pro} - \text{F1}^g_\text{anti}$$

and

$\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$

with mean forms

$\mu_\text{Stereo} \triangleq \frac{1}{2} \left( \left| \text{F1}^{\mars}_\text{pro} - \text{F1}^{\mars}_\text{anti} \right| + \left| \text{F1}^{\female}_\text{pro} - \text{F1}^{\female}_\text{anti} \right| \right)$

and

$\mu_\text{Skew} \triangleq \frac{1}{2}\left( \left| \text{F1}^{\mars}_\text{pro} - \text{F1}^{\female}_\text{pro} \right| + \left| \text{F1}^{\mars}_\text{anti} - \text{F1}^{\female}_\text{anti} \right| \right)$

(Manela et al., 2021).

Other authors emphasized uncertainty rather than point estimates. "Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds" (Ethayarajh, 2020) reframed fairness metrics as differences in expected costs,

$$\delta(f; c) = \mathbb{E}_{A}[c(y, \hat{y})] - \mathbb{E}_{B}[c(y, \hat{y})],$$

and used Bernstein’s inequality

$$\Pr\left[ |\bar{\delta} - \delta| > t \right] \le 2\exp\left( \frac{-nt^2}{2\sigma^2 + \frac{2}{3}tm} \right)$$

to place confidence intervals around bias estimates derived from small annotated samples (Ethayarajh, 2020). In that treatment, WinoBias is not only a benchmark but also an example of a bias-specific dataset whose finite size constrains statistical confidence.

3. Original empirical findings and early debiasing results

The original WinoBias paper evaluated three families of coreference systems: a rule-based system, a feature-rich system, and a neural system (Zhao et al., 2018). Across these systems, the reported average difference between pro- and anti-stereotypical resolutions was 21.1 in F1 score (Zhao et al., 2018). The rule-based system showed up to 39.2 F1 gap, while the feature-rich and neural systems also exhibited substantial disparities (Zhao et al., 2018). These results established that very different coreference paradigms linked gendered pronouns to pro-stereotypical entities with higher accuracy than anti-stereotypical entities.

The same paper introduced a debiasing strategy based on data augmentation ("gender swapping") combined with existing word-embedding debiasing techniques (Zhao et al., 2018). The augmentation constructs a parallel gender-swapped version of the OntoNotes 5.0 training data, after named entity anonymization and collection of gendered span rewriting rules (Zhao et al., 2018). The paper also used debiased GloVe vectors and balanced external gender statistics lists for systems that relied on such resources (Zhao et al., 2018). After these interventions, the bias measured on WinoBias dropped close to zero across systems, while OntoNotes F1 remained nearly unchanged (Zhao et al., 2018).

This early result shaped much of the later literature. WinoBias became a benchmark on which one could test whether a mitigation removed pro/anti disparities without materially degrading general coreference accuracy. A plausible implication is that the dataset’s balanced construction made it especially suitable for isolating stereotype-driven resolution behavior from overall task performance.

4. WinoBias as a lens on representation bias, data imbalance, and statistical power

Several papers used WinoBias to trace how gender bias propagates from corpora and representations into downstream coreference systems. "Gender Bias in Contextualized Word Embeddings" (Zhao et al., 2019) analyzed ELMo and reported that the One Billion Word Benchmark corpus used to train ELMo contains significantly more male than female entities, with male pronouns appearing approximately 3 times as often as female pronouns and co-occurring more frequently with occupation words (Zhao et al., 2019). The paper showed that a coreference system using GloVe+ELMo improved overall performance but increased the measured WinoBias bias in the Semantics Only setting from 26.6 to 29.6 absolute F1 difference (Zhao et al., 2019). Training-time data augmentation reduced this difference to 1.0, whereas test-time embedding neutralization reduced it only partially, to 11.1 (Zhao et al., 2019).

"WikiCREM: A Large Unsupervised Corpus for Coreference Resolution" (Kocijan et al., 2019) used WinoBias as one of several pronoun disambiguation benchmarks. The paper reported split results for WB T1-a, WB T1-p, WB T2-a, and WB T2-p, and stated that models trained with WikiCREM show increased performance and reduced bias on the gender diagnostic datasets WinoGender and WinoBias (Kocijan et al., 2019). Notably, the authors emphasized that a male-skewed WikiCREM corpus did not have negative impact on WinoBias bias in their experiments (Kocijan et al., 2019). This suggests that corpus imbalance alone is not a sufficient explanatory variable for downstream stereotype behavior.

"Is Your Classifier Actually Biased?" (Ethayarajh, 2020) shifted attention from effect size to inferential reliability. It described WinoBias as the largest bias-specific dataset currently available for the relevant task, with 3160 examples, and argued that current datasets are too small for conclusive bias detection except in very large-effect regimes (Ethayarajh, 2020). The paper stated that for a classifier that is 5% more accurate on gender-stereotypical sentences, detecting bias with 95% confidence requires a dataset at least 3.8 times larger than WinoBias, i.e. more than 11,900 annotated examples (Ethayarajh, 2020). It further stated that WinoBias can only be used when $\bar{\delta} \ge 0.0975$, meaning an observed bias estimate of at least 9.75 percentage points, if one seeks high-confidence claims (Ethayarajh, 2020). In the WinoBias literature, this is a central caution against overinterpreting small measured differences.

5. Extensions, variants, and repurposings

WinoBias has generated a family of derivative benchmarks and adaptations. "Second Order WinoBias (SoWinoBias) Test Set for Latent Gender Bias Detection in Coreference Resolution" (Dawkins, 2021) was motivated by the observation that real-world text often lacks explicit gender words even though gender bias can still persist. SoWinoBias removes explicit gender words and uses sentences such as "The doctor liked the nurse because they were beautiful," where "they" always refers to the second occupation and the adjective is female-stereotyped (Dawkins, 2021). The dataset includes 4096 pro-stereotypical and 4096 anti-stereotypical sentences (Dawkins, 2021). On this benchmark, vanilla GloVe showed a 17.4 percentage-point pro/anti gap, and combining data augmentation with GN-GloVe$(w_a)$ reduced the difference to 2.4 (Dawkins, 2021). The paper concluded that methods that reduce explicit first-order bias in WinoBias do not necessarily reduce latent second-order bias (Dawkins, 2021).

"NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives" (Vanmassenhove et al., 2021) introduced WinoBias+, described as an extended, curated, and manually corrected version of the original WinoBias with 3,167 synthetic sentences, each paired with a manually constructed gender-neutral alternative (Vanmassenhove et al., 2021). WinoBias+ served as the primary synthetic benchmark for gender-neutral rewriting. The neural rewriter achieved 0.00% WER and zero errors on WinoBias+ in manual evaluation, while the rule-based system had 0.06% WER with 21 errors on 3167 sentences (Vanmassenhove et al., 2021). In this line of work, WinoBias becomes source material for neutralization rather than merely a fairness probe.

"Evaluating Gender Bias in Machine Translation" (Stanovsky et al., 2019) created WinoMT by concatenating Winogender and WinoBias, producing a challenge set of 3,888 instances balanced across male, female, and neutral entities as well as stereotypical and anti-stereotypical roles (Stanovsky et al., 2019). The evaluation pipeline translated each English sentence, aligned the occupation using fast_align, extracted morphological gender with language-specific analyzers, and scored overall gender accuracy together with

$\Delta_G = F_1^\text{male} - F_1^\text{female}$

and

$$\Delta_S = F_1^\text{stereotypical} - F_1^\text{anti-stereotypical}$$

(Stanovsky et al., 2019). WinoBias thereby became a component in multilingual fairness auditing.

More recent work extended WinoBias along additional social axes. "Investigating Intersectional Bias in LLMs using Confidence Disparities in Coreference Resolution" (Khan et al., 9 Aug 2025) created WinoIdentity by augmenting WinoBias with 25 demographic markers across 10 attributes, intersected with binary gender, yielding 245,700 prompts and 50 identity groups (Khan et al., 9 Aug 2025). Instead of accuracy alone, the paper proposed Coreference Confidence Disparity: $\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$0 and

$\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$1

(Khan et al., 9 Aug 2025). This marks a shift from single-axis gender disparity to intersectional uncertainty analysis.

6. Role in LLM evaluation, prompting, and representation interventions

WinoBias has also become a standard probe for LLMs. "Prompting GPT-3 To Be Reliable" (Si et al., 2022) used WinoBias to assess social bias in pronoun resolution and defined Bias Gap as the absolute difference in accuracy between pro-bias and anti-bias subsets: $\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$2 (Si et al., 2022). The paper reported that balanced 16-shot prompts with equal representation from Type I-Pro, Type I-Anti, Type II-Pro, and Type II-Anti dramatically reduced the bias gap, whereas pro-only or anti-only prompts produced much larger gaps (Si et al., 2022). It also found that random shuffling of demonstrations is better than grouping pro-bias or anti-bias examples at the end (Si et al., 2022).

"Fairness Dynamics During Training" (Patel et al., 2 Jun 2025) used WinoBias Type 2 examples as a central probe for bias during pretraining. For each sample, the authors constructed two prompts asking for the gender of the occupation referred to by the pronoun and the occupation not referred to by the pronoun, with answer choices "male", "female", and "not specified" (Patel et al., 2 Jun 2025). They introduced Average Rank (AR) and Jensen-Shannon Divergence by Parts (JSD-P) as fairness dynamics metrics. On Pythia-6.9b, the paper found that the model becomes more performant and confident predicting "male" than "female" during training, that early stopping after about 80k training steps yields a 92.5% increase in fairness at a cost of only 1.7% LAMBADA accuracy, and that larger models can exhibit more bias than smaller ones (Patel et al., 2 Jun 2025). WinoBias in this setting becomes a temporal diagnostic of training trajectories rather than a static final benchmark.

"Preserving Task-Relevant Information Under Linear Concept Removal" (Holstege et al., 12 Jun 2025) evaluated SPLICE on WinoBias by removing linear information about the sensitive concept $\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$3 while preserving covariance with the profession label $\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$4. The optimization problem is

$\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$5

with closed-form solution

$\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$6

(Holstege et al., 12 Jun 2025). The paper reported that SPLICE improves overall and anti-stereotypical accuracy compared to LEACE and SAL in two out of three evaluated models, while all three methods remove the gender concept equally well (Holstege et al., 12 Jun 2025). This use of WinoBias situates the benchmark within representation surgery and linear guardedness.

The benchmark has also been repurposed for stereotype-free evaluation. "Are Models Biased on Text without Gender-related Language?" (Belém et al., 2024) filtered WinoBias using the UnStereoEval (USE) framework, which measures word-level gender association by

$\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$7

and defines sentence stereotype strength as

$\text{Skew}_\mathcal{D} = \text{F1}^{\mars}_\mathcal{D} - \text{F1}^{\female}_\mathcal{D}$8

(Belém et al., 2024). On filtered WinoBias and Winogender subsets, the authors reported low fairness across 28 tested models, with only 9\%-41\% of stereotype-free sentence pairs judged fair, and systematic preference for male pronoun completions persisting even after strong gender co-occurring words were removed (Belém et al., 2024). This suggests that WinoBias can still reveal bias after explicit stereotype cues are stripped away.

7. Limitations, critiques, and continuing significance

A recurring critique is that WinoBias is simultaneously influential and incomplete. One limitation concerns statistical power: current bias-specific datasets may be too small to support high-confidence claims except in the most egregious cases (Ethayarajh, 2020). Another concerns scope: WinoBias focuses mainly on gender and occupation bias in pronoun resolution, with limited treatment of intersectionality or culture-specific stereotypes (Bahl et al., 4 Feb 2025). ASCenD-BDS explicitly identifies static datasets such as WinoBias as providing finite scenario sets and contrasts them with dynamic, context-aware scenario generation (Bahl et al., 4 Feb 2025).

There are also concerns about benchmark contamination and task narrowness for modern LLMs. "Gender bias and stereotypes in LLMs" (Kotek et al., 2023) states that WinoBias is likely to be included in the training data of current LLMs and therefore introduces a new ambiguity-based paradigm rather than reusing WinoBias directly (Kotek et al., 2023). That paper reports that LLMs are 3–6 times more likely to choose an occupation that stereotypically aligns with a person's gender, ignore crucial ambiguities 95% of the time, and provide factually inaccurate rationalizations for their choices (Kotek et al., 2023). The critique is not that WinoBias is invalid, but that public availability and fixed templates may make it insufficient as a sole contemporary test.

Other work argues that WinoBias may conflate different manifestations of gender prejudice. "Stereotype and Skew" (Manela et al., 2021) states that existing gender bias benchmarks do not fully probe professional bias because pronoun resolution may be obfuscated by cross-correlations from other manifestations of gender prejudice. "Second Order WinoBias" (Dawkins, 2021) reaches a related conclusion from the opposite direction: methods that reduce explicit bias measured on WinoBias can fail on latent bias when explicit gender words are absent. Together these results indicate that WinoBias is best understood as a foundational first-order benchmark rather than an exhaustive account of gender bias in language understanding.

Despite these limitations, WinoBias remains central to the literature because it offers a controlled and interpretable test bed. It has been used to expose substantial gaps in rule-based, feature-rich, neural, contextualized, prompted, and pretrained systems; to evaluate data augmentation, embedding debiasing, neutral rewriting, prompt balancing, early stopping, and linear concept removal; and to seed derivatives such as WinoMT, SoWinoBias, WinoBias+, and WinoIdentity (Zhao et al., 2018). A plausible implication is that WinoBias endures not because it solves the problem of fairness evaluation, but because it provides a compact, balanced, and methodologically reusable substrate on which increasingly sophisticated fairness questions can be posed.