Coreference Confidence Disparity

Updated 8 July 2026

Coreference Confidence Disparity is a phenomenon where model certainty mismatches contextual evidence, often driven by superficial cues and calibration gaps.
Methodological evaluations use antecedent switching and calibrated metrics like ECE to assess models’ sensitivity to ambiguity and fairness across subgroups.
Mitigation strategies include data augmentation, contrastive fine-tuning, and multi-objective training to align model confidence with true evidential structure.

Coreference confidence disparity denotes a family of systematic mismatches between confidence and evidence in coreference resolution. In the narrowest sense, it is the persistence of a model’s preference for one antecedent because of superficial cues such as gender, number, lexical identity, or syntactic position rather than the sentence’s situational meaning. In broader recent usage, it also includes failures to lower confidence when a pronoun is genuinely ambiguous, subgroup differences in confidence across demographic identities, calibration gaps between predicted probability and empirical correctness, and divergences between human annotators’ confidence signals and their antecedent choices. Across these formulations, the common object of study is not only whether a resolver is correct, but whether its certainty is appropriately conditioned on context, ambiguity, and population structure (Emami et al., 2018, Yuan et al., 2023, Shore et al., 17 Sep 2025, Khan et al., 9 Aug 2025, Sabir et al., 12 Jan 2026, Nedoluzhko et al., 24 Jun 2026).

1. Conceptual scope and operationalizations

The term does not have a single canonical formalization. Instead, recent work operationalizes it according to the failure mode under examination. In cue-neutral pronominal anaphora, disparity is a stable confidence gap between candidate antecedents that should reverse under controlled perturbations. In ambiguity-sensitive evaluation, disparity is the failure to shift from decisive resolution to abstention or uncertainty when semantic evidence no longer supports a unique referent. In fairness-oriented work, disparity is the max–min spread in confidence across intersectional subgroups. In calibration work, disparity is the difference in calibration error between male- and female-associated predictions. In annotation-centered corpora, disparity is reflected indirectly in disagreement, context-need, and free-text expressions of doubt (Emami et al., 2018, Shore et al., 17 Sep 2025, Khan et al., 9 Aug 2025, Sabir et al., 12 Jan 2026, Nedoluzhko et al., 24 Jun 2026).

Setting	What varies	Main operationalization
KnowRef	Preference for antecedents under cue-neutralized context	$D_i$ , $\overline{D}$ , switching consistency $C$
AmbiCoref / Correct-Detect	Sensitivity to genuine ambiguity	$EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ , Correct vs. Detect
WinoIdentity	Confidence across intersectional identities	$CC(L_i)$ , $CCD = \Delta CC_G(L)$
Gender calibration	Reliability of pronoun probabilities by gender group	ECE, Gender-ECE
Human-label variation	Confidence signals across annotators	context-need, comments, exact-match agreement

A useful unifying interpretation is that coreference confidence disparity appears whenever the confidence profile remains too stable under a transformation that should alter certainty. The transformation may swap antecedents, toggle a minimal pair from unambiguous to ambiguous, add an identity marker, or expose the same item to multiple annotators. This suggests that the phenomenon is best understood as a discrepancy between confidence and the latent evidential structure of reference resolution rather than as a single benchmark-specific defect.

2. Context sensitivity, superficial cues, and antecedent switching

The KnowRef corpus was introduced precisely to remove gender and number shortcuts from difficult pronominal anaphora. It contains 8,724 instances, with 7,455 training and 1,269 test examples, drawn from 2018 English Wikipedia, OpenSubtitles, and Reddit and filtered to retain sentences with exactly two person-denoting antecedents before a connective and a pronoun after it. Gender-neutralization replaces names so that both antecedents match the pronoun’s gender, thereby removing overt “giveaways.” The test set was labeled by six human annotators; only items with strong agreement were retained, yielding Fleiss’ $\kappa = 0.78$ , and human accuracy on a 100-item subsample was 0.92 (Emami et al., 2018).

In this setting, a per-instance confidence disparity can be written as $D_i = p_i(A) - p_i(B)$ , where $p_i(A)$ and $p_i(B)$ are the model’s probabilities for the two antecedents, and its aggregate magnitude as $\overline{D}$ 0. The central diagnostic, however, is switching consistency. If an original sentence $\overline{D}$ 1 is paired with a version $\overline{D}$ 2 in which all mentions of the two antecedents are swapped, then a context-sensitive model should flip its prediction. Consistency is defined as $\overline{D}$ 3. High $\overline{D}$ 4 indicates confidence driven by context; low $\overline{D}$ 5, including values near zero, indicates confidence disparity due to superficial cues (Emami et al., 2018).

KnowRef shows a large human–model gap under these conditions. On task-specific accuracy, the rule-based resolver scores 0.52, the statistical resolver 0.50, deep RL 0.49, latent tree 0.54, end-to-end CoNLL-only 0.60, fine-tuned BERT on KnowRef 0.61, and end-to-end KnowRef+CoNLL 0.65, against human accuracy of 0.92. The switching results are more diagnostic: the rule-based system has 0% consistency, statistical 76%, latent tree 78%, deep RL 66%, end-to-end 62% when trained on CoNLL and 66–67% when trained on KnowRef or KnowRef+CoNLL, and BERT 69%. The rule-based resolver’s total failure to flip under antecedent exchange is direct evidence that its confidence distribution is anchored to lexicon-level gender or number cues rather than situational semantics (Emami et al., 2018).

Antecedent switching is also used as mitigation. For each instance $\overline{D}$ 6, the procedure swaps all mentions of $\overline{D}$ 7 and $\overline{D}$ 8 in $\overline{D}$ 9, keeps the pronoun unchanged, and flips the correct label to $C$ 0. Adding these switched instances doubles the training set and forces the same contextual scenario to appear with either name attached. On KnowRef, BERT rises from 0.61 to 0.71 task-specific accuracy and from 0.69 to 0.89 consistency; end-to-end KnowRef-only rises from 0.58 to 0.61 accuracy and from 0.66 to 0.71 consistency; end-to-end KnowRef+CoNLL rises from 0.65 to 0.66 accuracy and from 0.67 to 0.75 consistency. On GAP, BERT’s $C$ 1 increases from 69.2 to 71.1, and the female-to-male $C$ 2 ratio moves from 1.02 to 1.00. Before augmentation, fine-tuned BERT is inconsistent on 31% of paired original/switched instances; afterward, this falls to 11% (Emami et al., 2018).

3. Ambiguity sensitivity and the Correct-Detect trade-off

A second line of work studies disparity not as reliance on superficial cues, but as insensitivity to ambiguity. AmbiCoref constructs minimal sentence pairs in which a single verb phrase toggles a pronoun from unambiguous to ambiguous while preserving most of the surface form. The corpus covers four psycholinguistically motivated phenomena—ECO, ECS, IC, and TOP—and contains over 96,000 generated sentences without a train/dev/test split. Human judgments were collected on 625 sentences with three annotations each. For unambiguous items, the intended reading is selected as “likely” or “definitely” 83.2% of the time for ECO, 91.9% for ECS, 85.8% for IC, and 68.3% for TOP; ambiguous counterparts shift substantially toward “Ambiguous” or low-confidence guesses, with TOP approaching an almost uniform response pattern (Yuan et al., 2023).

Because many coreference systems do not expose calibrated probabilities, AmbiCoref evaluates sensitivity through changes in final cluster structure. Pronoun outputs are categorized into five cases: $C$ 3, $C$ 4, $C$ 5, $C$ 6, and $C$ 7, and ambiguity sensitivity is quantified as $C$ 8. Mean EMD is 11.7% for SpanBERT, 3.5% for CoreNLP Neural, 4.0% for NeuralCoref 4.0, 1.2% for CoreNLP Statistical, and 0.6% for CoreNLP Deterministic. Most systems therefore change very little when ambiguity is introduced, even though humans show a marked drop in certainty. The principal failure is misplaced confidence: models treat ambiguous inputs as if they were unambiguous, often linking the pronoun to the first noun phrase regardless of the verb-derived interpretive constraints (Yuan et al., 2023).

Correct-Detect extends this perspective to LLMs by separating two objectives: being correct on unambiguous items and detecting ambiguity on ambiguous ones. Using the 1,930-sentence AmbiCoref set of 962 unambiguous and 968 ambiguous examples, it defines “Correct” as accuracy on unambiguous sentences and “Detect” as the rate of uncertainty-bearing answers on ambiguous sentences. Human judgments under the Reflect condition achieve 76.77% near-correct accuracy and 78.47% Detect. LLMs can approach one objective at the expense of the other: Llama 3.1 Basic reaches 90.33% Correct but only 3.72% Detect, while GPT-4o Basic reaches 87.70% Correct and 22.86% Detect. Detect-oriented prompts invert the pattern: GPT-4o Ambi-Wait attains 99.55% Detect with only 5.23% Correct, and GPT-4o Ambi-CoT reaches 93.31% Detect with 17.08% Correct. The best balances remain far from human performance: GPT-4o Ambi-Ask yields 83.37% Detect with 41.93% Correct, and Llama 3.1 Ambi-Stop yields 44.35% Detect with 61.47% Correct (Shore et al., 17 Sep 2025).

The trade-off is also visible in response stability. Humans reduce per-sentence identical answers across runs from 21.61% on unambiguous items to 9.38% on ambiguous items, a 12.23-point drop. Llama 3.1 shifts only from 25.66% to 23.59%, and GPT-4o from 43.46% to 40.40%. Distributionally, humans move from a U-shaped pattern on unambiguous items toward an inverted U on ambiguous items, whereas LLMs either fail to flip at all or only weakly increase “Ambiguous” responses. In this formulation, coreference confidence disparity is the inability to coordinate decisiveness with abstention: prompt designs that elicit decisive resolution produce overconfidence on ambiguous inputs, while ambiguity-aware prompting induces underconfidence on resolvable ones (Shore et al., 17 Sep 2025).

4. Fairness, subgroup uncertainty, and calibration

A third formulation treats coreference confidence disparity as a group fairness problem. WinoIdentity augments WinoBias with 25 demographic markers across 10 attributes—age, body type, disability, gender identity, language, nationality, sexual orientation, socio-economic status, race, and religion—intersected with binary gender pronouns, yielding 245,700 prompts and 50 distinct bias patterns. For a prompt $C$ 9, coreference confidence is defined as $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 0. For attribute $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 1, CCD is the max–min difference between subgroup averages of $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 2 over all pronoun-by-marker identities $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 3. The metric is explicitly uncertainty-based: it captures whether models are more or less confident for some intersectional identities than for others, even when accuracy alone may appear acceptable (Khan et al., 9 Aug 2025).

The empirical spreads are large. Under referent augmentation, Mistral shows Type-2 CCD values of 0.400 for socio-economic status, 0.392 for body type, 0.382 for disability, 0.358 for language, and 0.160 for race; Llama3 shows 0.382 for socio-economic status, 0.346 for sexual orientation, and 0.251 for body type on Type-2; Mixtral reaches 0.349 for gender identity and 0.346 for sexual orientation on Type-1; Falcon reaches 0.251 for disability on Type-2. The greatest harms occur for doubly-disadvantaged identities in anti-stereotypical contexts. For Mistral on mechanic in Type-2 contrastive augmentation, the average confidence margin is $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 4 for fem, $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 5 for transgender_fem, and $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 6 for gay_fem, where negative values indicate confident, incorrect assignments. Non-referent augmentation can even raise accuracy while confidence falls, demonstrating that accuracy can obscure disparity. The paper also reports that confidence decreases even for hegemonic markers, which it interprets as evidence of memorization rather than logical reasoning (Khan et al., 9 Aug 2025).

Calibration work refines this fairness-oriented view by asking whether token-level pronoun probabilities are equally reliable across gender groups. “The Confidence Trap” evaluates six open-weight LLMs on WinoBias, Winogender, GenderLex, and WinoQueer, extracting the pronoun probability $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 7 directly from logits and measuring ECE, ICE, MacroCE, Brier score, and Gender-ECE. Gender-ECE averages subgroup ECE over predicted male and predicted female outputs, thus turning calibration into a fairness-aware diagnostic. Across GenderLex, WinoBias, and Winogender, Gemma-2-9B is the worst-calibrated model: its ECE is 0.327 on GenderLex, 0.429 on WinoBias, and 0.373 on Winogender. Pronoun-specific asymmetries are often sharper than aggregate ECE suggests. On WinoBias, Gemma-2-9B has male ECE 0.067 versus female ECE 0.895; on GenderLex, male ECE 0.056 versus female ECE 0.901. Most models are worse calibrated for female pronouns. Beta calibration reduces ECE about threefold across models, isotonic regression yields similar improvements, temperature scaling collapses probabilities toward approximately 0.5, and Platt scaling pushes probabilities toward extremes (Sabir et al., 12 Jan 2026).

These studies collectively show that fairness-related confidence disparity is not reducible to error rates. A model may remain accurate on average while being systematically less certain, less calibrated, or confidently wrong for some subgroups. This suggests that demographic auditing in coreference resolution requires direct inspection of confidence margins and calibration structure rather than sole reliance on aggregate correctness.

5. Human uncertainty, disagreement, and cognitive constraints

Human data show that confidence disparity is not only a model property. The Hlava Cor corpus, built for Czech coreference, contains 1,024 contexts annotated in parallel by three annotators, with mandatory comments and a context-need field coded as 0, 1, or 2 depending on how much additional context was required. The paper reports 49% full agreement, 83% partial agreement, and 60.4% average pairwise agreement. Agreement is lowest for pronominal coreference with to ‘it’ at 38% full agreement, higher for full noun phrases at 51%, and highest for local tam ‘there’ at 57%. Spoken texts reach 53% full agreement versus 45% for written texts. Most notably, items in the model-disagreement subset yield only 39% full agreement, compared with 66% in the model-agreement subset. Confidence is not numerically rated beyond the context-need field, but annotator comments record doubt, alternative analyses, and divergent referential granularity. A representative example involves vozidlo ‘vehicle’ and vůz ‘car’: all annotators assign context-need 0, yet still disagree on antecedent choice because they interpret the referential scope differently (Nedoluzhko et al., 24 Jun 2026).

This makes two points about the concept. First, confidence disparity can arise even when additional context is not judged necessary; it may reflect interpretive criteria rather than informational insufficiency. Second, model disagreement can be a useful heuristic for locating cases that are also difficult for humans, but model agreement does not imply human unanimity. Hlava AD, the related discourse-relation corpus, shows the same pattern of label variation: only 38% of items receive a single solution among five annotators, average pairwise agreement is 64.9%, and syntactic complexity substantially increases disagreement (Nedoluzhko et al., 24 Jun 2026).

A complementary human–model comparison appears in work grounded in dual-process theory. Using constrained-time question answering and self-paced reading, that study defines $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 8 as the accuracy gap between pro-stereotypical and anti-stereotypical instances and $EMD(P_{\text{unambiguous}}, P_{\text{ambiguous}})$ 9 as the response-time gap between anti- and pro-stereotypical pronouns. Humans make approximately 3% more gender-biased decisions than models on real-world BUG data, whereas models are approximately 12% more biased on synthetic data. On BUG, human $CC(L_i)$ 0 rises from 5.0 at $CC(L_i)$ 1 to 6.5 at $CC(L_i)$ 2 and 6.9 at $CC(L_i)$ 3, showing stronger stereotype effects as processing time is reduced. In the self-paced reading task, positive $CC(L_i)$ 4 emerges after about 50% of annotations on BUG but only after at least 80% on synthetic data, indicating earlier and more pronounced processing difficulty for anti-stereotypical cases in natural sentences (Lior et al., 2023).

Together, these results locate coreference confidence disparity partly in human cognition. Humans lower certainty, slow down, or diverge from one another when ambiguity, stereotype incongruence, or interpretive complexity rises. Models often fail to reproduce these confidence gradients, either remaining too stable across ambiguity manipulations or clustering their errors around strong social priors.

6. Evaluation practice, mitigation strategies, and unresolved problems

The literature converges on the view that standard single-label accuracy is insufficient for diagnosing confidence disparity. KnowRef recommends reporting task-specific accuracy together with switching consistency $CC(L_i)$ 5, because a model can attain nontrivial accuracy while still failing to flip under antecedent exchange. AmbiCoref recommends ambiguity-aware evaluation through minimal pairs and distributional change metrics such as EMD over $CC(L_i)$ 6, since CoNLL-style correctness scores do not reveal whether confidence drops when ambiguity rises. Correct-Detect argues for separate reporting of strict and near accuracy on unambiguous cases and explicit ambiguity detection on ambiguous cases. WinoIdentity shows that subgroup accuracy disparities should be paired with subgroup confidence spreads. Calibration work adds ECE, Gender-ECE, ICE, MacroCE, and Brier score to quantify whether confidence itself is trustworthy (Emami et al., 2018, Yuan et al., 2023, Shore et al., 17 Sep 2025, Khan et al., 9 Aug 2025, Sabir et al., 12 Jan 2026).

Mitigation strategies are similarly diverse. Antecedent switching is the most concrete intervention validated on a benchmark: it doubles the training set, discourages reliance on names, and improves both accuracy and consistency on KnowRef while also improving GAP performance (Emami et al., 2018). AmbiCoref-style contrastive fine-tuning and ambiguity-aware evaluation are proposed to increase sensitivity to verb-argument constraints and to penalize overconfident commitments on ambiguous inputs (Yuan et al., 2023). Correct-Detect proposes a multi-objective framing that balances correctness and ambiguity detection, and it recommends thresholding by entropy or margin when models expose probability distributions, as well as structured outputs that include explicit confidence scores; these are presented as useful calibration-oriented extensions rather than reported experimental results (Shore et al., 17 Sep 2025). WinoIdentity suggests pairing traditional fairness audits with uncertainty-based measures, since accuracy gains can coexist with increased confidence disparity (Khan et al., 9 Aug 2025). The confidence-calibration literature recommends post-hoc Beta calibration or isotonic regression and warns against treating raw token probabilities as deployable confidence in sensitive settings (Sabir et al., 12 Jan 2026). Hlava Cor argues for multi-annotator datasets, mandatory comments, and confidence-aware representations instead of collapsing inherently variable cases to a single gold antecedent (Nedoluzhko et al., 24 Jun 2026).

Several open problems recur. KnowRef is limited to sentences with two person-type antecedents and single pronouns, and the generalization of switching-based consistency to multi-sentence, multi-entity discourse remains open (Emami et al., 2018). AmbiCoref and Correct-Detect are English-only and focus on short sentence-level contexts (Yuan et al., 2023, Shore et al., 17 Sep 2025). WinoIdentity is US-centric, uses binary gender pronouns, and faces the combinatorial growth typical of intersectional evaluation (Khan et al., 9 Aug 2025). Hlava Cor is Czech-only and does not provide formal statistical tests linking context-need to disagreement (Nedoluzhko et al., 24 Jun 2026). Calibration studies emphasize that better ECE does not remove the underlying sources of bias, only the mismatch between confidence and empirical correctness (Sabir et al., 12 Jan 2026).

In aggregate, the field has shifted from asking whether a coreference system is correct to asking whether it is confident for the right reasons, uncertain in the right places, and equitable in how that uncertainty is distributed. This suggests that future progress will depend less on a single benchmark and more on joint evaluation of context sensitivity, ambiguity awareness, calibration, subgroup fairness, and human label variation.