LeWiDi in NLP: Modeling Annotator Disagreement

Updated 15 July 2026

Learning With Disagreements (LeWiDi) is a framework that preserves multiple annotator judgments in subjective NLP tasks, enabling nuanced soft-label and perspectivist predictions.
It contrasts distributional methods, which predict empirical label distributions, with perspectivist approaches that model individual annotator views using tailored metrics like cross-entropy, Manhattan, and Wasserstein distances.
LeWiDi benchmarks and shared tasks drive iterative advances in disagreement-aware metrics, annotator representation, jury learning, and in-context learning for superior model evaluation.

Learning With Disagreements (LeWiDi) denotes a disagreement-aware approach to supervised learning in which the variation among human judgments is preserved and modeled rather than collapsed into a single reconciled label. In NLP, the term is closely associated with the LEWIDI shared-task series, whose stated objective is to promote a framework for training and evaluating models on datasets that retain annotator disagreement, especially for subjective tasks where majority-vote aggregation is a misrepresentation of the data (Leonardelli et al., 2023, Leonardelli et al., 9 Oct 2025).

1. Conceptual foundations

LeWiDi starts from the observation that annotated NLP corpora are often “rife with disagreements between the judges,” particularly for subjective tasks such as sentiment analysis, offensive language detection, misogyny, sexism, sarcasm, irony, and related classifications (Leonardelli et al., 2023). In this setting, disagreement is not treated as mere noise. The central methodological claim is that preserving the full set of annotator judgments yields a more faithful representation of subjectivity, ambiguity, and plural interpretation than majority-vote aggregation.

This orientation is explicit in the shared-task literature. The second LeWiDi shared task argued that rather than eliminating disagreements from annotated corpora, they should be preserved, and that this is especially important for subjective NLP tasks (Leonardelli et al., 2023). The third edition generalized this position into a broader design principle: AI models should be trained to be aware of the possibility of variation and disagreement in human judgments, and should be evaluated by their ability to recognize such variation (Leonardelli et al., 9 Oct 2025).

A recurrent distinction in the literature is between modelling disagreement and modelling perspectives. Distributional approaches predict the empirical label distribution for an item, whereas perspectivist approaches attempt to predict how specific annotators, or groups of annotators, would label that item. The contrast is explicit in work that frames the choice as one between distributional soft-labelling and modelling perspectives of individual annotators or groups thereof, and argues that the choice of evaluation metric can privilege one philosophy over the other (Vitsakis et al., 2023). Related work on annotator representation learning similarly states that disagreement may arise from subjectivity of the task, difficult cases, unclear guidelines, and other factors, and therefore should be modeled directly rather than discarded via aggregation (2305.14663).

2. Problem formulations

Two formulations dominate LeWiDi research. The first is soft-label prediction, in which the target for each item is the empirical probability distribution induced by multiple annotators. The second is perspectivist prediction, in which the target is the label that a specific annotator would assign. LeWiDi-2025 formalized these as two complementary paradigms: the soft-label approach, where systems predict population-level distributions of judgments, and the perspectivist approach, where systems predict the interpretations of individual annotators (Leonardelli et al., 9 Oct 2025).

In soft-label settings, supervision is distributional. A representative formulation is the soft cross-entropy objective used for sexism identification under the learning-with-disagreements regime:

$\mathcal{L}_\text{CE}(x) = - \sum_{i=1}^C p_i \log \hat{p}_i$

where $\mathbf{p}$ is the empirical annotator distribution and $\hat{\mathbf{p}}$ is the model prediction (Paula et al., 2023). This preserves the proportion of votes for each class rather than projecting them onto a single hard label.

In perspectivist settings, the model conditions on annotator identity or annotator-related features. A general objective given for annotator-aware transformer models is

$\theta^* = \arg\max_\theta \sum_{i=1}^E \log P(y_i \mid x_i, a_j; \theta),$

with repeated texts retained as separate training instances for different annotators and no aggregation applied (2305.14663). This formulation makes annotator-specific prediction a first-class target rather than an auxiliary variable.

A further extension is jury learning, which asks not only how to predict disagreement, but whose labels a model should emulate. Jury learning models every annotator in a dataset, samples jurors according to a practitioner-specified composition, and aggregates their predictions. This architecture enables juries that dynamically adapt their composition, explore counterfactuals, and visualize dissent (Gordon et al., 2022). This suggests a stronger form of perspectivism in which aggregation is explicit, configurable, and normatively interpretable.

3. Shared-task evolution and benchmark design

The LeWiDi shared tasks provide the principal benchmark infrastructure for the field. The 2023 SemEval edition was the second LeWiDi shared task and differed from the first edition in three respects stated explicitly in the task overview: it focused entirely on NLP rather than NLP and computer vision; it focused on subjective tasks; and for evaluation it concentrated on soft approaches (Leonardelli et al., 2023). The datasets were HS-BREXIT, ArMIS, CONVABUSE, and MD-AGREEMENT, covering hate, offensiveness, sexism, misogyny, and abusive language, with access to both individual annotations and derived soft labels (Leonardelli et al., 2023).

The 2025 edition, presented as the third edition of the shared task, extended the benchmark to four datasets spanning paraphrase identification, irony detection, sarcasm detection, and natural language inference, and introduced both categorical and ordinal labeling schemes (Leonardelli et al., 9 Oct 2025).

Dataset	Task	Labeling
CSC (≈7,000 pairs; 31,984 annotations)	Sarcasm detection	Likert scale $[1,6]$
MP (18,778 pairs; 94,342 annotations)	Irony detection	Binary $[0,1]$
Par (500 pairs; 2,000 annotations)	Paraphrase detection	Likert scale $[-5,5]$
VEN (500 items; 1,933 explanations)	Natural language inference	Multilabel: Entailment, Neutral, Contradiction

LeWiDi-2025 also emphasized annotator-level metadata and harmonized formatting. The task overview reports 53 registrants, 15 teams submitting, and 9 system papers, while the 2023 overview reports over 130 registered teams, 29 submitting, and 13 system description papers (Leonardelli et al., 9 Oct 2025, Leonardelli et al., 2023). This benchmark trajectory indicates a shift from demonstrating the feasibility of disagreement-aware evaluation toward building a reusable infrastructure for comparative model development.

4. Model families and technical strategies

A first family of methods learns directly from soft labels without using aggregated hard targets. In the EXIST 2023 sexism characterization system, mBERT and XLM-RoBERTa were fine-tuned on empirical annotator distributions, with ensembling by mean probability and an optional adjustment to the closest valid distribution matching feasible annotator distributions (Paula et al., 2023). The same paper reports that the ensemble approach outperformed the individual models under both soft and hard evaluations, obtaining fourth place in Task 2 and first place in Task 3, with the highest ICM-Soft of -2.32 and a normalized ICM-Soft of 0.79 (Paula et al., 2023).

A second family introduces explicit annotator representations. One line of work adds learnable annotator embeddings and annotation embeddings to transformer models, modifies the $[CLS]$ representation by weighted annotator- and annotation-specific terms, and evaluates the resulting models on TID-8, a benchmark of eight language-understanding datasets with inherent disagreement (2305.14663). That study reports improvements on six datasets in TID-8 while increasing model size by fewer than 1% parameters, with gains up to 17% over baselines on datasets with substantial annotator disagreement (2305.14663).

A third family operationalizes strong perspectivism through annotator-specific heads or tasks. The iLab SemEval-2023 system used a multi-task BERT-based architecture with one output head per annotator, trained by summing individual cross-entropies across observed annotations (Vitsakis et al., 2023). The authors found that this multi-task approach performed poorly on datasets containing distinct annotator opinions, and argued that although strongly perspectivist approaches may not optimize the shared-task metrics used by distributional approaches, they can preserve minority views more explicitly (Vitsakis et al., 2023).

LeWiDi-2025 intensified two additional trends. One was disagreement-aware distributional modeling with richer annotator features. The LPI-RIT system adapted DisCo by incorporating annotator metadata such as age, nationality, gender, and education, modernizing the MLP used to combine item and annotator embeddings, and replacing the loss with a metric-aligned multi-objective variant; the best reported weighting for the combined Wasserstein and MAE loss was $\alpha = 0.6$ (Sawkar et al., 11 Aug 2025). The other was large-language-model-based in-context learning. Opt-ICL conditioned predictions on a rater’s previous labels and used a two-step meta-learning procedure consisting of post-training on many in-context datasets and dataset-specific fine-tuning, while DeMeVa compared similarity-based and stratified label-based prompt sampling and combined in-context learning with label distribution learning using RoBERTa (Sorensen et al., 8 Oct 2025, Ignatev et al., 11 Sep 2025).

5. Evaluation metrics and empirical findings

Metric design is central in LeWiDi because disagreement-aware systems are not well characterized by conventional hard-label accuracy alone. The 2023 shared task used cross-entropy between predicted and gold soft-label distributions as the primary metric:

$CE(p, q) = -\sum_i p_i \log(q_i),$

with micro-F1 on majority labels as a secondary hard metric (Leonardelli et al., 2023). The task overview notes that cross-entropy gives a more nuanced signal than hard-label evaluation when annotator agreement is low (Leonardelli et al., 2023).

LeWiDi-2025 moved beyond standard cross-entropy and introduced dataset- and structure-sensitive metrics. For soft-label prediction it used Average Manhattan Distance (AMD) for binary and multiclass data and Average Wasserstein Distance (AWD) for ordinal data. For perspectivist evaluation it used Average Error Rate (AER) for categorical labels and Average Normalized Absolute Distance (ANAD) for ordinal labels, together with multilabel extensions such as MAMD and MER (Leonardelli et al., 9 Oct 2025). Representative formulas include

$\mathbf{p}$ 0

and

$\mathbf{p}$ 1

These metrics explicitly measure not only whether a prediction is wrong, but how far it is from the observed human distribution or ordinal judgment (Leonardelli et al., 9 Oct 2025).

The empirical pattern reported for LeWiDi-2025 is that LLM-based approaches with in-context learning consistently ranked highest overall across datasets and tasks, especially when they incorporated annotator-specific context (Leonardelli et al., 9 Oct 2025). Opt-ICL was the overall winner on both tasks, and its ablations showed that including many previous ratings from the target rater in context was crucial, dataset-specific fine-tuning was especially helpful on larger datasets, post-training on other in-context datasets helped on MP, and performance improved with model scale (Sorensen et al., 8 Oct 2025). DeMeVa placed second overall and reported that in-context learning can effectively predict perspectivist annotations and that aggregating those predictions into soft labels yields competitive performance, while label distribution learning remains promising for soft-label prediction (Ignatev et al., 11 Sep 2025).

The LPI-RIT system paper provides a concrete illustration of metric-aligned improvement: adapting DisCo produced reductions such as CSC Soft from 1.45 to 0.87 in Wasserstein Distance, CSC PE from 0.33 to 0.22 in Absolute Distance, MP Soft from 0.54 to 0.45 in Manhattan Distance, and Par Soft from 3.71 to 2.21 in Wasserstein Distance (Sawkar et al., 11 Aug 2025). By contrast, a test-time scaling study found that simple Model Averaging and Majority Voting improved LLM performance consistently on LeWiDi tasks, whereas Best-of-N sampling did not transfer well from mathematics to disagreement-rich NLP tasks (Ruiz et al., 14 Oct 2025).

6. Debates, limitations, and open directions

The main controversy in LeWiDi concerns the relationship between distributional modeling and perspectivist modeling. Distributional methods align well with soft-label metrics and often perform strongly in shared tasks, but perspectivist work argues that these metrics may obscure the value of models that preserve identifiable viewpoints. The iLab system paper explicitly contends that while strongly perspectivist approaches might not achieve state-of-the-art performance under distributional evaluation metrics, they allow a more nuanced understanding of individual perspectives and enable decision makers to amplify minority views (Vitsakis et al., 2023). Jury learning radicalizes this position by making the composition of the “jury” itself an explicit, controllable part of inference (Gordon et al., 2022).

Several technical limits remain open. The LeWiDi-2025 overview notes that all annotators in the test sets were also seen during training, so generalization to completely unseen raters remains unresolved (Leonardelli et al., 9 Oct 2025). The benchmark also remains text-centered, leaving multimodal disagreement-aware modeling comparatively underdeveloped (Leonardelli et al., 9 Oct 2025). Another issue is metric comparability: some metrics, especially Wasserstein-based scores on ordinal datasets, have dataset-dependent absolute interpretations, which complicates cross-dataset comparison (Leonardelli et al., 9 Oct 2025).

A broader methodological implication is that LeWiDi is not a single model class but a family of representational and evaluative commitments. This suggests that future progress is likely to depend less on replacing one architecture with another than on better coupling among data collection, annotator modeling, metric design, and downstream decision rules. The recent literature already points in that direction: some systems emphasize metadata and loss reweighting, some condition on prior rater examples, some learn annotator embeddings, and some treat the choice of whose labels to emulate as the primary design question (Sawkar et al., 11 Aug 2025, Sorensen et al., 8 Oct 2025, 2305.14663, Gordon et al., 2022).