Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

Abstract: Emotion annotation is inherently subjective, yet most NLP pipelines still assume "gold" labels, typically produced by majority voting, and treat annotator variation as noise. In this paper, we present a multilabel emotion annotation case study and use it to examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion classifiers. Rather than collapsing disagreement into a single label, we represent targets as soft vote-share labels (including an intensity-weighted variant) and evaluate models using both thresholded metrics (macro-/micro-F1) and probabilistic alignment (Bernoulli cross-entropy SoftBCE), alongside data-derived disagreement diagnostics. Across annotation regimes, we show that disagreement is structured and leaves measurable traces in model behavior: hard labels may maximize F1 metrics, while soft supervision yields predictions that better reflect empirical annotator variance and uncertainty. Our results provide practical guidance for designing, aggregating, and evaluating multilabel emotion datasets when multiple interpretations are plausible.

• The paper introduces soft and intensity-weighted targets to capture structured ambiguity in multilabel emotion annotations.
• It compares hard, soft, and intensity-based aggregation methods using metrics like Krippendorff’s alpha and macro-F1 scores.
• The study demonstrates that embracing annotator disagreement improves model calibration and better reflects real-world emotion nuances.

Quality and Agreement in Multilabel Emotion Annotation: Disagreement, Soft Aggregation, and Model Alignment

Introduction

This paper addresses the persistent challenge of subjectivity and disagreement in multilabel emotion annotation within NLP. Conventionally, emotion datasets are collapsed into “gold” labels via majority voting, treating inter-annotator variation as noise. The authors challenge this paradigm, arguing that such disagreement encodes real, structured ambiguity and that soft label representations and appropriate evaluation metrics can both preserve and elucidate this uncertainty. The study is anchored on a case analysis of three annotators labeling sentences from Hemingway's The Old Man and the Sea using Plutchik’s expanded emotion taxonomy, with the novel addition of emotion intensity ratings.

Annotation Regime, Annotator Behavior, and Disagreement Patterns

The annotation protocol involved three expert annotators, each allowed to assign multiple Plutchik emotion labels plus intensities (0–10 scale) per sentence. One annotator (Annotator 2) provided sparse labels, while the other two were more consistent, but with substantial variation in their emotion assignments.

Figure 1 visualizes annotation overlap between two annotators, showing systematic confusion among emotion categories, notably between fear and anticipation, and anger and disgust.

Figure 1: Annotation overlap between Annotator 1 and Annotator 3 reveals structured agreement and systematic confusion, indicating well-structured ambiguity in emotion interpretations.

Agreement analysis shows that raw inter-annotator agreement (Krippendorff's $\alpha$) is uniformly low across emotions, confirming the difficulty of emotion interpretation and the inadequacy of traditional categorical agreement metrics for such settings. Pairwise macro-F1 scores increase substantially when filtered to items where both annotators provided labels, separating disagreement from annotator disengagement.

Off-taxonomy “other” emotion labels (e.g., pessimism, sentimentality) emerged frequently, suggesting that even the expanded taxonomy cannot exhaustively capture reader interpretations.

Aggregation Strategies and Data-Derived Uncertainty Measures

To systematize label aggregation, the authors compare three schemas:

• Hard targets: Any emotion selected by any annotator is marked positive (union).
• Soft targets: Probabilities are derived from the empirical vote share for each label.
• Intensity-weighted soft targets: Aggregation weights annotator votes by their self-reported intensity, producing a graded affect signal.

To quantify ambiguity independent of model behavior, the study introduces per-instance disagreement measures: mean per-label Bernoulli variance ($D_{\mathit{var}}$) and pairwise Jaccard disagreement, both derived from raw annotation distributions.

Emotion co-occurrence matrices for each annotator further highlight which emotions are likely to co-occur or be conflated.

Figure 2: Annotator 1’s emotion co-occurrence matrix demonstrates frequent within-item overlap among negative and adjacent affective categories.

Figure 3: Annotator 2’s co-occurrence is sparse, supporting the conclusion of low task engagement rather than true interpretive divergence.

Figure 4: Annotator 3’s co-occurrence emphasizes cross-category ambiguity central to the annotation task.

Evaluation Metrics and Model Diagnostics

For modeling, transformer-based multilabel classifiers (BERT) were trained using both hard and soft targets, with supervision signals encoding either empirical vote shares or intensity-weighted probabilities. Evaluation metrics include standard macro-/micro-F1 (after thresholding predictions) as well as unthresholded Bernoulli cross-entropy (SoftBCE) alignment with the aggregated soft targets.

Results demonstrate that:

• Models trained on hard targets maximize F1, but yield poorly calibrated probabilities that do not reflect annotator uncertainty.
• Training with soft targets (especially intensity-weighted) improves probabilistic calibration—SoftBCE loss is notably reduced—without significantly sacrificing F1, indicating better model alignment with empirical label distributions.

The classifier confusion matrix with hard labels (Figure 5) indicates strong diagonal dominance in high-agreement classes (e.g., sadness, trust) and off-diagonal confusion among negative-valence emotions (e.g., disgust and sadness).

Figure 5: Row-normalized BERT confusion matrix (hard targets) captures robust prediction for clear categories and systematic error among ambiguous classes.

Disagreement diagnostics (Bernoulli variance, Jaccard disagreement) remain stable across aggregation regimes, confirming that these metrics are robust to the modeling choices downstream.

Theoretical and Practical Implications

The findings reinforce the perspective that for subjective NLP tasks, especially emotion recognition, annotator disagreement should be interpreted as a feature of the data, not as annotation error or label noise. The choice of label aggregation—hard vs. soft, intensity weighting, treatment of missing data—influences both model behavior and evaluation. Embracing multiple perspectives via soft targets produces classifiers whose output distributions encode real-world annotation ambiguity, a critical property for downstream applications requiring calibrated uncertainty (e.g., affective computing in social analysis, user-facing decision systems).

Notably, the use of intensity-weighted soft targets outperforms naive soft labels in probabilistic alignment, supporting the value of capturing graded affect in the supervision signal.

There are, however, boundary conditions: More granular or nuanced annotation detail does not always lead to higher thresholded performance. Annotation protocol and supervision schema must be carefully aligned with end task requirements.

Explicit modeling of missing data (versus treating blanks as negatives) is essential for valid agreement and model evaluation, especially as annotation pools scale or diversify.

Future Research Directions

The study points toward several technical avenues:

• Evaluation of alternative soft-label metrics (e.g., Manhattan/Wasserstein distances) on multilabel emotion data
• Incorporation of rare and free-text emotion categories through hierarchical taxonomies or explicit "Other" classes
• Best-worst-scaling for improved intensity annotation reliability
• Larger, more demographically diverse annotator pools to test generalizability of structured disagreement and model calibration effects
• Application of the case-study pipeline to more ambiguous text genres (e.g., social media, dialogue) to measure domain transfer
• Integration of multi-perspective label modeling in downstream tasks requiring user-centered affect analysis

Conclusion

This work establishes a methodological framework for making annotation ambiguity explicit in multilabel emotion recognition. By systematically representing and modeling annotator disagreement, the paper contributes practical tools for transparent evaluation and principled dataset curation. The results support moving beyond static gold labels toward probabilistic targets that acknowledge inherent subjectivity—a necessary step for responsible deployment of affective NLP systems and for bridging the gap between annotation theory and empirical modeling.