LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps

Abstract: Human annotators frequently disagree on emotion labels, yet most evaluations of LLM emotion annotation collapse these judgments into a single gold standard, discarding the distributional information that disagreement encodes. We ask whether LLMs capture the structure of this disagreement, not just majority labels, by comparing emotion judgment distributions between human annotators and four zero-shot LLMs, plus a fine-tuned RoBERTa baseline, across two complementary benchmarks: GoEmotions and EmoBank, totaling 640,000 LLM responses. Zero-shot models diverge substantially from human distributions, and in-domain fine-tuning, not model scale, is required to close the gap. We formalize a lexical-grounding gradient through a quantitative transparency score that predicts per-category human--LLM agreement: LLMs reliably capture emotions with explicit lexical markers but systematically fail on pragmatically complex emotions requiring contextual inference, a pattern that replicates across both categorical and continuous emotion frameworks. We further propose three lightweight post-hoc calibration methods that reduce the distributional gap by up to 14\%, and provide actionable guidelines for when LLM emotion annotations can, and cannot, substitute for human labeling.

• The paper demonstrates that zero-shot LLMs replicate emotion label distributions but fail to capture the nuanced uncertainty evident in human annotations.
• It employs two diverse datasets and metrics like Jensen-Shannon divergence, entropy correlation, and a lexical transparency score to quantify human–LLM judgment gaps.
• Post-hoc calibration methods such as isotonic regression reduce divergence by up to 14%, yet fine-tuned models still outperform zero-shot approaches in reflecting human uncertainty.

Distributional Divergence Between Human and LLM Emotion Annotations

Background and Motivation

The paper "LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps" (2604.27345) addresses a critical gap in the evaluation of LLMs as emotion annotators. Traditional assessments typically reduce the richness of human annotation disagreement to a single "gold" label, losing the distributional information that captures interpretive variability. By treating both human and LLM emotion judgments as probability distributions, the authors provide a structural comparison of uncertainty representation, especially targeting whether LLMs replicate patterns of human label variation (HLV) and uncertainty.

Methodological Framework

Two datasets—GoEmotions (categorical, 28 emotion classes) and EmoBank (continuous VAD)—are employed, both preserving individual annotator labels rather than aggregating them. Four zero-shot LLMs (GPT-5.4-mini, Claude Haiku 4.5, Llama 3.1 8B, Qwen3-8B) and a supervised RoBERTa-base baseline are evaluated across 2,000 stratified texts per dataset, yielding 640,000 LLM responses. The LLMs are assessed using standardized emotion annotation prompts and prediction temperature sampling. Evaluation metrics include Jensen-Shannon divergence (JSD), entropy correlation, per-category profiling, and a novel lexical transparency score combining embedding similarity and lexicon coverage.

Structural Divergence and Uncertainty Representation

All zero-shot LLMs diverge substantially from human emotion distributions (JSD $\geq$ 0.45), whereas the fine-tuned RoBERTa model achieves roughly half this gap. Among zero-shot models, Qwen3-8B exhibits the lowest aggregate JSD and more conservative output. Critically, uncertainty correspondence (Spearman correlation between human and LLM entropy) is weak for all zero-shot models (p $\approx$ 0.20–0.24), while RoBERTa achieves higher alignment (p $\approx$ 0.47). Divergence increases monotonically with human disagreement level; LLMs track full agreement cases better, but fail to capture nuanced uncertainty in ambiguous cases.

Temperature-based sampling increases distributional diversity and marginally improves aggregate alignment, primarily for open-source models. However, higher temperature does not close the structural gap, indicating that sampling randomness cannot replicate genuine human interpretive variation.

Lexical-Transparency Gradient and Emotion-Category Predictability

A key finding is the quantified lexical-grounding gradient. Emotions with explicit lexical markers ("thank", "love", "sad") exhibit higher human-LLM correlation ($\rho$ up to 0.76 for gratitude), whereas pragmatically complex emotions (approval, realization, nervousness) have systematically lower correlations ($\rho$ as low as 0.05–0.18). The combined lexical transparency score significantly predicts per-category human-LLM agreement (Spearman $r = 0.51$, $p = 0.005$), demonstrating that LLM annotation reliability is a function of lexical signal presence.

Cross-dataset replication shows consistent trends: in EmoBank, valence (the most lexically transparent VAD dimension) is predicted much better (Pearson $r = 0.49$–$0.67$) compared to arousal and dominance ($r = 0.19$–$\approx$0), reinforcing the theoretical claim that LLMs leverage surface-level cues but lack pragmatic and contextual inference capabilities.

Post-Hoc Calibration and Model-Specific Profiles

The authors propose three post-hoc calibration methods: temperature scaling, bias correction, and isotonic regression. Isotonic regression yields the most substantial JSD reduction (up to 14%), reflecting a monotonic mapping between LLM and human ordinal emotion intensity. However, these corrections do not close the gap; calibrated zero-shot LLMs still perform distinctly below the fine-tuned baseline in both aggregate and structural metrics.

Model-specific analysis reveals qualitatively different failure modes. GPT-5.4-mini, Claude, and Llama systematically over-predict negative emotions and under-predict neutral, while Qwen3-8B adopts a conservative, balanced strategy, skewing towards neutral and achieving optimal JSD through compressed prediction variance rather than genuine distributional alignment. In EmoBank, low MAE can result from conservative predictions that fail to capture intra-sample variability, underscoring that aggregate metrics are insufficient for structural evaluation.

Practical and Theoretical Implications

The findings provide actionable guidelines for NLP practitioners:

• Emotion Category Caution: Use per-category correlations as the deployment criterion. Categories with correlation $\approx$1 should always involve human oversight.
• Calibration Utility: When a small labeled development set exists, isotonic regression is effective for models with systematic biases.
• Fine-Tuning Supremacy: In-domain supervised training is necessary for capturing human uncertainty; model scale and prompt engineering alone are insufficient.
• Reporting Standards: Distributional metrics, entropy correlation, and per-category profiles are essential alongside aggregate error.

Theoretically, the results underscore essential boundaries in current LLMs’ ability to model interpretive uncertainty, reinforcing the signal-noise distinction in HLV and challenging the ecological fallacy of treating majority aggregation as ground truth. The lexical transparency gradient provides a principled predictor of emotion annotation reliability, suggesting new directions for adaptive annotation protocols and disagreement-aware fine-tuning.

Future Directions

The study highlights several avenues for advancement:

• Scaling open-source models beyond 8B-class, chain-of-thought prompting, and disagreement-aware supervised fine-tuning.
• Cross-lingual replication to account for cultural and linguistic variability in emotion perception.
• Deeper analysis of individual annotator profiles and demographic drivers of disagreement.

Conclusion

The paper demonstrates that zero-shot LLMs replicate emotion label distributions, but not the uncertainty structure encoded by human disagreement. Aggregate metrics can obscure structural misalignment; lexical transparency reliably predicts annotation reliability; and post-hoc calibration improves surface alignment but fails to close deeper gaps. In-domain fine-tuning and distributional evaluation are necessary for genuine alignment. These insights inform both practical annotation pipelines and theoretical understanding of LLM interpretive limitations, motivating research on models that capture richer structure in human emotion perception.