---
title: Empirical Annotation Distributions
url: https://www.emergentmind.com/topics/empirical-annotation-distributions
type: topic
---

# Empirical Annotation Distributions

Empirical annotation distributions are probability distributions derived directly from the full set of human (or artificial) annotations assigned to data instances in subjective or ambiguous labeling tasks. Distinct from the conventional reduction to majority vote or mean values, these distributions capture the pattern of inter-annotator agreement, disagreement, ambiguity, and uncertainty intrinsic to the annotation process. They serve as foundational signals in modeling subjectivity, ambiguity, and human variability in domains such as affective computing, conversational systems, subjective classification, linguistic sense annotation, and biological curation.

## 1. Formal Definitions and Core Constructs

Empirical annotation distributions—or "soft labels"—consist of the normalized counts or frequencies of responses for each possible category (or score) on a given item, optionally including abstention or “can’t solve” (cs) categories. For categorical labeling with $C$ classes, $R$ annotators, and explicit abstentions:
$$
q = (q_1, ..., q_C, q_{cs}), \qquad q_k = \frac{\#\text{annotators choosing }k}{R}
$$
where $q_{cs}$ denotes the fraction abstaining [2510.04366].

For continuous annotation (e.g., affect dimensions), empirical distributions are often constructed by binning annotator scores (histogram) or by nonparametric kernel density estimation (KDE) in the label space [2408.11956].

In more specialized contexts, empirical annotation distributions can refer to distributions over complex spaces:
- Over slot-value sets in dialog state tracking [2010.12758]
- Over semantic similarity ratings in sense annotation [2311.12664]
- Over word type frequencies for text annotation meta-analysis [1208.2175]

These distributions retain all observable aspects of annotator variability, including consensus (sharp peaks), disagreement (spread), and abstention (mass on cs).

## 2. Methods of Collection and Construction

Empirical annotation distributions are constructed under various annotation regimes:

- **Exhaustive annotation**: Each example is labeled by $R$ annotators. The empirical distribution is computed directly from counts [2510.04366].
- **Sparse/crowdsourced annotation**: Each annotator labels only a subset. Soft distributions are defined over the observed subset, but this can introduce bias when annotators differ systematically or sample sizes are small [2305.15070].
- **Augmentation/Imputation**: Missing annotations are predicted via imputation models (matrix factorization, neural collaborative filtering, or multitask annotator models), with the resulting distributions reflecting both observed and imputed responses. This enables estimation of full distributions even under partial observation at the cost of model-induced distributional bias [2305.15070].

Continuous annotation distributions may be constructed by:
- Binning continuous annotator outputs and normalizing counts
- Applying kernel density estimators (classically non-parametric, or via differentiable “soft-histogram” layers in an end-to-end model) [2408.11956]

In dialog or semantic annotation (e.g., NUANCED, DURel), distributions reflect ontological preference or proximity ratings, respectively, possibly structured as products of distributions across multiple slots or semantic axes [2010.12758, 2311.12664].

## 3. Modeling, Inference, and Evaluation Frameworks

Empirical annotation distributions underpin several classes of statistical and machine learning models:

- **Direct distribution modeling**: Models are trained to predict the full response distribution rather than a single label. For categorical outputs this can be via cross-entropy or mean squared error on soft labels [2010.12758, 2510.04366]. For continuous outputs, models predict discretized histograms or parametric (e.g., Beta) distributions over the response space [2408.11956, 2604.07198*].
- **Annotator modeling**: To preserve fine-grained human variability, models with per-annotator heads or learned annotator embeddings are trained to predict each annotator's response; the empirical distribution is then obtained by aggregating these predictions. This improves the modeling of variance, skew, and multi-modality, as demonstrated in speech emotion recognition [2408.11956].
- **Ambiguity and uncertainty quantification**: The full distribution enables calculation of entropy, Gini impurity, or specifically designed ambiguity measures (e.g., $\alpha(q)$, which reflects both class confusion and unresolvability via "can't solve") [2510.04366].
- **Agreement/Calibration**: Empirical distributions enable measurement of inter-annotator reliability (Cohen’s $\kappa$, Krippendorff’s $\alpha$), divergence from computational annotator distributions (KL, Jensen-Shannon), and calibration of automated prediction models [2311.12664, 2405.01299].

*Methodological Note: Details for "modeling annotation consensus using the Beta distribution" as in [2604.07198] were not available in the supplied content.

## 4. Empirical Characterization and Statistical Summaries

Empirical annotation distributions support detailed statistical analyses:

- **Error and disagreement distribution**: In geometric annotation (e.g., polygon boundaries), per-sample distributions of distances (e.g., boundary mean $d_B$) reveal systematic effects of object shape, quality assurance, and image complexity. Skewness, kurtosis, and fitted parametric forms (log-normal, gamma) characterize these patterns [2311.02707].
- **Ambiguity measures and inference**: $\alpha(q)$ provides a scalar summary of distributional ambiguity (aleatoric uncertainty), with Bayesian inference under Dirichlet priors yielding credible intervals and posterior densities for $\alpha$ [2510.04366].
- **Sense frequency and semantic change**: In sense annotation, empirical distributions over clusters of usage types yield sense-frequency distributions, time-series of change, and entropy-based measures of semantic variation [2311.12664].
- **Zipfian distribution analysis**: In large-scale text annotation, the empirical word-frequency distribution is used as a diagnostic for annotation "effort" and information density, fitting discrete power-laws to compare manual vs automatic curation [1208.2175].

Empirical measures such as mean absolute error (MAE), total variation distance (TVD), Jensen–Shannon divergence (JSD), and Earth Mover’s Distance (EMD) are applied to compare predicted to gold soft-label distributions [2010.12758, 2408.11956, 2305.15070, 2405.01299].

## 5. Applications Across Domains

Empirical annotation distributions are essential in:

- **Subjectivity modeling**: Training and evaluating models that reflect genuine human disagreement in NLP, sentiment, offensiveness, or dialog state tasks [2010.12758, 2305.15070, 2405.01299].
- **Downstream machine learning**: Soft labels derived from empirical distributions enable models to learn aleatoric uncertainty directly—either via probabilistic loss or as example weighting (e.g., weighted by $1-\alpha(q)$) [2510.04366].
- **Quality assurance and dataset curation**: Distributions highlight high-ambiguity or poorly resolved items for re-annotation, guideline revision, or exclusion [2510.04366, 2311.02707].
- **Evaluation of AI annotators (LLMs)**: Direct elicitation of opinion distributions from large language models enables benchmarking of their ability to reproduce the patterns of human disagreement, revealing systematic deviations such as distributional flattening, bimodality, or bias toward specific classes [2405.01299].
- **Sense and semantic change analysis**: Aggregated distributions over semantic similarity ratings provide robust tools for tracking shifts in meaning, frequency, or sense usage across time and corpora [2311.12664].

Empirical distributions have further been used in imputation to expand the set of available annotations, with careful analysis revealing that many imputation methods tend to contract distributional diversity and overemphasize consensus [2305.15070].

## 6. Limitations, Bias, and Methodological Cautions

Several empirical and methodological challenges are intrinsic to annotation distributions:

- **Bias from survey design or annotator sampling**: Sparse coverage or imputation can bias the soft label toward over-smoothing and majority collapse, as in collaborative filtering or multitask imputation models [2305.15070].
- **Artificial flattening or bimodality in AI annotators**: LLMs prompted to output direct probability assignments tend toward a limited set of probability patterns and may systematically favor the positive class ("false positive" bias >90%), failing to capture the spectrum of human uncertainty [2405.01299].
- **Representativeness and calibration**: Naive approaches to probability elicitation from LLMs or end-to-end predictors may yield uncalibrated distributions with altered entropy structures, deviating from the true empirical human response pattern [2405.01299].
- **Ambiguity metric limitations**: Most scalar indices (entropy, Gini, etc.) fail to capture the full structure of ambiguity—hence the need for measures like $\alpha(q)$ that treat abstention and class confusion asymmetrically [2510.04366].
- **Task and domain dependence**: Empirical annotation distributions and their statistical regularities are sensitive to task structure, annotation guidelines, and data domain (e.g., shape complexity in polygon annotation, or ontology design in dialog) [2311.02707, 2010.12758].

## 7. Summary Table: Key Empirical Annotation Distribution Frameworks

| Domain/Task                  | Empirical Distribution Definition                    | Notable Evaluation Metrics / Effects                               |
|------------------------------|-----------------------------------------------------|--------------------------------------------------------------------|
| Categorical Classification   | $q = (q_1, ..., q_C, q_{cs})$                      | Entropy, Gini impurity, ambiguity $\alpha(q)$, KL/JSD divergence   |
| Continuous Affect            | KDE or binned histogram of annotator scores         | TVD, JSD, CCC, soft-histogram cross-entropy                       |
| Dialog/Slot-filling          | Distributions over slot values per utterance        | MAE between predicted/empirical priors, slot-update accuracy       |
| Semantic Sense Annotation    | Median-based similarity ratings; cluster frequencies| Sense-frequency distributions, temporal $\ell_1$-distance          |
| Polygon Annotation (vision)  | Distributions of boundary distances/errors          | Mean, std. dev., skewness, fitted parametric models                |
| Bulk Biological Annotation   | Word frequency distributions (Zipfian analysis)     | Power-law exponent $\alpha$, audience-/annotator-effort balance    |

## References

- [2010.12758] NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions
- [2311.12664] The DURel Annotation Tool: Human and Computational Measurement of Semantic Proximity, Sense Clusters and Semantic Change
- [2408.11956] The Whole Is Bigger Than the Sum of Its Parts: Modeling Individual Annotators to Capture Emotional Variability
- [2510.04366] Quantifying Ambiguity in Categorical Annotations: A Measure and Statistical Inference Framework
- [2311.02707] An Empirical Study of Uncertainty in Polygon Annotation and the Impact of Quality Assurance
- [2308.07179] Incorporating Annotator Uncertainty into Representations of Discourse Relations
- [2305.15070] Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions
- [1208.2175] An approach to describing and analysing bulk biological annotation quality: a case study using UniProtKB
- [2405.01299] The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation

Source: https://www.emergentmind.com/topics/empirical-annotation-distributions