---
title: Annotator-Centric Active Learning
url: https://www.emergentmind.com/topics/annotator-centric-active-learning-acal
type: topic
---

# Annotator-Centric Active Learning

Annotator-Centric Active Learning (ACAL) encompasses a family of strategies for active learning that explicitly integrate the characteristics, behaviors, cognitive states, and diverse perspectives of human annotators into the data selection, annotation, and model evaluation loop. ACAL recognizes that, in practical annotation scenarios, annotators are non-ideal: they may be inconsistent, bring differing expertise or subjectivity, experience fatigue, and contribute valuable rationales. By systematically modeling and leveraging these factors, ACAL aims to improve the efficiency, accuracy, and diversity of machine learning model training, particularly for subjective, complex, or cost-sensitive tasks.

## 1. Key Principles and Theoretical Foundation

At the foundation of ACAL is the recognition that annotation quality and annotation cost are heterogeneous across annotators, instances, and time. Unlike traditional active learning frameworks that assume a perfect, omniscient oracle, ACAL relies on models and strategies that address annotator variability, subjectivity, and cognitive factors:

- **Annotator Performance Modeling:** Annotator performance is formalized via mappings such as $\psi: Q \times A \to \mathbb{R}$, estimating for each query-annotator pair the expected annotation quality based on historical data, expertise, or internal state [2109.11301].
- **Query–Annotator Pair Selection:** Data selection couples sample informativeness (e.g., uncertainty, diversity, region coverage) with annotator suitability, allowing dynamic allocation that maximizes information gain per annotation cost [2109.11301, 2507.23756].
- **Incorporation of Rationale and Explanation:** Annotator rationales, such as feature rankings or natural language explanations, are incorporated into acquisition or disagreement measures, e.g., by weighting committee members by their fidelity to annotator-provided rankings using Kendall’s tau, or by using explanations to guide model learning and selection [2009.04568, 2305.12710].
- **Diversity of Human Judgments:** Especially in subjective NLP tasks, ACAL explicitly seeks to approximate the soft label distribution by sampling annotators to achieve broad coverage of differing judgments, supporting both minority and majority perspectives equally [2404.15720].
- **Cognitive State Modeling:** Internal annotator states such as mood and fatigue are factored into annotator selection heuristics or recommendation systems, enabling real-time adaptation to maximize annotation quality [2507.23756].

## 2. Methodological Variants and Architectural Components

ACAL systems manifest a range of architectural augmentations and query strategies:

| Methodological Feature                   | Implementation Example                  | Benefit                                                      |
|:-----------------------------------------|:----------------------------------------|:-------------------------------------------------------------|
| Annotator selection after data sampling  | Two-stage selection (data, then annotator) [2404.15720]        | Efficiently covers diversity in subjective labeling           |
| Annotator-centric head architectures     | Multi-head BERT models with one head per annotator [2310.14979] | Models individual behaviors; captures disagreement            |
| Rationale-based weighting                | Model committee weighting by feature ranking agreement [2009.04568] | Integrates expert rationale into sample selection             |
| Mood/fatigue-aware selection             | Knowledge-based recommender using accuracy, mood, and fatigue [2507.23756] | Reduces label errors due to annotator state fluctuations      |
| Noise filtering and resampling           | Partitioning and resampling of “noisy” regions [2504.04506]    | Maintains label quality, especially in low-budget regimes     |
| Explanation-augmented selection          | Using semantic similarity to explanations for sample ranking [2305.12710] | Guides data selection by rationale diversity                  |

Local explanation techniques (such as LIME or SHAP) are pivotal where feature-level annotator rationales are involved, and annotator history (such as semantic diversity or representation-based vectors) underpins sampling strategies in subjective tasks [2404.15720, 2305.12710]. Advanced strategies, such as using bandit algorithms for source selection in video annotation [2402.06560], further generalize ACAL to complex media.

## 3. Evaluation Metrics and Human-Centric Assessment

Traditional metrics (e.g., macro F₁, accuracy, Jensen–Shannon divergence) are often extended in ACAL with annotator-centric metrics:

- **Per-Annotator F₁ and JS Divergence:** Compute and aggregate performance metrics per annotator, quantifying how well minority perspectives are captured [2404.15720].
- **Worst-Off Annotator Metrics:** Evaluate model performance for the least-well-represented annotators, applying fairness concepts to ensure no viewpoints are neglected.
- **Uncertainty–Disagreement Correlation:** Assess the match between model uncertainty estimates and empirical annotator disagreement, particularly with multi-head models [2310.14979].
- **Human Trust and Explanation Usefulness:** In settings where explanations are solicited, human evaluation aims to quantify the informativeness and trust engendered by explanation-augmented models [2305.12710].

The use of consensus label estimators and annotator trustworthiness weights, e.g., via CROWDLAB or majority/cohen’s kappa, also plays a central role in multi-annotator and noisy label scenarios [2301.11856, 2211.06224].

## 4. Domains of Application and Empirical Findings

ACAL frameworks have demonstrated empirical advantages in diverse real-world settings:

- **Subjective NLP Tasks:** Substantial gains in data efficiency (up to 62% annotation budget reduction), broader representation of annotator perspectives, and improved modeling of minority views in hate speech, moral sentiment, and safety judgment datasets [2404.15720].
- **Text, Image, and Video Domains:** Integrations with pre-trained language or vision–language models, leveraging domain expertise, show both annotation efficiency and improvements in average precision and classification performance across datasets [2402.06560, 2112.11914, 2301.03252].
- **Low-Budget and Noisy Annotation Regimes:** Noise-aware sampling and transfer learning approaches (e.g., using SVMs over SSL features with resilience to annotation noise) provide strong advantages over traditional fine-tuning or random selection, particularly when only sparse annotations can be acquired [2201.07200, 2504.04506].
- **Multi-Annotator Systems:** Unified frameworks (e.g., ALANNO) support balanced assignment, continuous agreement monitoring, adaptive batch selection, and extensibility to a variety of AL strategies [2211.06224].

Some of the strongest improvements are observed in relabeling scenarios and in early-phase active learning, where correct early selection and repeated annotation significantly impact final generalization [2301.11856].

## 5. Challenges, Limitations, and Open Directions

The success of ACAL methods is subject to a series of practical and theoretical constraints:

- **Annotator Pool Heterogeneity:** The effectiveness of diversity-driven annotator selection is limited by the size and heterogeneity of the available annotator pool. With low diversity or few annotators per item, performance gains diminish [2404.15720].
- **Dependency on Annotation History:** Several strategies require sufficient history to compute diversity or trustworthiness—necessitating warm-up phases and careful design in early rounds [2404.15720].
- **Cognitive State Measurement:** Real-time tracking of mood and fatigue requires accurate, possibly intrusive measurement; simulations often replace real-world signals, which may miss subtle influences or be difficult to scale [2507.23756].
- **Reliance on Explanation Quality:** Methods that integrate annotator rationale or explanation-augmented sampling can be misled if explanations (either human or model-generated) are of low fidelity or inconsistent [2009.04568, 2305.12710].
- **Label Noise:** While noise-aware methods such as NAS improve robustness, all coverage- and typicality-based AL approaches remain sensitive to systematic noise, especially when data is scarce or the noise filtering itself is imperfect [2504.04506].

Future work is highlighted across several axes: modeling richer cognitive and behavioral annotator features, dynamically learning cost trade-offs (MC/AC), scaling to extreme annotator or instance counts, integrating better biometrics in annotator models, and developing more holistic annotator selection and evaluation strategies [2109.11301, 2507.23756]. Real-world validation with active, online human annotators remains an important direction for assessing live annotator-centric systems.

## 6. Broader Impact and Significance

ACAL frameworks align the machine learning model development lifecycle with the realities of human annotation, striving to minimize annotation cost while maximizing utility, validity, and fairness. By accounting for annotator reliability, diversity, rationale, and cognitive conditions, ACAL systems have demonstrated improvements in annotation efficiency, performance on subjective or ambiguous tasks, and robustness to noise and annotator disagreement.

Specific results include statistically significant reductions in queries for target performance (e.g., halving for some QBC variants [2009.04568]), improved model trust through explanation alignment, and more representative model behavior for both majority and minority annotator perspectives. These advances are particularly salient as datasets grow in complexity and as societal demands for reliable and fair annotation—especially in subjective domains—continue to increase.

The field continues to evolve toward frameworks that manage the full spectrum of annotator diversity, cost, and cognition, positioning ACAL as a cornerstone for scalable, human-aligned AI annotation systems across domains.

Source: https://www.emergentmind.com/topics/annotator-centric-active-learning-acal