REALM: Fine-Tuning with Annotator Expertise

• REALM is a supervised fine-tuning method that models annotator identity as a learnable signal, distinguishing expert input from noise without needing gold labels.
• It employs a probabilistic mixture model where each observed label is a blend of the model’s prediction and random guessing, with annotator expertise learned unsupervised via backpropagation.
• The approach extends to multi-task settings using an expertise matrix, enabling per-task reliability estimation and resulting in significant accuracy gains on multiple-choice QA benchmarks.

REALM is a method for supervised fine-tuning of LLMs under noisy human annotation, designed for settings in which multiple crowdworkers of heterogeneous expertise label the same class of tasks. Instead of aggregating annotations by majority vote or simple averaging, REALM preserves annotator identity and jointly learns model parameters and annotator expertise entirely unsupervised, requiring no supervision beyond annotator identity. Its central assumption is that each observed label is generated as a mixture of the model’s predictive distribution and a uniform random guess, with the mixture weight given by a learned expertise parameter for the annotator; the method is further extended to a multi-task setting through a learned expertise matrix (Ghiasvand et al., 19 Apr 2026).

1. Problem setting and motivation

Supervised fine-tuning pipelines for LLMs commonly rely on human-annotated data, yet annotation is often performed by multiple annotators whose reliabilities differ substantially. Standard practice aggregates labels via majority vote or simple averaging, discarding annotator identity and thereby allowing the model to absorb the errors of unreliable annotators directly into its parameters. REALM addresses this failure mode by treating annotator identity as a learnable source of signal rather than a nuisance variable (Ghiasvand et al., 19 Apr 2026).

The formulation assumes a $K$-way classification task, such as multiple-choice question answering, where each training example is a triple $(x,Q,a,\hat y)$ consisting of a prompt $x$ (or $Q$), an annotator $a \in \{1,\dots,A\}$, and an observed noisy label $\hat y \in \{1,\dots,K\}$. The LLM defines a predictive distribution $p_\theta(y \mid x)$ over the $K$ answer tokens. REALM then introduces a scalar expertise value for each annotator and learns it jointly with $\theta$.

A common misconception is that annotator reliability estimation requires gold labels or an external calibration set. REALM directly challenges that premise: annotator identity alone is used, and no ground-truth expertise supervision is ever provided. This suggests that, in settings where annotation provenance is logged, some of the information ordinarily destroyed by aggregation can be recovered during fine-tuning itself.

2. Probabilistic mixture model

The core probabilistic assumption is that the annotator’s observed label is a mixture between the model’s own prediction and uniform guessing. Concretely,

$$p(\hat y = y \mid x,a) = e_a \cdot p_\theta(y \mid x) + (1-e_a)\cdot U(y),$$

where $(x,Q,a,\hat y)$0 is annotator $(x,Q,a,\hat y)$1’s expertise and $(x,Q,a,\hat y)$2 for $(x,Q,a,\hat y)$3 (Ghiasvand et al., 19 Apr 2026).

REALM parameterizes expertise as

$(x,Q,a,\hat y)$4

with a scalar $(x,Q,a,\hat y)$5 for each annotator. The joint negative log-likelihood over the dataset is

$(x,Q,a,\hat y)$6

For multiple-choice QA, the implementation uses a restricted softmax only over the $(x,Q,a,\hat y)$7 answer tokens so that the uniform component $(x,Q,a,\hat y)$8 is defined on the same support. This design is technically important: the random-guess channel and the model channel operate over identical answer spaces, so the learned expertise weight remains interpretable as a reliability coefficient rather than a scale parameter coupled to vocabulary size.

The limiting case clarifies the relation to conventional fine-tuning. As $(x,Q,a,\hat y)$9, one has $x$0, and REALM recovers standard cross-entropy. In that sense, the method strictly generalizes standard SFT: ordinary supervised fine-tuning appears as the special case in which every annotator is treated as fully reliable.

3. Unsupervised expertise learning and optimization

REALM learns both the LLM parameters $x$1 and the annotator expertise scalars $x$2 jointly by back-propagating through the mixture loss. The expertise estimate $x$3 is therefore an unsupervised quantity, but it remains directly interpretable as a measure of annotator reliability at convergence (Ghiasvand et al., 19 Apr 2026).

Optimization uses AdamW with two learning rates: a small learning rate for $x$4, exemplified by $x$5, and a larger learning rate for the low-dimensional expertise parameters, exemplified by values in $x$6. The stated rationale is that expertise should adapt quickly early in training before $x$7 has overfit the noise. The corresponding practical recommendation is explicit: use a higher learning rate for $x$8 than for $x$9.

This separation of timescales is central to how the method operates. If expertise learning is too slow, the model can fit noisy labels before reliable annotators are distinguished from unreliable ones. The ablations make this concrete: the expertise learning rate must be large, at least $Q$0, or REALM underfits annotator weights. A plausible implication is that the method’s benefit depends not only on the mixture model itself but also on early optimization dynamics, particularly in regimes where model capacity is sufficient to memorize annotation noise.

The approach also reframes noisy-label learning in LLM fine-tuning. Rather than constructing a separate annotator model or pre-filtering labels offline, REALM embeds annotator reliability directly in the supervised objective. This keeps the intervention entirely within the SFT stage and adds no labeling overhead.

4. Multi-task expertise modeling

REALM extends naturally to the case of multiple tasks or datasets. If there are $Q$1 distinct tasks and example $Q$2 belongs to task $Q$3, the scalar parameter $Q$4 is replaced by a row vector $Q$5, yielding an expertise matrix $Q$6. The expertise of annotator $Q$7 on task $Q$8 is then

$Q$9

The loss becomes

$a \in \{1,\dots,A\}$0

No extra regularizer is needed: learning $a \in \{1,\dots,A\}$1 jointly with $a \in \{1,\dots,A\}$2 is reported as sufficient to recover per-annotator, per-task reliabilities unsupervised (Ghiasvand et al., 19 Apr 2026).

This extension changes the statistical object being learned. In the single-task version, expertise is a fixed scalar per annotator; in the multi-task version, reliability becomes task-conditional. That is important in annotation settings where an annotator may be dependable on one dataset and unreliable on another. The matrix $a \in \{1,\dots,A\}$3 captures this heterogeneity without changing the basic mixture interpretation.

The method was evaluated in a multi-task configuration with $a \in \{1,\dots,A\}$4 annotators and $a \in \{1,\dots,A\}$5 tasks on ARC, OpenBookQA, and RiddleSense. In that setting, REALM learns a $a \in \{1,\dots,A\}$6 expertise matrix and outperforms noisy fine-tuning in all settings, with up to $a \in \{1,\dots,A\}$7 improvement for Flan-T5-Large under asymmetric noise. This suggests that annotator identity can remain informative even when expertise is not globally consistent across tasks.

5. Experimental configuration and quantitative results

The evaluation uses five multiple-choice QA benchmarks: OpenBookQA, ARC, PIQA, RiddleSense, and PubMedQA. OpenBookQA is 4-way commonsense QA, ARC is 4-way science QA, PIQA is 2-way physical reasoning, RiddleSense is 5-way commonsense riddles, and PubMedQA is 3-way biomedical QA. Models are Flan-T5-Small (80M), Flan-T5-Base (250M), and Flan-T5-Large (780M). Training uses AdamW, $a \in \{1,\dots,A\}$8, $a \in \{1,\dots,A\}$9, batch size $\hat y \in \{1,\dots,K\}$0 with gradient accumulation $\hat y \in \{1,\dots,K\}$1, $\hat y \in \{1,\dots,K\}$2 steps, and $\hat y \in \{1,\dots,K\}$3 seeds (Ghiasvand et al., 19 Apr 2026).

The noisy-annotation setup uses simulated annotators with $\hat y \in \{1,\dots,K\}$4 by default, with ablations up to $\hat y \in \{1,\dots,K\}$5. Annotator true expertise $\hat y \in \{1,\dots,K\}$6 is sampled from three distributions: $\hat y \in \{1,\dots,K\}$7 (“one good, two random”), $\hat y \in \{1,\dots,K\}$8 (“two good majority”), and $\hat y \in \{1,\dots,K\}$9 (“spread”). When an annotator errs, the corruption mechanism is one of three types: uniform noise, asymmetric noise, or systematic noise. Uniform chooses a wrong answer uniformly over the remaining $p_\theta(y \mid x)$0 classes; asymmetric places $p_\theta(y \mid x)$1 mass on the “next” class and the remaining $p_\theta(y \mid x)$2 uniformly over others; systematic uses deterministic offset errors $p_\theta(y \mid x)$3.

Under the worst regime, namely $p_\theta(y \mid x)$4 with uniform noise, REALM improves average accuracy across the five datasets by $p_\theta(y \mid x)$5 for Flan-T5-Small, $p_\theta(y \mid x)$6 for Flan-T5-Base, and $p_\theta(y \mid x)$7 for Flan-T5-Large relative to noisy SFT. The largest reported single gain is $p_\theta(y \mid x)$8 on PIQA for Flan-T5-Large. The method continues to outperform noisy SFT under asymmetric and even systematic noise, with gains remaining in the $p_\theta(y \mid x)$9 to $K$0 range depending on model size and dataset. Learning curves show that REALM converges stably above noisy fine-tuning, which often degrades as training proceeds, especially on PIQA (Ghiasvand et al., 19 Apr 2026).

Two empirical regularities are especially notable. First, gains steadily increase with model size from Small to Base to Large, indicating that larger models benefit more. Second, when varying the number of annotators up to $K$1, REALM still reliably wins by about $K$2 on average, although the advantage shrinks slightly as annotator reliabilities become harder to separate. This suggests that the method is robust to increased annotator multiplicity, but that identifiability becomes more difficult as the reliability spectrum becomes less distinct.

6. Interpretation, practical recommendations, and limitations

The main conceptual claim of REALM is that annotator identity alone suffices to recover reliability without gold labels: by treating each label as a mixture of “model-correct” versus “random,” the method down-weights noisy annotators and preserves signal from experts. The reported robustness under asymmetric and systematic noise is important because the model’s explicit corruption assumption is uniform guessing. The fact that performance remains improved when that assumption is violated indicates that the mixture model functions as a useful inductive bias rather than a literal noise-process description (Ghiasvand et al., 19 Apr 2026).

Several practical recommendations follow directly from the reported study. REALM is lightweight to integrate into existing pipelines because it works entirely within the SFT stage and adds no labeling overhead. The most important hyperparameter recommendation is to use a larger learning rate for expertise parameters, typically $K$3 to $K$4, than for model parameters. The scaling results further indicate that larger models benefit more, suggesting that capacity is needed both to model the task and to leverage the reliability signal.

The principal stated limitation is that REALM assumes a fixed expertise value per annotator, or per annotator-task pair in the multi-task version. In practice, expertise can vary by prompt. A natural future step proposed in the work is a state-dependent variant,

$K$5

This would allow annotator reliability to depend on input characteristics rather than remaining static across all prompts in a task.

A broader implication is methodological. REALM relocates noisy-annotation handling from external data curation into the fine-tuning objective itself. This suggests a general direction for LLM supervision in heterogeneous labeling environments: instead of collapsing the annotation process into a single target before learning begins, one can preserve provenance and let the model infer which parts of the supervision stream are trustworthy.