REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations

Abstract: Supervised fine-tuning of LLMs relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise. Standard practice aggregates labels via majority vote or simple averaging, discarding annotator identity and causing the model to absorb the errors of unreliable annotators directly into its parameters. We propose REALM, a method that jointly learns the model parameters and a scalar expertise value for each annotator entirely unsupervised, requiring no supervision beyond annotator identity. The key idea is to model each observed label as a mixture between the model's prediction and a uniform random guess, weighted by the annotator's learned expertise. We extend REALM to a multi-task setting via a learned expertise matrix that captures per-annotator reliability across tasks. We evaluate on five question answering benchmarks, fine-tuning three sizes of Flan-T5 under simulated noisy annotations. The proposed algorithm consistently outperforms the naive noisy SFT in the large majority of single- and multi-task settings, across datasets, model sizes, and noise types, with accuracy improvements of up to $50\%$ in the most adversarial regime and gains that grow with model capacity.

• The paper introduces an unsupervised mixture model that learns per-annotator expertise to fine-tune LLMs using noisy supervision.
• The approach adapts the training objective with negative log-likelihood to down-weight unreliable annotators and mitigate label noise.
• Empirical results show up to 50% absolute gains in test accuracy, demonstrating robustness across multiple QA datasets and tasks.

REALM: Reliable Expertise-Aware Fine-Tuning from Noisy Annotations

Motivation and Problem Formulation

Supervised fine-tuning (SFT) is central to effective LLM deployment. In standard pipelines, labels come from multiple annotators spanning heterogeneous expertise levels. Existing frameworks typically aggregate annotations using majority vote or simple averaging, thus disregarding annotator identity—this practice leads models to blend unreliable labels into their parameters. The central question addressed is whether we can reliably fine-tune LLMs from noisy human supervision, automatically identifying and down-weighting unreliable annotators, using only annotator identity.

The REALM Framework

REALM (Reliable Expertise-Aware LLM) is formulated as a mixture model that simultaneously learns both model parameters and per-annotator expertise scores, entirely unsupervised. Each observed label is modeled as a convex combination of the model’s prediction and a uniform random guess, weighted by a learned annotator-specific expertise parameter $\beta_i$. When $\beta_i=1$, the annotator is a perfect expert; when $\beta_i=0$, the annotator provides only uniform noise.

The training objective replaces standard cross-entropy with a negative log-likelihood under the mixture model, and expertise parameters are trained jointly with the model using gradient-based optimization. The design employs a restricted softmax over candidate answers, keeping both mixture components defined over the true label space.

The mixture formulation generalizes standard SFT. When all annotators have $\beta_i \approx 1$, the loss reduces to cross-entropy; as unreliable annotators are identified, their impact on training is automatically minimized.

Extension to Multi-Task and State-Dependent Expertise

REALM extends naturally to the multi-task scenario, learning a per-annotator, per-task expertise matrix, capturing structured patterns of reliability. The model uses observed dataset origin as a task indicator. Furthermore, the framework admits generalization to state-dependent expertise, parameterizing $\beta_i$ as a function of both annotator and prompt representations. This has the potential to capture input-conditional reliability and is positioned as a future direction.

Empirical Evaluation and Results

REALM is benchmarked on five multiple-choice QA datasets (OBQA, ARC, PIQA, RiddleSense, PubMedQA) using three sizes of Flan-T5 (Small, Base, Large) under simulated noisy supervision. Experiments encompass:

• Varying expertise distributions (e.g., a single expert vs. two near-random annotators, majority-expert regime, and uniformly spread expertise).
• Diverse noise types: (i) uniform label noise, (ii) asymmetric (biased) noise, and (iii) systematic deterministic offsets (violating the generative assumption).
• Both single- and multi-task training.

Performance is compared to two baselines: Oracle SFT (trained on gold labels, not available in practice) and Noisy SFT (standard cross-entropy, identity-agnostic). All reported results average over five random seeds with detailed accuracy breakdowns.

Immediately upon deployment, the benefits of expertise-aware modeling are apparent. When unreliable annotators dominate (Dist.~\textcircled{1}), REALM yields substantial improvements on test accuracy—up to 50% absolute gain—against standard SFT, especially in high-noise, high-capacity (Large) model regimes.

Figure 1: Test accuracy (%) over training steps under Dist.~\textcircled{1} for all noise types and datasets, demonstrating that REALM prevents noisy signal absorption throughout training.

The empirical findings include:

• Single-Task, High-Adversity: Under Dist.~\textcircled{1} (one expert and two highly unreliable annotators), REALM stably outperforms Noisy SFT in all datasets and model sizes, with accuracy gains increasing with model capacity (e.g., up to 50.18% on PIQA/Large).
• Noise-Type Robustness: REALM maintains performance gains even when annotator noise deviates from the assumed mixture (asymmetric/systematic noise). Although gains diminish in the hardest, fully deterministic wrong-label regime, REALM consistently mitigates the impact of unreliable annotators.
• Model Scaling: Improvements widen for larger models, indicating higher capacity enables more effective utilization of reliability signals.
• Multi-Task: When annotator expertise is task-specific, the jointly learned expertise matrix allows REALM to recover per-task annotator reliability and outperforms Noisy SFT across all configurations and for all underlying noise structures.
• Ablations: Effective expertise learning requires a sufficiently large scalar learning rate. REALM’s robustness persists as the annotator pool grows, albeit with decreasing margins as discrimination between annotators becomes more challenging due to a denser spectrum of expertise.

Practical, Theoretical, and Future Implications

REALM advances unsupervised reliability estimation for supervised fine-tuning, requiring only annotator identity as input—removing reliance on gold references or meta-information about annotator quality. The method generalizes to federated learning or distributed annotation scenarios and holds utility in practical pipelines characterized by low-budget annotation, crowdsourced data, or multi-domain deployment. Theoretically, the mixture model connects to prior imitation learning literature, offering a statistically-grounded approach to expert identification with ties to robust parameter learning from heterogeneous data.

The multi-task and possible state-dependent extensions set the stage for capturing more nuanced reliability, including input-conditional trust, potentially inferred from annotations without explicit task or domain labels.

Future directions include:

• Generalizing prompt-dependent expertise via shared annotation embeddings.
• Investigating combinations with federated/distributed optimization for massive decentralized pipelines.
• Understanding the limitations of the mixture assumption in real-world, non-synthetic annotation pipelines.

Conclusion

REALM offers a rigorous, unsupervised method for expertise-aware LLM fine-tuning, consistently mitigating the risk of overfitting to unreliable annotators across diverse, challenging supervision conditions. The approach is robust, scalable, and demonstrates increasing gains as model capacity and annotation noise increase, suggesting broad applicability in realistic NLP annotation workflows and beyond.