SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation

Abstract: Recent advances in LLMs have substantially improved Natural Language Understanding (NLU). Although widely used benchmarks suggest that LLMs can effectively disambiguate, their practical applicability in real-world narrative contexts remains underexplored. SemEval-2026 Task 5 addresses this gap by introducing a task that predicts the human-perceived plausibility of a word sense within a short story. In this work, we propose an LLM-based framework for plausibility scoring of homonymous word senses in narrative texts using a structured reasoning mechanism. We examine the impact of fine-tuning low-parameter LLMs with diverse reasoning strategies, alongside dynamic few-shot prompting for large-parameter models, on accurate sense identification and plausibility estimation. Our results show that commercial large-parameter LLMs with dynamic few-shot prompting closely replicate human-like plausibility judgments. Furthermore, model ensembling slightly improves performance, better simulating the agreement patterns of five human annotators compared to single-model predictions

• The paper presents a framework combining fine-tuning, dynamic few-shot prompting, and ensemble methods to generate human-like plausibility scores in narrative contexts.
• The approach employs techniques such as reasoning-driven scoring, difficulty-aware classification, and FAISS-based retrieval to enhance disambiguation performance.
• Results show that commercial LLMs and meta-ensemble strategies significantly improve Spearman correlation and accuracy for narrative word sense disambiguation.

LLM-Based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation

Introduction and Problem Formulation

This work addresses the challenge of modelling fine-grained human plausibility judgments for word sense disambiguation (WSD) in narrative contexts, a task introduced by SemEval-2026 Task 5. Standard WSD benchmarks mainly operate at the sentence level, leveraging local word clues and often ignore extended narrative dependencies. AmbiStory, the benchmark dataset underlying this task, presents short stories with embedded lexical ambiguities and gathers 5-point plausibility ratings from human annotators for each candidate meaning in context. The requirement is to compute a plausibility score, simulating human agreement patterns, for each ambiguous story.

Methodological Contributions

The proposed framework evaluates and integrates three primary approaches: supervised fine-tuning (SFT) of low-parameter LLMs, dynamic few-shot prompting with commercial LLMs, and ensembling methods to approximate multi-annotator consensus.

Fine-tuning of Low-Parameter LLMs

The fine-tuning strategy implements four distinct paradigms, each incrementally incorporating reasoning processes observed in human annotators:

• Single-annotator simulation: Mimics a single human judgment for a target sense.
• Five-annotator simulation: Outputs vectorized judgments emulating inter-annotator variability.
• Reasoning-driven scoring: Explicitly structures the model’s reasoning pathway, introducing intermediate sense prediction (with GLOSSGPT as semantic oracle) prior to rating assignment.
• Difficulty-aware scoring: Classifies each sample as "human-easy" or "human-ambiguous" (see (Figure 1)), using this as an additional input for scoring, further aligning with empirical difficulty judgments detected in annotation statistics.

  Figure 1: The flow used to classify whether a case is easy or difficult for humans.

Fine-tuning is conducted using LoRA for parameter-efficient adaptation, chat-style prompt formatting, and explicit integration of the aforementioned contextual and meta-analytic cues.

Dynamic Few-Shot Learning for Commercial LLMs

The commercial LLM pipeline employs a Retrieval Augmented Generation (RAG) mechanism: stories are embedded, indexed via FAISS, and representative prior cases are retrieved as few-shot demonstrations. This is operationalized for models such as GPT-4o, DeepSeek V3, and Gemini 2.5. Prompts are structured to require systematic evidence-based analysis culminating in a final plausibility rating, adhering to explicit scoring rubrics. Hyperparameter sweeps on number of retrieved shots ($K=1,3$) are performed, trading off computational cost and marginal performance gain.

Ensemble Methods for Multi-Annotator Agreement Simulation

Recognizing the limitations of single-model simulation in ambiguous cases, a meta-ensemble is constructed: predictions from the five most performant base models are combined using majority voting, equal-weight or performance-weighted averaging, linear regression, SVR, and XGBoost-based meta-learners. The ensemble is trained on development data with $k$-fold cross-validation; the test set is reserved for final evaluation.

Results and Analysis

Supervised Fine-Tuning Performance

Difficulty-aware finetuning consistently yields the highest Spearman correlation ($0.491$ for Qwen 4B test) and accuracy ($0.631$), outperforming strategies that only simulate raw annotation distribution. Notably, five-annotator simulations underperform relative to single-annotator pathways, likely due to over-regularization towards annotator diversity that does not always align with gold aggregated ratings. Structured CoT-based reasoning and explicit difficulty cues both enhance accuracy, especially in "human-easy" cases.

Few-Shot Prompting with Commercial LLMs

Commercial LLMs, especially GPT-4o, outperform fine-tuned smaller LLMs by a wide margin (GPT-4o achieves Spearman $0.755$ and accuracy $0.798$ with $K=3$ shots on the test set). Improvements from few-shot retrieval are statistically significant over zero-shot; however, performance plateaus or mildly regresses for larger $K$, suggesting a trade-off between context informativeness and overfitting.

Ensemble Approach Evaluation

The best-performing ensemble (linear regression, XGBoost) achieves accuracy up to $0.797$ and Spearman $0.724$ on the test set, surpassing all single-model predictions. The ensemble’s improvement is most pronounced in ambiguous contexts, validating the efficacy of combining diverse model reasoning styles to better capture inter-annotator disagreement and aggregate plausibility.

Implications for AI and Future Work

These results underscore several key points for NLU and WSD research:

• Context-sensitive, narrative-driven sense disambiguation remains challenging for both small and large LLMs, particularly in ambiguous cases with diffuse human agreement.
• Explicit difficulty tagging, evidence-based reasoning chains, and multi-perspective simulation via ensemble methods all improve human-likeness and robustness of plausibility scoring.
• Commercial LLMs, when augmented with structurally relevant in-context examples retrieved via similarity search, nearly match (though do not fully replicate) mean human ratings, supporting their use for cognitively plausible annotation simulation.
• Despite these advances, persistent weaknesses remain in ambiguous, highly context-dependent cases, highlighting the necessity of further developments in uncertainty modeling, self-consistency, and discourse-level representation learning for WSD.

Conclusion

This study demonstrates a comprehensive LLM-based framework for plausibility scoring in narrative WSD, achieving strong alignment with human annotator judgments—most notably via dynamic few-shot prompting in commercial LLMs and meta-ensemble prediction. The explicit integration of reasoning strategies and difficulty-awareness advances the field toward reliable modeling of interpretive nuance in computational semantics. Future progress will hinge on deeper discourse modeling and finer-grained simulation of human disagreement patterns, with applicability extending to broader narrative understanding and explainable AI domains in NLP.