What Makes Words Hard? Sakura at BEA 2026 Shared Task on Vocabulary Difficulty Prediction

Abstract: We describe two types of models for vocabulary difficulty prediction: a high-accuracy black-box model, which achieved the top shared task result in the open track, and an explainable model, which outperforms a fine-tuned encoder baseline. As the black-box model, we fine-tuned an LLM using a soft-target loss function for effective application to the rating task, achieving r > 0.91. The explainable model provides insights into what impacts the difficulty of each item while maintaining a strong correlation (r > 0.77). We further analyze the results, demonstrating that the difficulty of items in the British Council's Knowledge-based Vocabulary Lists (KVL) is often affected by spelling difficulty or the construction of the test items, in addition to the genuine production difficulty of the words. We make our code available online at https://github.com/adno/vocabulary-difficulty .

• The paper demonstrates a soft-target fine-tuning approach for LLMs/MLMs that significantly improves regression accuracy (RMSE ≈ 0.7, r > 0.91) on a multilingual vocabulary difficulty task.
• It introduces an explainable XGBoost model using SHAP to leverage corpus-based and LLM-derived features, highlighting key determinants such as production frequency and spelling difficulty.
• The study’s methodologies enable transparent, production-oriented vocabulary assessments and set a blueprint for applying continuous label regression in future NLP tasks.

Vocabulary Difficulty Prediction via Soft-Target Fine-Tuning and Explainable Modeling

Task Definition and Dataset Characteristics

The primary objective addressed in "What Makes Words Hard? Sakura at BEA 2026 Shared Task on Vocabulary Difficulty Prediction" (2605.14257) is the development of models for predicting the difficulty of English vocabulary test items for L2 learners, conditional on the learners' L1 (Chinese, German, or Spanish). The shared task utilizes the KVL dataset, which contains difficulty responses gathered as log-odds from generalized linear mixed models (GLMM), providing a robust, empirically anchored target for regression. The input combines an L1 equivalent, native context, and an English word clue, pushing the task away from standard LCP/CWI setups towards a production-oriented, bilingual format.

Methodological Contributions

Soft-Target Cross-Entropy Fine-Tuning for LLMs and MLMs

The paper introduces a fine-tuning strategy for both decoder LLMs and encoder MLMs that bypasses the discretization artifacts present in prior work. The method operationalizes cross-entropy loss with "soft targets": given a continuous difficulty target $y$ on a scale $S = \{1, 2, 3, 4, 5\}$, $y$ is represented as a convex combination of its adjacent discrete values, with target probabilities proportional to the proximity to $y$. During fine-tuning, these probabilities are used rather than hard labels, and at inference, the expected value under the model's output for $S$ is computed, yielding a continuous prediction.

This approach generalizes across both LLMs and MLMs, is adaptable to token-based prediction formats, and supports easy ensembling or feature fusion via linear stacking. Results demonstrate a robust increase in regression accuracy (mean RMSE ≈ 0.7, $r > 0.91$), establishing a new state-of-the-art on the open track.

Explainable Feature-Based Model with SHAP Attribution

A secondary modeling paradigm emphasizes interpretability. Here, a feature-driven XGBoost regressor is built, leveraging both corpus-derived (production and reception frequency, word length, CEFR level) and LLM-prompted features (spelling difficulty, lexical ambiguity, L1-calque status, L1-similarity). For LLM-driven features, prompt design employs temperature scaling and probability-weighted means (G-SCALE), further aligning class probabilities with ground-truth distributions.

Model explanations are rendered using SHAP, exposing localized and global attributions of each input factor. This enables rigorous post-hoc causal analysis of determinants of vocabulary production difficulty, contrasting the opaque behavior of large-scale LLM or MLM fine-tuning. The explainable model, when including LLM-derived attributes, outperforms encoder-based baselines (mean RMSE 1.067 vs. 1.218).

Empirical Results

Across multilingual open and closed tracks, ensembles of fine-tuned LLMs (GLM-4-32B, Qwen2.5-32B, Ministral-3-14B) achieve RMSEs in the 0.70–0.75 range, approaching the "statistical optimum" imposed by test set confidence limits. Notably, ablation studies confirm the superiority of the soft-target approach over standard regression heads and hard-target classification in both LLM and MLM settings.

The explainable XGBoost model, when augmented with LLM-based features, closes much of the gap to large neural models, lending credence to the sufficiency of judiciously chosen features (especially those harnessed from LLM probabilistic outputs) for production-centric vocabulary difficulty prediction.

Analysis: Determinants of Item Difficulty

SHAP analyses (global and local) reveal several high-impact predictors:

• Production Frequency: Learner-written corpus frequencies dominate across L1s, indicating productive familiarity as a critical determinant for accurate written production under clued recall.
• Spelling Difficulty: LLM-derived spelling scores are especially influential for alphabetic-L1 populations (Spanish, German), less so for Chinese, possibly due to a lack of native orthographic interference from ideographic scripts.
• CEFR Level and Lexical Ambiguity: These provide significant complementary information, particularly in distinguishing items with multiple plausible senses or non-cognate translations.
• Test Item Design Effects: The authors introduce the "trickiness" construct—item-specific confoundedness measured by LLM answer probability—even for high-frequency, low-ambiguity lexis. This exposes confounds in dataset item construction that may surface in authentic assessments.

The inclusion of less relevant features (L1-calque, word length, certain frequency measures) is justified but shows lower SHAP attributions, aligning with theoretical predictions about their secondary role in L2 productive vocabulary access.

Theoretical and Practical Implications

This work demonstrates that fine-grained regression on language production difficulty can be robustly operationalized using foundation model architectures when the loss is well-calibrated to the continuous, empirically observed outcome distribution. The extension of soft-target losses to token-level regression in both MLM and decoder LLM paradigms circumvents precision loss due to discretization, with direct impact for any future work requiring continuous label regression in NLG problems.

From an explainability perspective, integrating LLMs as feature generators in a classic interpretable regressor introduces a scalable pathway for transparent educational NLP modeling: high performance with actionable, linguistic rationale for predictions.

Future Research Directions

Key directions include:

• Transfer and Generalization: Validation on alternate, non-KVL datasets or tasks, especially those with different item formats, scale granularity, or more ambiguous L1/L2 relationships.
• Granular User-Level Inference: Extending the model to account for user-specific histories, error types (e.g., specific spelling errors), or metacognitive states.
• Test Item Generation: Leveraging explainable attributions to design less "tricky," more diagnostic test items, minimizing confounding factors in L2 vocabulary assessment.
• Domain Adaptive Regression: Investigation of the soft-target technique for other NLP regression tasks, such as document readability prediction, essay scoring, or continuous sentiment rating.

Conclusion

The paper provides a comprehensive, empirically validated framework for vocabulary difficulty prediction integrating advanced neural fine-tuning strategies and interpretable modeling. The effectiveness of soft-target cross-entropy for LLM/MLM regression, and the extraction of linguistically plausible feature attributions via SHAP, establishes a template for future NLP-for-education research—balancing predictive performance with domain insight. The findings further recommend ongoing refinement of assessment datasets by diagnosing item construction confounds surfaced through neural and feature-based explainability frameworks.