Improving Lexical Difficulty Prediction with Context-Aligned Contrastive Learning and Ridge Ensembling

Abstract: Lexical difficulty prediction is a fundamental problem in language learning and readability assessment, requiring models to estimate word difficulty across different first-language (L1) backgrounds. However, existing approaches rely on regression-only training with scalar supervision, which does not explicitly structure the representation space, limiting their ability to capture cross-lingual alignment and ordinal difficulty. To mitigate these issues, we propose Context-Aligned Contrastive Regression, which integrates Ridge regression ensemble with two complementary objectives, i.e., Cross-View Context and Ordinal Soft Contrastive Learning. Experiments on three L1 datasets show that (i) contrastive objectives improve cross-lingual representation alignment while preserving language-specific nuances, (ii) the learned representations capture the ordinal structure of lexical difficulty, and (iii) the ensemble effectively mitigates systematic biases of individual models, leading to more stable performance across difficulty levels.

• The paper introduces a multi-objective framework that integrates contrastive losses and ridge ensembling to improve lexical difficulty prediction.
• It leverages cross-view and ordinal soft contrastive losses to align linguistic representations and capture ordinal relationships across languages.
• Empirical results on Spanish, German, and Chinese datasets demonstrate robust accuracy improvements and error corrections compared to individual models.

Context-Aligned Contrastive Learning and Ridge Ensembling for Lexical Difficulty Prediction

Problem Overview and Motivation

The task of lexical difficulty prediction is directly relevant to personalized learning and assessment, requiring the estimation of word difficulty for English as a foreign language (EFL) learners with diverse L1 (first language) backgrounds. Traditional regression-based approaches do not explicitly shape the representation space, thus failing to encode either the cross-lingual alignment or the ordinal structure critical for this task. The problem is compounded by L1 interference effects: lexical difficulty varies depending on the linguistic background of the learner, making multilingual, context-sensitive modeling crucial. Furthermore, simple scalar supervision does not ensure that words with similar difficulty are embedded closely, nor does it guarantee stable predictions across multiple languages and levels of proficiency.

Methodology

Context-Aligned Contrastive Regression

The core contribution is a multi-objective learning framework incorporating both representation-level regularization and prediction. The model jointly optimizes:

• Direct Regression Loss: Supervises scalar prediction accuracy using MSE.
• Cross-View Context Contrastive Loss: Aligns the representation of target words under different contextual viewpoints (translated context, L1-aware context), by pairing L1-based and English-based views. The objective encourages context-invariant representations, mitigating context-induced representation drift.
• Ordinal Soft Contrastive Loss: Models the graded, ordinal structure of lexical difficulty by weighting pairwise similarities in the embedding space according to target score proximity. This contrasts with classic supervised contrastive objectives by leveraging continuous rather than discrete targets.

The input is encoded as a concatenation of L1 source word, source context, English clue, and English target word, separated by special tokens. The context-aligned view augments this through MT-based translation of context spans, allowing for cross-lingual supervision.

Ensemble Model with Ridge Regression

Three strong multilingual encoder backbones—XLM-RoBERTa, multilingual DeBERTaV3, and mmBERT—serve as base predictors. Their independent predictions are stacked and ensembled with a Ridge regression meta-model, which adaptively learns the optimal weighting of each base model under regularization constraints. This design leverages the complementary prediction patterns identified in the base encoders, mitigating systematic biases such as mean-reversion, overestimation, or underestimation.

Empirical Results

Quantitative Performance

Evaluations are conducted on a multilingual L1-aware dataset comprising Spanish, German, and Mandarin Chinese learner groups, using GLMM-derived (generalized linear mixed model) difficulty scores as targets. The proposed ensemble approach achieves:

• Spanish: Pearson's $r = 0.832$, RMSE $\approx 1.05$
• German: Pearson's $r = 0.832$, RMSE $\approx 0.995$
• Chinese: Pearson's $r = 0.853$, RMSE $\approx 0.88$

The model maintains stable performance across difficulty bins, outperforming individual encoder architectures by narrowing errors both at the high and low ends of the difficulty spectrum. Notably, ensembling counterbalances the systematic errors: for example, XLM-RoBERTa compresses toward the mean, mDeBERTa-v3 tends to overestimate difficult cases, while mmBERT is more balanced. The Ridge ensemble consistently corrects the pronounced under- or overestimation errors observed in these base models.

On the BEA 2026 shared task, the proposed system achieves top-15 rankings in all L1 tracks, reinforcing its robustness and generality across typologically distinct L1s.

Representation Analysis

Contrastive objectives substantially improve structural properties of the representation space:

• Cross-Lingual Alignment: Aligned L1-English pairs exhibit higher cosine similarity than random pairs; however, perfect alignment is intentionally avoided, preserving informative L1-specific nuances.
• Ordinal Structure: There is a monotonic relationship between embedding distances and absolute differences in difficulty scores (Spearman $r = 0.22, 0.20, 0.39$ for ES, DE, CN respectively). This demonstrates that the embedding space effectively captures the ordinal nature of the prediction target.

Input-level analysis highlights that context length, lexical diversity, and orthographic features influence prediction error. Richer context and higher lexical diversity are associated with lower error, particularly in Spanish. This is less pronounced in German and Chinese, indicating a non-trivial interaction between context utility and the base language.

Error Analysis

Case studies indicate that ensemble modeling corrects large prediction errors for challenging items (e.g., underestimation of "cruise," "run," "character" in Spanish), but some semantically ambiguous or highly context-dependent lexical items remain problematic. Such failure cases highlight the need for capturing subtler semantic and pragmatic cues, which are not fully resolved even with advanced ensembling and contrastive objectives.

Implications and Future Directions

Theoretical Implications

The results validate the hypothesis that scalar regression loss alone induces an information-poor and brittle latent space, especially in the presence of L1 variation. Augmenting with contrastive regularization (cross-view, ordinal soft) yields representations with increased cross-lingual alignment and better encoding of ordinal semantics, which is critical for psychometrically principled lexical difficulty modeling. The consistent improvements across architectures and languages reinforce the universality of this approach.

Practical Implications

This framework directly benefits readability assessment, adaptive curriculum design, and personalized text simplification in multilingual educational environments. By explicitly modeling context and L1 effects, practitioners can select or generate texts that are calibrated to both learner background and evolving proficiency, mitigating the risk of over-generalized difficulty assignment. The Ridge ensemble also offers a robust, tractable means to integrate diverse encoders, potentially improving deployment in real-world educational technology systems.

Limitations and Future Work

• Limited L1 Coverage: Current evaluations are restricted to three major L1s; further extension is needed to typologically diverse and low-resource backgrounds.
• Increased Computational Overhead: Ridge ensembling, while improving robustness, incurs higher training and inference costs. Investigation into parameter-efficient or sparsity-inducing meta-ensembling is warranted.
• Objective Balancing: Contrastive loss weighting remains a hyperparameter-sensitive aspect, requiring strategies for automated or curriculum-style objective balancing.
• Semantic/Pragmatic Failures: Future research should explore improvements in capturing deep semantic ambiguity and context-dependent meaning, possibly via large-context or multimodal models.

Conclusion

The integration of context-aligned contrastive learning with Ridge ensembling demonstrably advances L1-aware lexical difficulty prediction, achieving high predictive accuracy, structural representation fidelity, and robust generalization across L1s. The approach provides a strong foundation for further exploration of personalized language modeling architectures in education and related domains.

---

Reference:

"Improving Lexical Difficulty Prediction with Context-Aligned Contrastive Learning and Ridge Ensembling" (2605.08950)