Predicting CEFRL levels in learner English on the basis of metrics and full texts

Abstract: This paper analyses the contribution of language metrics and, potentially, of linguistic structures, to classify French learners of English according to levels of the Common European Framework of Reference for Languages (CEFRL). The purpose is to build a model for the prediction of learner levels as a function of language complexity features. We used the EFCAMDAT corpus, a database of one million written assignments by learners. After applying language complexity metrics on the texts, we built a representation matching the language metrics of the texts to their assigned CEFRL levels. Lexical and syntactic metrics were computed with LCA, LSA, and koRpus. Several supervised learning models were built by using Gradient Boosted Trees and Keras Neural Network methods and by contrasting pairs of CEFRL levels. Results show that it is possible to implement pairwise distinctions, especially for levels ranging from A1 to B1 (A1=>A2: 0.916 AUC and A2=>B1: 0.904 AUC). Model explanation reveals significant linguistic features for the predictiveness in the corpus. Word tokens and word types appear to play a significant role in determining levels. This shows that levels are highly dependent on specific semantic profiles.

• The paper demonstrates that machine learning models, particularly Gradient Boosted Trees and neural networks, effectively classify CEFRL proficiency levels in learner English.
• It employs a comprehensive set of linguistic features, including lexical diversity and syntactic complexity, derived from the extensive EFCAMDAT corpus.
• The study highlights challenges such as topic-based overfitting and data sparsity, suggesting future improvements like refined POS tagging and L1-specific feature analysis.

Predicting CEFRL Levels in Learner English

Introduction

The study "Predicting CEFRL levels in learner English on the basis of metrics and full texts" (1806.11099) endeavors to establish a predictive framework for assessing the English proficiency of French learners using the CEFRL schema. The paper primarily focuses on constructing models that utilize linguistic complexity metrics to predict language proficiency levels. By leveraging the extensive EFCAMDAT corpus, the research emphasizes building models that can make pairwise distinctions between CEFRL levels, especially from A1 to B1, using advanced machine learning techniques.

Methodology

The dataset comprises extensive text corpora drawn from the EFCAMDAT database, involving raw text from French learners classified according to CEFRL levels. Various linguistic metrics were computed, including lexical diversity and syntactic complexity, using tools like LCA, LSA, and koRpus. The models employed include Gradient Boosted Trees (GBT) and Keras Neural Networks, with a particular focus on pairwise classification between adjacent CEFRL levels. Emphasis was placed on considering task-specific prompts to avoid skewed predictive features linked to specific essay topics.

Data Processing

The corpus used encompassed 41,626 texts from 7,695 learners, with metrics computed on syntactic and lexical dimensions. Both traditional metrics, such as TTR and lexically derived indices, as well as second-generation metrics designed to account for text length dependency, were utilized to form a comprehensive feature set. Additionally, syntactic measures derived from the L2 Syntactic Complexity Analyzer were employed to strengthen the feature representation.

Learner Classification Models

Predictive Task and Model Evaluation

The training data's skew towards beginner levels necessitated the use of pairwise models for classification, which facilitated distinguishing subtle differences in proficiency. Gradient Boosted Trees, Elastic Net regression, LSTMs, and combined metrics models were explored, with AUC metrics serving as a primary evaluation measure. GBT models based on readability metrics and custom features demonstrated robust predictive performance, with AUC values evidencing effective generalization for lower-level distinctions (e.g., A1 to A2 at 0.916).

Challenges and Approaches

The classification challenges included managing topic-based overfitting and data sparsity at advanced levels. Strategies employed included segregating datasets by prompts to mitigate topic effects and incorporating advanced linguistic features. The models revealed that word tokens and types played a crucial role in defining proficiency features.

Discussion

The study highlights key insights into leveraging linguistic metrics for proficiency classification, noting potential overfitting with task-based corpora and data distribution issues favoring lower-level samples. Employing a broad range of metrics, including syntactic and lexical complexity measures, was pivotal in achieving high classification accuracy. Moreover, potential improvements were identified through more complex POS tagging and L1-specific learner feature analysis.

Conclusion

The research successfully demonstrates the use of machine learning models to predict language proficiency levels using linguistic metrics. It underscores the feasibility of employing complexity measures, providing a foundational approach for automating language proficiency assessments. The exploration opens avenues for future work, such as integrating advanced POS patterns and L1-specific errors, to further enhance predictive accuracy and generalization across diverse learner populations.