Kaggle ASAP Essay Scoring

• Kaggle ASAP Essay Scoring is a benchmark that uses machine learning and transformer architectures to evaluate essays based on human-aligned scoring.
• The system integrates sophisticated preprocessing and context augmentation techniques such as prompt metadata, EDU/AC segmentation, and essay length features.
• Empirical results show that context-augmented transformers achieve near-state-of-the-art QWK scores, demonstrating robust performance across varied essay sets.

Automated Essay Scoring (AES) in the Kaggle ASAP (Automated Student Assessment Prize) competition context refers to the task of training and evaluating machine learning models to assign human-aligned scores to student essays across diverse prompts, genres, and rubrics. ASAP AES drives research on both holistically and analytically scored writing and has become a benchmark for deep learning, transformer-based architectures, context-enriched models, and modern evaluation methodologies.

1. The Kaggle ASAP-AES Dataset and Preprocessing

The ASAP-AES corpus consists of 12,876 essays grouped into eight essay sets, each aligned to a distinct prompt and rubric. Each set exhibits its own score range, content genre, and topical focus. Essays are scored by at least two human raters, with a resolved score as ground truth. Preprocessing involves:

• Lower-casing and stripping non-UTF8 characters.
• Prepended special tokens ([PROMPT]) for models using prompt metadata.
• Sentence tokenization for discourse segmentation; further splitting into elementary discourse units (EDUs) or argument components (ACs) using trained BiLSTM-CRF models (EDU F1=0.947, AC F1=0.835).
• Prompt-specific Z-score normalization of gold-standard scores for stable loss scaling.
• Absence of public validation/test splits; evaluation is by 10-fold cross-validation (CV) over the training set.
• Data augmentation explored through back-translation, adjusted-score essays (per Jong et al., 2022), and—crucially—contextual enrichment rather than synthetic essay generation (Chakravarty, 17 Aug 2025).

Essay Set	Size	Raw Score Range	Max Token Length
1	1,783	2–12	1,075
2	1,800	1–6	1,321
3/4/5/6	~1,800	0–3	462–766
7	1,569	0–3	978
8	723	0–3	1,489

2. Transformer-Based AES Architectures and Contextual Enhancement

The canonical AES backbone is a transformer capable of encoding long sequences (e.g., Longformer: 24 layers, 16 heads, hidden size 1,024, >4k context). The scoring head comprises a hierarchical structure:

• Token representations are processed by a Bi-LSTM (input 1,024 → hidden 256), then by a feed-forward network (FFN: 256→128→64→1, ReLU, dropout 0.4).
• Bi-LSTM at the essay level (hidden 128) plus another 3-layer FFN yields a scalar essay score.

Key context augmentation strategies, orthogonal to architecture, that yield robust gains include:

• Prompt metadata: Joint attention to prompt and essay text improves rubric alignment.
• Essay length and structure: Z-normalized essay and component counts concatenated to the high-level embedding, exploiting length–score correlations (ρ ≈ 0.7).
• EDU and AC segmentation: Explicit markers help the model focus on discourse and argument units; AC spans yield higher performance increments than EDU spans, reflecting the predictive value of argument structure for AES.
• Relative context (Margin-Ranking loss): A margin-ranking loss encourages correct ordering between essays, supplementing mean-squared error (MSE) to better mimic human rater ordinal judgments.
• Integration: Textual ([PROMPT], [EDU], [AC]) and numeric contexts are combined into a multi-dimensional embedding space (Chakravarty, 17 Aug 2025).

Context Added	QWK (mean, 10-fold CV)
Longformer Baseline	0.784
+MR Loss	0.788
+Prompt	0.808
+EDU spans	0.791
+AC spans	0.799
+AC+Prompt+Features	0.821

3. Evaluation Metrics and Supervised Training Protocols

Quadratic Weighted Kappa (QWK) is the principal metric, quantifying ordinal agreement between model and gold scores, penalizing large discrepancies quadratically:

$$QWK = 1 - \frac{\sum_{i,j}w_{ij}O_{ij}}{\sum_{i,j}w_{ij}E_{ij}},$$

with $w_{ij} = \frac{(i - j)^2}{(K - 1)^2}$, $O_{ij}$ the joint histogram, $E_{ij}$ the chance-expected histogram, and $K$ the number of discrete score levels.

Training details for contextual transformers include RMSProp optimization (AES), Adam (span tasks), learning rates $3\times10^{-5}$ to $1\times10^{-3}$, batch sizes 128 (AES), up to 150 epochs with early stopping, dropout 0.4, L2-regularization, and loss weighting ($\alpha = 0.9$ for MSE, $\beta = 0.1$ for MR loss) (Chakravarty, 17 Aug 2025).

4. Empirical Performance and Comparison to State of the Art

The most effective context-augmented transformer model on ASAP AES achieves mean QWK 0.823 (entire set, cross-prompt), 0.870 when trained essay-set-wise, outperforming all prior transformer-based models:

• Tran-BERT-MS-ML-R: 0.791
• R²-BERT: 0.794
• Context-augmented Longformer: 0.821 (+3.3% over R²-BERT)

Absolute state-of-the-art is held by the DeLAES architecture (CNN+BiGRU), reaching 0.903 QWK essay-set-wise (Tashu et al., 2022), but the context-augmented transformer closes the gap to within 3.83% on average, and surpasses DeLAES on three out of eight sets (Chakravarty, 17 Aug 2025).

Contextual adaptation is architecture-agnostic and can enhance BERT, RoBERTa, and convolutional/RNN hybrids. Stacking context enrichment atop CNN-RNN models further boosts performance.

5. Analysis of Contextual Contributions and Adaptability

Prompt context grounds the model in rubric expectations, improving domain adaptation across prompts. Margin-ranking loss imparts ordinal discrimination comparable to human rater calibration. Structural cues via EDUs/ACs boost coherence assessment, and length features capture high-variance, high-signal attributes, especially for prompts with greater score spread.

Empirical ablations show that adding AC spans with prompts provides the largest gains (notably for analytic or argument-focused prompts), while numeric feature counts are decisive on high-variance prompts (e.g., a +3.71 QWK increase for Set 4) (Chakravarty, 17 Aug 2025).

The context-injection pipeline is portable and seamlessly integrates with transformers and hybrid models; it is strictly orthogonal to core architectural advances.

6. Practical Deployment and Future Directions

Context-enriched transformer-based AES narrows the performance gap to SOTA, offers robustness under cross-prompt and high-variance conditions, and supports direct interpretability by including topical and structural markers in the input sequence. This contextual methodology positions AES systems for further advances through:

• Modular adaptation to new or unseen prompts.
• Stacking with newer architectures (e.g., Mamba, CNN-RNN hybrids).
• Application to analytic, trait-level feedback, supplementing holistic scores.
• Straightforward extension to teacher-in-the-loop pipelines by outputting interpretable features.
• Orthogonality to data augmentation or adversarial robustness techniques, allowing flexible integration in larger educational assessment frameworks (Chakravarty, 17 Aug 2025).

This factual synthesis demonstrates that the empirically validated, context-augmented transformer methodology defines a current best practice for Kaggle ASAP Essay Scoring, offering a clear path for both immediate deployment and future research increments.