Kaggle ASAP 2.0 Dataset Overview

• The Kaggle ASAP 2.0 dataset is a benchmark for Automated Essay Scoring (AES) that emphasizes long-context essays to address current model limitations.
• It comprises grade-stratified persuasive essays from grades 6–10 and challenges models with variable essay lengths and a realistic train/test split.
• Modern models, including Longformer and state-space approaches, are evaluated using holistic score prediction and Quadratic Weighted Kappa as a key metric.

Searching arXiv for the specified paper and closely related ASAP 2.0 work. The Kaggle ASAP 2.0 dataset is a benchmark for Automated Essay Scoring (AES) centered on long-context student writing. In the literature, it is presented as the Automated Student Assessment Prize v2 (ASAP 2.0) dataset and used as a main benchmark for evaluating whether contemporary language-model architectures can score essays without relying on severe truncation, a practice that raises validity concerns for rubric dimensions such as organization and global coherence (Ormerod et al., 12 Sep 2025). ASAP 2.0 is described as a “much-needed update” of the original ASAP dataset, which is treated as saturated as a benchmark, and as an extension of the PERSUADE corpus designed to support automated scoring in addition to argumentation and discourse annotation. The resulting benchmark is therefore inherently persuasive/argumentative in orientation, with essays written by students in grades 6–10 (Ormerod et al., 12 Sep 2025).

1. Origin, scope, and benchmark role

ASAP 2.0 is positioned as a successor benchmark for AES rather than as a minor revision of the original ASAP corpus. In the reported characterization, the original ASAP benchmark is regarded as no longer sufficiently demanding, particularly for modern LLMs, whereas ASAP 2.0 introduces essay lengths and discourse demands that materially stress model context handling (Ormerod et al., 12 Sep 2025).

The dataset is derived as an extension of the PERSUADE corpus (Persuasive Essays for Rating, Selecting, and Understanding Argumentative and Discourse Elements). In the study using it for long-context AES, ASAP 2.0 is explicitly framed not merely as a corpus for discourse or argument mining, but as one designed to support automated scoring. This linkage to PERSUADE implies that the essays are persuasive/argumentative essays, although the article does not enumerate separate rhetorical domains or prompt families (Ormerod et al., 12 Sep 2025).

The benchmark is organized around grade-level stratification rather than prompt-level exposition. Essays are reported for students in grades 6–10, but the paper does not provide the number of prompts or the prompt texts themselves. It states only that essays are written on a wide range of prompts. A plausible implication is that ASAP 2.0 is intended to assess scoring robustness across heterogeneous persuasive writing tasks without foregrounding prompt-specific modeling in the benchmark description.

2. Corpus composition and length characteristics

The reported dataset statistics emphasize both split size and essay length, because the latter is central to the long-context problem formulation. The paper gives the following word-length statistics by grade for the provided train and test partitions (Ormerod et al., 12 Sep 2025).

Grade	Train Count / Avg. Train Words	Test Count / Avg. Test Words
6	2094 / 292.2	527 / 268.3
8	1648 / 339.9	921 / 295.9
9	4002 / 426.1	0 / –
10	9563 / 385.8	5973 / 356.4
Total	17307 / 376.1	7421 / 342.7

These counts establish several properties of the benchmark. First, the dataset includes 17,307 training essays and 7,421 test essays. Second, essays are comparatively long for AES, with average word counts ranging from approximately 268 to 426 depending on grade and split, and a global training average of 376.1 words. Third, grade 9 has no test set in the table reported by the paper, which has implications for how evaluation results are aggregated and interpreted (Ormerod et al., 12 Sep 2025).

The study does not report unified token counts because tokenization varies across models. Instead, it uses word counts as the common descriptive statistic. This matters because the benchmark’s long-context status is argued through the mismatch between essay lengths in words and the subword-token limits of widely used transformer encoders. The paper explicitly states that “many responses in the ASAP 2.0 dataset are too long for most LLMs” (Ormerod et al., 12 Sep 2025).

No token-length histograms, percentile summaries, or counts of essays beyond thresholds such as 512 or 1024 tokens are provided. Consequently, fine-grained token-level burden must be inferred from the reported word counts and from the known variability of model-specific subword tokenizers. This suggests that the benchmark’s difficulty is not merely that essays are “long” in an informal sense, but that their lengths interact nontrivially with architecture-specific context budgets.

3. Scoring formulation and psychometric framing

In the reported use of ASAP 2.0, the task is holistic score prediction: each essay is assigned a single score rather than a vector of trait-level scores. The paper treats the problem as discrete score classification, not regression and not multi-trait assessment. Classification heads are configured to have “as many targets as there are scores”, which means each permissible score value is modeled as a separate class (Ormerod et al., 12 Sep 2025).

The underlying ASAP 2.0 rubric is not described in full detail in the paper. No explicit score range is given, and no score normalization or transformation procedure such as z-scoring, min–max scaling, or other rescaling is reported. Scores are used directly in discrete form. Predictions are likewise discrete: either the argmax of a classifier head or a parsed generated score in the generative setting (Ormerod et al., 12 Sep 2025).

The study reports human–human Quadratic Weighted Kappa (QWK) on ASAP 2.0 as:

$\text{Human rater QWK} = 0.745$

This value is used as a reliability reference point rather than as proof of a definitive ceiling. The paper explicitly cautions that model performance exceeding human–human QWK does not necessarily mean that the model is more valid than human scoring. That caution situates ASAP 2.0 within a psychometric rather than merely leaderboard-oriented framing: agreement statistics are informative, but they are not identical to construct validity (Ormerod et al., 12 Sep 2025).

A common misconception is that exceeding human inter-rater agreement automatically establishes superior assessment quality. The study rejects that inference. A plausible implication is that ASAP 2.0 should be interpreted as a benchmark for scoring reliability under realistic long-context conditions, not as a complete substitute for broader validity studies.

4. Preprocessing, partitioning, and experimental protocol

Preprocessing is described only at a high level. Models use their native subword tokenizers, including tokenizer schemes such as SentencePiece for many models, but the paper does not describe lowercasing, punctuation stripping, spelling normalization, essay removal, or malformed-text handling. Nor does it mention sub-sentence segmentation or any explicit decomposition of essays into smaller units before scoring (Ormerod et al., 12 Sep 2025).

The benchmark protocol follows the provided train/test split from ASAP 2.0. Within the training partition, 10% of the training set was designated as a development set. Thus the optimization split corresponds to 90% of 17,307 training essays, the development split to 10% of 17,307, and the held-out test set to 7,421 essays (Ormerod et al., 12 Sep 2025).

The paper does not characterize the splits as prompt-wise, cross-prompt, or domain-shift partitions. It also does not discuss prompt bias mitigation, domain adaptation, or domain generalization. Reported analyses are primarily grade-level, with results shown for grades 6, 8, and 10, and overall performance aggregated across the test set (Ormerod et al., 12 Sep 2025).

This protocol has two important consequences. First, the benchmark is treated as a realistic operational scoring setting using the dataset’s native partition rather than a deliberately adversarial cross-prompt generalization setup. Second, because prompt-specific evaluation is absent, claims supported by ASAP 2.0 in this study concern long-context scoring performance on the provided split, not prompt transfer.

5. Long-context problem formulation

The paper’s central claim is that ASAP 2.0 is useful because it exposes the limitations of conventional transformer encoders with short context windows. Standard models such as BERT and DeBERTa are described as typically supporting 512 tokens of maximum sequence length. Higher-grade essays in ASAP 2.0 often exceed those limits, making truncation a common practice in AES workflows (Ormerod et al., 12 Sep 2025).

The study frames truncation as a validity concern rather than merely an engineering inconvenience. Its explicit argument is that if essays are truncated at 512 tokens, it becomes difficult to maintain that conventional LLMs are faithfully modeling rubric dimensions such as organization, which depend on access to long-range discourse structure (Ormerod et al., 12 Sep 2025).

The paper formalizes context length through model-specific caps $L$:

• DeBERTa-Base: $L = 512$
• Longformer: $L = 4\text{k}$
• ModernBERT: $L = 8\text{k}$
• XLNet-Base: $L = 8\text{k}^*$
• Mamba-130m: $L = 8\text{k}^*$
• Llama-3.2-8B: $L = 8\text{k}$

For XLNet and Mamba, the asterisk denotes a training-imposed cap rather than an intrinsic architectural limit (Ormerod et al., 12 Sep 2025).

The XLNet discussion introduces a more formal long-context perspective via effective receptive field:

the output of any token is only a function of at most $L D$ of the previous tokens where $D$ is the depth of the network.

Using the reported base-model values, $L$0 and $L$1 yield approximately 6,000 tokens of effective dependence; for a large model with $L$2, the figure is approximately 12,000 tokens (Ormerod et al., 12 Sep 2025). This is used to motivate the claim that some architectures can, in principle, model essays whose full informational dependencies exceed the nominal local segment length.

A plausible implication is that ASAP 2.0 functions as a stress test not just for raw sequence length, but for the ability to preserve rubric-relevant discourse information over long spans.

6. Modeling strategies applied to ASAP 2.0

The study evaluates six fine-tuned model families on ASAP 2.0: DeBERTa-Base, XLNet-Base, Longformer, ModernBERT, Mamba-130m, and Llama-3.2-8B (Ormerod et al., 12 Sep 2025). Their inclusion makes ASAP 2.0 a comparative benchmark across standard transformers, sparse-attention transformers, recurrent-segment transformers, state-space models, and generative decoder-only models.

For DeBERTa-Base, ASAP 2.0 serves as a benchmark for common AES practice under a 512-token limit. The model is described as a strong baseline, but essays longer than 512 tokens must be truncated. The truncation policy itself is not specified (Ormerod et al., 12 Sep 2025).

For Longformer, the context length is 4k tokens and the architecture uses sliding-window local attention with sparse global attention to selected tokens. In the study’s framing, this allows full coverage of most essays in ASAP 2.0 and reduces or eliminates truncation relative to 512-token models (Ormerod et al., 12 Sep 2025).

For XLNet, the model inherits recurrent attention from Transformer-XL, using segment-level recurrence:

$L$3

where $L$4 is stop-gradient, $L$5 is concatenation, and $L$6 is multihead attention (Ormerod et al., 12 Sep 2025). This architecture is used to support cross-segment dependency propagation on long essays.

For ModernBERT, the context length is 8k tokens. The paper attributes its long-context capability to Rotary Position Embeddings (RoPE) and a two-phase pretraining strategy that integrates short- and long-range positional information (Ormerod et al., 12 Sep 2025).

For Mamba-130m, the architecture is a state-space model (SSM) rather than an attention-based transformer. The paper gives the continuous-time formulation

$L$7

with discretization

$L$8

and notes that its computational complexity scales linearly with sequence length (Ormerod et al., 12 Sep 2025). The model is therefore presented as especially suitable for long ASAP 2.0 essays.

For Llama-3.2-8B, the context length is 8k tokens. Unlike the encoder-style classifiers, it is fine-tuned in an instruction-following format. The input explicitly includes a rubric and essay, and the model is trained to generate the score:

$L = 4\text{k}$3

with output of the form:

$L = 4\text{k}$4

The rubric is paraphrased in 20 different ways for robustness, and the best-performing rubric variant on the development set is chosen by QWK (Ormerod et al., 12 Sep 2025).

The paper does not describe explicit essay chunking for any model. Long-context architectures are instead relied upon to process whole essays where possible.

7. Optimization, evaluation, and empirical findings

For the discriminative models—DeBERTa, XLNet, Longformer, ModernBERT, and Mamba—the original token-prediction head is removed and replaced with a classification head having as many targets as there are scores. Training uses cross-entropy loss over discrete score classes. The paper presents the standard form:

$L$9

For most classification models, optimization uses Adam with decoupled weight decay, an initial learning rate of $L = 512$0, a linear scheduler decaying the learning rate to zero over 10 epochs, batch size 4 or 1 depending on essay length, and early stopping based on development-set QWK (Ormerod et al., 12 Sep 2025).

Mamba departs from this pattern in one critical respect. To prevent model collapse, the study reports that it was necessary to effectively freeze the weights associated with the SSM, $L = 512$1, and the convolutional layer. Fine-tuning is then performed on the embedding layer and the $L = 512$2 and $L = 512$3 weights of every layer, using Adam, learning rate $L = 512$4, and batch size 8 (Ormerod et al., 12 Sep 2025).

For Llama-3.2-8B, fine-tuning uses LoRA and QLoRA, with LoRA applied to the attention matrices $L = 512$5, $L = 512$6, and $L = 512$7. The objective is standard next-token prediction loss, with the score generated as text (Ormerod et al., 12 Sep 2025).

Evaluation is based primarily on Quadratic Weighted Kappa (QWK), defined as

$L = 512$8

with

$L = 512$9

where $L = 4\text{k}$0 is observed agreement, $L = 4\text{k}$1 is expected agreement, and $L = 4\text{k}$2 is the number of score categories (Ormerod et al., 12 Sep 2025). Although the paper mentions “the standard metrics specified for AES”, only QWK is actually reported in the presented tables.

The reported test-set results are as follows (Ormerod et al., 12 Sep 2025):

Model	Overall QWK	Grade 6 / Grade 8 / Grade 10
Human (from Crossley et al. 2025)	0.745	– / – / –
DeBERTa-Base	0.790	0.696 / 0.659 / 0.800
XLNet-Base	0.784	0.654 / 0.640 / 0.798
Longformer	0.798	0.698 / 0.658 / 0.811
ModernBERT	0.790	0.639 / 0.658 / 0.804
Mamba-130m	0.797	0.674 / 0.640 / 0.812
Llama-3.2-8B	0.792	0.667 / 0.672 / 0.803

These results support several conclusions. Longformer achieves the highest overall QWK at 0.798, while Mamba-130m is nearly identical overall at 0.797 and slightly higher on grade 10 at 0.812 versus 0.811 for Longformer. Llama-3.2-8B slightly exceeds DeBERTa-Base overall, 0.792 versus 0.790, but does not surpass the strongest encoder-based long-context models (Ormerod et al., 12 Sep 2025).

The paper’s interpretation is nuanced. Long-context models do not necessarily yield large QWK gains relative to a strong 512-token baseline, but they address a validity problem by enabling scoring of full essays without truncation. This is especially relevant for rubric components requiring access to organization and long-range discourse structure. The study further argues that efficient long-context models such as Mamba, and related hybrid architectures such as Jamba, are promising for operational AES because they scale linearly with sequence length and can provide 2–8x speed improvements over transformers on long sequences, while performing strongly on long-context benchmarks such as RULER (Ormerod et al., 12 Sep 2025).

No qualitative essay examples, no fine-grained error analysis, and no explicit ablations over context window size are provided. As a result, the benchmark evidence is strongest on