---
title: Kaggle ASAP 2.0 Dataset Overview
url: https://www.emergentmind.com/topics/kaggle-asap-2-0-dataset
type: topic
---

# Kaggle ASAP 2.0 Dataset Overview

Searching arXiv for the specified paper and closely related ASAP 2.0 work.
The **Kaggle ASAP 2.0 dataset** is a benchmark for **Automated Essay Scoring (AES)** centered on **long-context** student writing. In the literature, it is presented as the **Automated Student Assessment Prize v2 (ASAP 2.0)** dataset and used as a main benchmark for evaluating whether contemporary language-model architectures can score essays without relying on severe truncation, a practice that raises validity concerns for rubric dimensions such as organization and global coherence [2509.10417]. ASAP 2.0 is described as a **“much-needed update”** of the original ASAP dataset, which is treated as saturated as a benchmark, and as an extension of the **PERSUADE corpus** designed to support automated scoring in addition to argumentation and discourse annotation. The resulting benchmark is therefore inherently **persuasive/argumentative** in orientation, with essays written by students in **grades 6–10** [2509.10417].

## 1. Origin, scope, and benchmark role

ASAP 2.0 is positioned as a successor benchmark for AES rather than as a minor revision of the original ASAP corpus. In the reported characterization, the original ASAP benchmark is regarded as no longer sufficiently demanding, particularly for modern language models, whereas ASAP 2.0 introduces essay lengths and discourse demands that materially stress model context handling [2509.10417].

The dataset is derived as an extension of the **PERSUADE corpus** (*Persuasive Essays for Rating, Selecting, and Understanding Argumentative and Discourse Elements*). In the study using it for long-context AES, ASAP 2.0 is explicitly framed not merely as a corpus for discourse or argument mining, but as one designed to support **automated scoring**. This linkage to PERSUADE implies that the essays are **persuasive/argumentative essays**, although the article does not enumerate separate rhetorical domains or prompt families [2509.10417].

The benchmark is organized around **grade-level stratification** rather than prompt-level exposition. Essays are reported for students in **grades 6–10**, but the paper does not provide the number of prompts or the prompt texts themselves. It states only that essays are written on a **wide range of prompts**. A plausible implication is that ASAP 2.0 is intended to assess scoring robustness across heterogeneous persuasive writing tasks without foregrounding prompt-specific modeling in the benchmark description.

## 2. Corpus composition and length characteristics

The reported dataset statistics emphasize both **split size** and **essay length**, because the latter is central to the long-context problem formulation. The paper gives the following word-length statistics by grade for the provided train and test partitions [2509.10417].

| Grade | Train Count / Avg. Train Words | Test Count / Avg. Test Words |
|---|---:|---:|
| 6 | 2094 / 292.2 | 527 / 268.3 |
| 8 | 1648 / 339.9 | 921 / 295.9 |
| 9 | 4002 / 426.1 | 0 / – |
| 10 | 9563 / 385.8 | 5973 / 356.4 |
| **Total** | **17307 / 376.1** | **7421 / 342.7** |

These counts establish several properties of the benchmark. First, the dataset includes **17,307 training essays** and **7,421 test essays**. Second, essays are comparatively long for AES, with average word counts ranging from approximately **268** to **426** depending on grade and split, and a **global training average of 376.1 words**. Third, **grade 9 has no test set** in the table reported by the paper, which has implications for how evaluation results are aggregated and interpreted [2509.10417].

The study does **not** report unified token counts because tokenization varies across models. Instead, it uses **word counts** as the common descriptive statistic. This matters because the benchmark’s long-context status is argued through the mismatch between essay lengths in words and the subword-token limits of widely used transformer encoders. The paper explicitly states that **“many responses in the ASAP 2.0 dataset are too long for most language models”** [2509.10417].

No token-length histograms, percentile summaries, or counts of essays beyond thresholds such as **512** or **1024** tokens are provided. Consequently, fine-grained token-level burden must be inferred from the reported word counts and from the known variability of model-specific subword tokenizers. This suggests that the benchmark’s difficulty is not merely that essays are “long” in an informal sense, but that their lengths interact nontrivially with architecture-specific context budgets.

## 3. Scoring formulation and psychometric framing

In the reported use of ASAP 2.0, the task is **holistic score prediction**: each essay is assigned a **single score** rather than a vector of trait-level scores. The paper treats the problem as **discrete score classification**, not regression and not multi-trait assessment. Classification heads are configured to have **“as many targets as there are scores”**, which means each permissible score value is modeled as a separate class [2509.10417].

The underlying ASAP 2.0 rubric is not described in full detail in the paper. No explicit score range is given, and no score normalization or transformation procedure such as **z-scoring**, **min–max scaling**, or other rescaling is reported. Scores are used directly in discrete form. Predictions are likewise discrete: either the **argmax** of a classifier head or a parsed generated score in the generative setting [2509.10417].

The study reports **human–human Quadratic Weighted Kappa (QWK)** on ASAP 2.0 as:

$$\text{Human rater QWK} = 0.745$$

This value is used as a reliability reference point rather than as proof of a definitive ceiling. The paper explicitly cautions that model performance exceeding human–human QWK does **not necessarily mean** that the model is more valid than human scoring. That caution situates ASAP 2.0 within a psychometric rather than merely leaderboard-oriented framing: agreement statistics are informative, but they are not identical to construct validity [2509.10417].

A common misconception is that exceeding human inter-rater agreement automatically establishes superior assessment quality. The study rejects that inference. A plausible implication is that ASAP 2.0 should be interpreted as a benchmark for **scoring reliability under realistic long-context conditions**, not as a complete substitute for broader validity studies.

## 4. Preprocessing, partitioning, and experimental protocol

Preprocessing is described only at a high level. Models use their **native subword tokenizers**, including tokenizer schemes such as **SentencePiece** for many models, but the paper does not describe lowercasing, punctuation stripping, spelling normalization, essay removal, or malformed-text handling. Nor does it mention sub-sentence segmentation or any explicit decomposition of essays into smaller units before scoring [2509.10417].

The benchmark protocol follows the **provided train/test split** from ASAP 2.0. Within the training partition, **10% of the training set was designated as a development set**. Thus the optimization split corresponds to **90% of 17,307 training essays**, the development split to **10% of 17,307**, and the held-out test set to **7,421 essays** [2509.10417].

The paper does not characterize the splits as **prompt-wise**, **cross-prompt**, or **domain-shift** partitions. It also does not discuss prompt bias mitigation, domain adaptation, or domain generalization. Reported analyses are primarily **grade-level**, with results shown for **grades 6, 8, and 10**, and overall performance aggregated across the test set [2509.10417].

This protocol has two important consequences. First, the benchmark is treated as a realistic operational scoring setting using the dataset’s native partition rather than a deliberately adversarial cross-prompt generalization setup. Second, because prompt-specific evaluation is absent, claims supported by ASAP 2.0 in this study concern **long-context scoring performance on the provided split**, not prompt transfer.

## 5. Long-context problem formulation

The paper’s central claim is that ASAP 2.0 is useful because it exposes the limitations of conventional transformer encoders with short context windows. Standard models such as **BERT** and **DeBERTa** are described as typically supporting **512 tokens** of maximum sequence length. Higher-grade essays in ASAP 2.0 often exceed those limits, making truncation a common practice in AES workflows [2509.10417].

The study frames truncation as a **validity concern** rather than merely an engineering inconvenience. Its explicit argument is that if essays are truncated at **512 tokens**, it becomes difficult to maintain that conventional language models are faithfully modeling rubric dimensions such as **organization**, which depend on access to long-range discourse structure [2509.10417].

The paper formalizes context length through model-specific caps \(L\):

- **DeBERTa-Base**: \(L = 512\)
- **Longformer**: \(L = 4\text{k}\)
- **ModernBERT**: \(L = 8\text{k}\)
- **XLNet-Base**: \(L = 8\text{k}^*\)
- **Mamba-130m**: \(L = 8\text{k}^*\)
- **Llama-3.2-8B**: \(L = 8\text{k}\)

For **XLNet** and **Mamba**, the asterisk denotes a **training-imposed cap** rather than an intrinsic architectural limit [2509.10417].

The XLNet discussion introduces a more formal long-context perspective via effective receptive field:

> the output of any token is only a function of at most \(L D\) of the previous tokens where \(D\) is the depth of the network.

Using the reported base-model values, \(L = 512\) and \(D = 12\) yield approximately **6,000 tokens** of effective dependence; for a large model with \(D = 24\), the figure is approximately **12,000 tokens** [2509.10417]. This is used to motivate the claim that some architectures can, in principle, model essays whose full informational dependencies exceed the nominal local segment length.

A plausible implication is that ASAP 2.0 functions as a **stress test** not just for raw sequence length, but for the ability to preserve rubric-relevant discourse information over long spans.

## 6. Modeling strategies applied to ASAP 2.0

The study evaluates six fine-tuned model families on ASAP 2.0: **DeBERTa-Base**, **XLNet-Base**, **Longformer**, **ModernBERT**, **Mamba-130m**, and **Llama-3.2-8B** [2509.10417]. Their inclusion makes ASAP 2.0 a comparative benchmark across standard transformers, sparse-attention transformers, recurrent-segment transformers, state-space models, and generative decoder-only models.

For **DeBERTa-Base**, ASAP 2.0 serves as a benchmark for common AES practice under a **512-token** limit. The model is described as a strong baseline, but essays longer than 512 tokens must be **truncated**. The truncation policy itself is not specified [2509.10417].

For **Longformer**, the context length is **4k tokens** and the architecture uses **sliding-window local attention** with **sparse global attention** to selected tokens. In the study’s framing, this allows **full coverage of most essays** in ASAP 2.0 and reduces or eliminates truncation relative to 512-token models [2509.10417].

For **XLNet**, the model inherits **recurrent attention** from Transformer-XL, using segment-level recurrence:

$$
\begin{aligned}
\tilde{h}_{\tau+1}^{n-1} &= [SG(h_{\tau}^{n-1}) \circ h_{\tau+1}^{n-1}], \\
q_{\tau+1}^{n} &= h_{\tau+1}^{n-1} W_q, \\
k_{\tau+1}^{n} &= \tilde{h}_{\tau+1}^{n-1} W_k, \\
v_{\tau+1}^{n} &= \tilde{h}_{\tau+1}^{n-1} W_v, \\
h_{\tau+1}^{n} &= \mathrm{MHA}(q_{\tau+1}^{n}, k_{\tau+1}^{n}, v_{\tau+1}^{n}),
\end{aligned}
$$

where \(SG\) is stop-gradient, \(\circ\) is concatenation, and \(\mathrm{MHA}\) is multihead attention [2509.10417]. This architecture is used to support cross-segment dependency propagation on long essays.

For **ModernBERT**, the context length is **8k tokens**. The paper attributes its long-context capability to **Rotary Position Embeddings (RoPE)** and a **two-phase pretraining** strategy that integrates short- and long-range positional information [2509.10417].

For **Mamba-130m**, the architecture is a **state-space model (SSM)** rather than an attention-based transformer. The paper gives the continuous-time formulation

$$
\begin{aligned}
x'(t) &= A x(t) + B u(t), \\
y(t) &= C x(t) + D u(t),
\end{aligned}
$$

with discretization

$$
\begin{aligned}
h_t &= A h_{t-1} + B x_t,\\
x &= C h_t.
\end{aligned}
$$

and notes that its computational complexity **scales linearly** with sequence length [2509.10417]. The model is therefore presented as especially suitable for long ASAP 2.0 essays.

For **Llama-3.2-8B**, the context length is **8k tokens**. Unlike the encoder-style classifiers, it is fine-tuned in an **instruction-following** format. The input explicitly includes a rubric and essay, and the model is trained to generate the score:

```text
Assign a **Score** to the
**Essay** using the **Rubric**
provided.

**Rubric**: {rubric}

**Essay**:
```

with output of the form:

```text
**Score**: {score}
```

The rubric is **paraphrased in 20 different ways** for robustness, and the best-performing rubric variant on the development set is chosen by **QWK** [2509.10417].

The paper does not describe explicit essay chunking for any model. Long-context architectures are instead relied upon to process **whole essays where possible**.

## 7. Optimization, evaluation, and empirical findings

For the discriminative models—**DeBERTa, XLNet, Longformer, ModernBERT, and Mamba**—the original token-prediction head is removed and replaced with a classification head having **as many targets as there are scores**. Training uses **cross-entropy loss** over discrete score classes. The paper presents the standard form:

$$
\mathcal{L}_{\text{CE} = - \sum_{i} \sum_{k=1}^K \mathbf{1}[y_i = k] \log \hat{p}_{i,k}.
$$

For most classification models, optimization uses **Adam with decoupled weight decay**, an **initial learning rate of \(10^{-6}\)**, a **linear scheduler** decaying the learning rate to zero over **10 epochs**, **batch size 4 or 1 depending on essay length**, and **early stopping** based on development-set **QWK** [2509.10417].

**Mamba** departs from this pattern in one critical respect. To prevent model collapse, the study reports that it was necessary to **effectively freeze the weights associated with the SSM, \(L_{gate}\), and the convolutional layer**. Fine-tuning is then performed on the **embedding layer** and the **\(L_{in}\)** and **\(L_{out}\)** weights of every layer, using **Adam**, **learning rate \(10^{-5}\)**, and **batch size 8** [2509.10417].

For **Llama-3.2-8B**, fine-tuning uses **LoRA** and **QLoRA**, with LoRA applied to the attention matrices \(L_q\), \(L_k\), and \(L_v\). The objective is standard **next-token prediction loss**, with the score generated as text [2509.10417].

Evaluation is based primarily on **Quadratic Weighted Kappa (QWK)**, defined as

$$
\kappa = 1 - \dfrac{\sum_{i,j} W_{i,j}\, O_{i,j}}{\sum_{i,j} W_{i,j}\, E_{i,j}},
$$

with

$$
W_{i,j} = \frac{(i-j)^2}{(n-1)^2},
$$

where \(O_{i,j}\) is observed agreement, \(E_{i,j}\) is expected agreement, and \(n\) is the number of score categories [2509.10417]. Although the paper mentions **“the standard metrics specified for AES”**, only **QWK** is actually reported in the presented tables.

The reported test-set results are as follows [2509.10417]:

| Model | Overall QWK | Grade 6 / Grade 8 / Grade 10 |
|---|---:|---:|
| Human (from Crossley et al. 2025) | 0.745 | – / – / – |
| DeBERTa-Base | 0.790 | 0.696 / 0.659 / 0.800 |
| XLNet-Base | 0.784 | 0.654 / 0.640 / 0.798 |
| Longformer | 0.798 | 0.698 / 0.658 / 0.811 |
| ModernBERT | 0.790 | 0.639 / 0.658 / 0.804 |
| Mamba-130m | 0.797 | 0.674 / 0.640 / 0.812 |
| Llama-3.2-8B | 0.792 | 0.667 / 0.672 / 0.803 |

These results support several conclusions. **Longformer** achieves the highest **overall QWK** at **0.798**, while **Mamba-130m** is nearly identical overall at **0.797** and slightly higher on **grade 10** at **0.812** versus **0.811** for Longformer. **Llama-3.2-8B** slightly exceeds **DeBERTa-Base** overall, **0.792** versus **0.790**, but does not surpass the strongest encoder-based long-context models [2509.10417].

The paper’s interpretation is nuanced. Long-context models do **not necessarily yield large QWK gains** relative to a strong **512-token** baseline, but they address a **validity problem** by enabling scoring of full essays without truncation. This is especially relevant for rubric components requiring access to **organization** and **long-range discourse structure**. The study further argues that efficient long-context models such as **Mamba**, and related hybrid architectures such as **Jamba**, are promising for operational AES because they scale **linearly with sequence length** and can provide **2–8x speed improvements** over transformers on long sequences, while performing strongly on long-context benchmarks such as **RULER** [2509.10417].

No qualitative essay examples, no fine-grained error analysis, and no explicit ablations over context window size are provided. As a result, the benchmark evidence is strongest on

Source: https://www.emergentmind.com/topics/kaggle-asap-2-0-dataset