---
title: CoNLL-2013 Shared Task
url: https://www.emergentmind.com/topics/conll-2013-shared-task
type: topic
---

# CoNLL-2013 Shared Task

Searching arXiv for the cited CoNLL-2013 paper and related GEC shared-task context.
CoNLL-2013 Shared Task was a community evaluation for grammatical error correction in English learner essays. It evaluated end-to-end systems that automatically detect and correct grammatical errors in English essays written by learners of English as a second language, with systems receiving essays and returning corrected essays [2507.09474]. The task was defined around five targeted error categories—articles/determiners, prepositions, noun number, verb form, and subject–verb agreement—and used the NUS Corpus of Learner English together with a blind test set of newly collected essays [2507.09474]. Its methodological significance lies in coupling a constrained error-correction setting with edit-based evaluation via the $M^2$ (MaxMatch) scorer, which was introduced in the shared-task framework to avoid scoring failures caused by alternative but valid edit segmentations [2507.09474].

## 1. Task definition and scope

The task goal was grammatical error correction in English ESL essays at the essay-to-essay level. Teams were given training data with manual corrections and a blind test set. Preprocessed test essays were provided in sentence-segmented and tokenized form, and submissions had to be sentence-segmented and tokenized corrected essays [2507.09474].

Gold corrections were represented as edits defined by a span, a replacement string, and an error type. In NUCLE, these edits were encoded as stand-off SGML annotations with the fields `start_par`, `start_off`, `end_par`, `end_off`, `TYPE`, and `CORRECTION` [2507.09474]. The paper illustrates article/determiner correction with the sentence “From past to the present, many important innovations have surfaced.” and the edit `past → the past`, represented as an `ArtOrDet` mistake [2507.09474].

The shared task focused on five categories:

- **ArtOrDet**: articles/determiners  
- **Prep**: prepositions  
- **Nn**: noun number  
- **Vform**: verb form/tense/aspect  
- **SVA**: subject–verb agreement [2507.09474]

The scope restriction was explicit: essays naturally contain other errors, including pronouns, sentence structure, punctuation, capitalization, word choice, and orthography, but systems were instructed to correct only the five targeted types, while other errors should be left unchanged [2507.09474]. This design made the task narrower than unconstrained grammatical error correction, while still preserving realistic interactions among edits. The paper’s example “Although we have to admit some bad effect which is brought by the new technology...” shows such interaction: `effect → effects` induces the need for `is → are`, demonstrating that multiple edits within a sentence can be coupled rather than independent [2507.09474].

## 2. Data resources and annotation design

The training resource was NUCLE release 2.3, an English learner corpus consisting of academic argumentative essays written by non-native English speakers at the National University of Singapore [2507.09474]. The essays cover topics such as surveillance technology and population aging. The release used for the task contained 1,397 essays, 57,151 sentences, and 1,161,567 tokens [2507.09474].

The essays were corrected by professional English instructors using a custom GUI, and the corpus originally classified errors into 27 types [2507.09474]. Release 2.3 removed 17 duplicate essays. To make prepositions and determiners explicit for the task, the organizers mapped original types so that `Wcip` and `Rloc` were algorithmically split into `Prep`, `Wci`, `ArtOrDet`, and `Rloc−`, using POS tags and parses; mapping code was provided [2507.09474]. Organizers also released a preprocessed version with sentence segmentation and tokenization using NLTK, together with POS tagging and constituency and dependency parsing using the Stanford parser, and character-level edits were mapped to token-level [2507.09474].

The blind test set consisted of 50 essays written by 25 NUS ESL students, with each student responding to two prompts: RFID and surveillance tracking, and rising life expectancy as challenge and achievement [2507.09474]. The test set contains 1,381 sentences and 29,207 tokens. A native English-speaking lecturer at NUS CELC annotated the essays, and the edits reflect the five targeted error types. Inter-annotator agreement was not reported [2507.09474].

The error distributions motivated the focus on the five categories. In training data, `ArtOrDet` accounted for 6,658 edits (14.8%), `Prep` for 2,404 (5.3%), `Nn` for 3,779 (8.4%), `Vform` for 1,453 (3.2%), and `SVA` for 1,527 (3.4%), totaling 15,821 edits or 35.1% of all 45,106 edits [2507.09474]. In test data, the corresponding counts were 690 (19.9%), 312 (9.0%), 396 (11.4%), 122 (3.5%), and 124 (3.6%), totaling 1,644 edits or 47.4% of all 3,470 edits [2507.09474]. This suggests that the task was designed around error types that constituted one-third to nearly half of observed corrections, while deliberately excluding broader categories from the correction objective.

## 3. Evaluation methodology and the $M^2$ scorer

A central contribution of the shared task was its evaluation methodology. The paper states that the earlier HOO scorer aligns system edits by `wdiff` and can mismatch valid alternate segmentations, yielding erroneous scores [2507.09474]. The canonical example is the sentence “There is no a doubt, tracking system has brought ...”, where the gold edits are `{a doubt → doubt, system → systems, has → have}` and a system output correcting only `a doubt → doubt` is wrongly scored as zero under the HOO alignment because `wdiff` produces `{a → \epsilon}` instead of the gold representation [2507.09474].

The $M^2$ scorer was designed to avoid this failure mode. It builds an edit lattice over tokenized source $S$ and hypothesis $H$ and searches efficiently for the set of system edits that maximally matches the specified gold edits, ensuring consistent scoring across equivalent segmentations [2507.09474]. In the example above, $M^2$ recovers `{a doubt → doubt}` and assigns $R=1/3$, $P=1.0$, and $F1=0.5$ [2507.09474].

CoNLL-2013 further extended the original $M^2$ assumption of one gold set per sentence to allow multiple alternative gold edit sets per sentence. For each sentence $i$, the scorer chooses the gold set $g_i$ that maximizes cumulative $F1$ across sentences $1..i$; ties are broken by maximizing the numerator $\sum |g_i \cap e_i|$, then minimizing the denominator $\sum (|g_i|+|e_i|)$, then selecting the earliest alternative in the list [2507.09474].

The paper reports the following sentence-aggregated definitions:

$$
R = \frac{\sum_i |g_i \cap e_i|}{\sum_i |g_i|}
$$

$$
P = \frac{\sum_i |g_i \cap e_i|}{\sum_i |e_i|}
$$

$$
F1 = \frac{2RP}{R + P}
$$

and equivalently,

$$
F1 = \frac{2 \times \sum_i |g_i \cap e_i|}{\sum_i (|g_i| + |e_i|)}
$$

[2507.09474]

The paper also states the standard pointwise definitions

$$
P = \frac{TP}{TP + FP}
$$

$$
R = \frac{TP}{TP + FN}
$$

$$
F_{\beta} = \frac{(1+\beta^{2}) \cdot P \cdot R}{\beta^{2} \cdot P + R},\ \text{with } \beta = 0.5
$$

but notes that CoNLL-2013 itself reported $F1$ with $\beta=1$, not $F0.5$; later GEC tasks commonly use $F0.5$ [2507.09474]. No sentence-level accuracy was reported [2507.09474].

This evaluation choice has enduring technical importance. The task framed edit scoring as a maximal-match problem over tokenized inputs rather than a literal string-diff problem, thereby making representation-equivalent corrections comparable. A plausible implication is that the shared task helped normalize edit-based scoring practice in GEC by emphasizing alignment robustness over superficial segmentation choices.

## 4. Participating systems and technical approaches

Seventeen teams submitted systems: CAMB, HIT, IITB, KOR, NARA, NTHU, SAAR, SJT1, SJT2, STAN, STEL, SZEG, TILB, TOR, UAB, UIUC, and UMC, corresponding respectively to Cambridge, Harbin Institute of Technology, IIT Bombay, Korea University, NAIST, National Tsing Hua University, Saarland, Shanghai Jiao Tong, Stanford, Stellenbosch, Szeged, Tilburg, Toronto, Autònoma de Barcelona, Illinois Urbana–Champaign, and Macau [2507.09474].

The paper groups approaches into several families [2507.09474]:

| Family | Description | Example teams |
|---|---|---|
| Classifier-based (M) | Maximum entropy, SVM, naive Bayes, averaged perceptrons; often one classifier per error type | HIT, IITB, KOR, SZEG, TILB, UIUC, UMC |
| Rule-based (R) | Deterministic handcrafted rules | HIT, SAAR, UAB |
| Machine Translation (T) | Phrase-based or syntax-based SMT treating “bad English → good English” | CAMB, NARA, STEL |
| Language modeling (L) | n-gram and dependency-based LMs for re-ranking or direct correction | NTHU, NARA |
| Hybrid (H) | Pipelines combining rules, ML classifiers, and LM rescoring | UMC, STAN |

Classifier-based systems often used one classifier per error type, such as noun-number choices between singular and plural or article choices among `a/an`, `the`, and $\epsilon$ [2507.09474]. HIT combined maximum entropy with rules for verbs, IITB used maximum entropy with rules for subject–verb agreement, UIUC used a perceptron for articles and naive Bayes for others, and UMC used semi-supervised maximum entropy within a pipeline [2507.09474].

Machine-translation approaches treated correction as translation from erroneous to corrected English. CAMB used factored phrase-based SMT with an IRST language model, NARA used phrase-based SMT together with separate classifiers, and STEL used tree-to-string GHKM [2507.09474]. Language-modeling components included n-gram, dependency-based, and treelet language models [2507.09474]. Hybrid systems combined rule-based filters, machine-learned classifiers, and LM confidence; STAN used ERG with mal-rules in HPSG [2507.09474].

Features and external resources included lexical n-grams, lemmas, POS tags, constituency and dependency parses, and sometimes semantic roles, together with Google Web-1T, Gigaword, Wikipedia, WordNet, Cambridge Learner Corpus, Europarl, ERG grammar, CMU Pronouncing Dictionary, FreeLing dictionaries, JMySpell, Lang-8, and Penn Treebank [2507.09474]. By error type, articles/determiners and noun number were often treated with local classifiers using POS and head–modifier context; prepositions relied on collocational or LM evidence; subject–verb agreement and verb form used verb POS patterns, parses, or rule sets; SMT systems attempted broader phrasal rewrites [2507.09474].

This distribution of methods shows that CoNLL-2013 was not dominated by a single paradigm. Instead, it functioned as a comparative benchmark across symbolic, discriminative, and translation-based formulations of GEC.

## 5. Results and performance patterns

The shared task reported an overall leaderboard using $M^2$ and $F1$ without alternative answers [2507.09474]. The top systems were UIUC, NTHU, HIT, NARA, and UMC. UIUC achieved $R=23.49$, $P=46.45$, and $F1=31.20$, the highest overall score [2507.09474].

| Rank | Team | R | P | F1 |
|---|---|---:|---:|---:|
| 1 | UIUC | 23.49 | 46.45 | 31.20 |
| 2 | NTHU | 26.35 | 23.80 | 25.01 |
| 3 | HIT | 16.56 | 35.65 | 22.61 |
| 4 | NARA | 18.62 | 27.39 | 22.17 |
| 5 | UMC | 17.53 | 28.49 | 21.70 |

The full ranking continued through STEL, SJT1, CAMB, IITB, STAN, TOR, KOR, TILB, SZEG, UAB, SAAR, and SJT2 [2507.09474]. No official baseline system or statistical significance testing was reported [2507.09474].

Five teams—NTHU, STEL, TOR, UIUC, and UMC—submitted proposed alternative gold edits post-release, and re-evaluation with extended $M^2$ improved all scores [2507.09474]. UIUC remained first with $R=31.87$, $P=62.19$, and $F1=42.14$; NTHU reached $F1=32.46$, UMC $28.90$, NARA $28.14$, and HIT $27.31$ [2507.09474]. The organizers recommended that future work report scores without alternative answers to avoid bias favoring teams who submit alternatives [2507.09474].

Per-error-type analysis showed clear asymmetries. Noun number yielded the highest $F1$ values across teams, with examples including UIUC at 44.25, NTHU at 43.28, and STEL at 26.18 [2507.09474]. Preposition errors were the hardest, with low $F1$s, often in single digits [2507.09474]. Article/determiner and verb categories occupied a middle range, depending on system design, and precision–recall trade-offs were substantial: some SMT systems had high precision but low recall, while some classifiers achieved more balanced gains [2507.09474].

These results suggest that local morphosyntactic phenomena such as noun number were more amenable to the available modeling strategies than prepositional choice, which depends more heavily on idiomatic usage and collocational constraints.

## 6. Analytical findings, limitations, and legacy

The paper identifies several core challenges. Interacting errors require non-pipelined or carefully coordinated corrections, because naive pipelines can miss coupled edits; the example given is applying subject–verb agreement correction before noun-number correction [2507.09474]. Prepositions remain particularly difficult due to idiomatic usage and collocational constraints [2507.09474]. Multiple valid corrections necessitate evaluation that can handle alternative golds, and $M^2$’s maximal match alleviates representation sensitivity [2507.09474].

The strongest-performing systems shared several characteristics. Strong classifier pipelines with robust lexical, POS, and contextual features plus LM rescoring, exemplified by UIUC and UMC, performed best overall [2507.09474]. SMT-style approaches, especially factored or syntax-aware models with strong language models, provided high precision for certain categories, as exemplified by CAMB [2507.09474]. Noun number benefited from clear POS patterns and local cues, whereas verb form and subject–verb agreement benefited from parse-based features or mal-rule grammars such as those used by STAN [2507.09474].

The task also had explicit limitations. Coverage was restricted to five error types that represented 35% of training edits and 47% of test edits; broader categories such as word choice, sentence structure, and punctuation were present but not to be corrected [2507.09474]. Evaluation with alternative answers could inflate scores for teams that submit many alternatives, leading the organizers to recommend reporting no-alternative scores [2507.09474]. The earlier HOO scorer’s reliance on `wdiff` mismatched multi-token edits, motivating the use of $M^2$ for fair evaluation [2507.09474].

In historical context, prior shared tasks HOO-2011 and HOO-2012 focused heavily on articles and prepositions, whereas CoNLL-2013 expanded to five error types and emphasized end-to-end GEC with interacting errors and the $M^2$ scorer [2507.09474]. The paper states that CoNLL-2013 served as a benchmark for subsequent GEC work; later shared tasks often adopted $F0.5$ and broader error sets, but this evaluation employed $F1$ [2507.09474]. This suggests that CoNLL-2013 occupies a transitional position in GEC: broader than early article/preposition-focused evaluations, but still narrower than later fully general correction benchmarks.

## 7. Reproducibility, resources, and exemplars

The task was accompanied by substantial resources for replication. NUCLE release 2.3 was made publicly available for research, and the CoNLL-2013 shared task page provided data, test set, and scoring information [2507.09474]. The $M^2$ scorer was released with documentation, and organizers provided mapping code for the `Wcip`/`Rloc` conversion into `Prep`, `ArtOrDet`, `Wci`, and `Rloc−`, together with a preprocessed NUCLE version containing token-level edits [2507.09474]. CoNLL-2013 test essays and annotations were freely available after the task, unlike HOO-2012 [2507.09474].

The paper also provides illustrative edit examples for each target category [2507.09474]:

- **ArtOrDet**: “In late nineteenth century...” with `late → the late`
- **Prep**: “controlled by bad men in a not good purpose.” with `in → for`
- **Nn**: “such powerful device shall not be made...” with `device → devices`
- **Vform**: “our society is progressed well...” with `progressed → progressing`
- **SVA**: “People still prefers to bear the risk...” with `prefers → prefer`

These examples clarify the operational granularity of the task: the unit of evaluation was not the sentence as a whole, but the edit as a typed local transformation embedded in an end-to-end corrected essay [2507.09474]. A plausible implication is that CoNLL-2013’s enduring influence derives not only from its leaderboard, but from the combination of public data, explicit edit formalization, and a scorer tailored to equivalence-preserving alignment.

Source: https://www.emergentmind.com/topics/conll-2013-shared-task