CoNLL-2013 Shared Task

• CoNLL-2013 Shared Task is a community evaluation that benchmarks automatic grammatical error correction in ESL essays using targeted error categories and an edit-based M² scoring method.
• The task employs pre-segmented and tokenized texts from the NUCLE corpus with expert manual corrections to focus on articles, prepositions, noun number, verb form, and subject–verb agreement.
• It showcases diverse system approaches—including classifier-based, rule-based, SMT, and hybrid methods—while pioneering robust evaluation metrics that influence future GEC research.

Searching arXiv for the cited CoNLL-2013 paper and related GEC shared-task context. CoNLL-2013 Shared Task was a community evaluation for grammatical error correction in English learner essays. It evaluated end-to-end systems that automatically detect and correct grammatical errors in English essays written by learners of English as a second language, with systems receiving essays and returning corrected essays (Ng et al., 13 Jul 2025). The task was defined around five targeted error categories—articles/determiners, prepositions, noun number, verb form, and subject–verb agreement—and used the NUS Corpus of Learner English together with a blind test set of newly collected essays (Ng et al., 13 Jul 2025). Its methodological significance lies in coupling a constrained error-correction setting with edit-based evaluation via the $M^2$ (MaxMatch) scorer, which was introduced in the shared-task framework to avoid scoring failures caused by alternative but valid edit segmentations (Ng et al., 13 Jul 2025).

1. Task definition and scope

The task goal was grammatical error correction in English ESL essays at the essay-to-essay level. Teams were given training data with manual corrections and a blind test set. Preprocessed test essays were provided in sentence-segmented and tokenized form, and submissions had to be sentence-segmented and tokenized corrected essays (Ng et al., 13 Jul 2025).

Gold corrections were represented as edits defined by a span, a replacement string, and an error type. In NUCLE, these edits were encoded as stand-off SGML annotations with the fields start_par, start_off, end_par, end_off, TYPE, and CORRECTION (Ng et al., 13 Jul 2025). The paper illustrates article/determiner correction with the sentence “From past to the present, many important innovations have surfaced.” and the edit past → the past, represented as an ArtOrDet mistake (Ng et al., 13 Jul 2025).

The shared task focused on five categories:

• ArtOrDet: articles/determiners
• Prep: prepositions
• Nn: noun number
• Vform: verb form/tense/aspect
• SVA: subject–verb agreement (Ng et al., 13 Jul 2025)

The scope restriction was explicit: essays naturally contain other errors, including pronouns, sentence structure, punctuation, capitalization, word choice, and orthography, but systems were instructed to correct only the five targeted types, while other errors should be left unchanged (Ng et al., 13 Jul 2025). This design made the task narrower than unconstrained grammatical error correction, while still preserving realistic interactions among edits. The paper’s example “Although we have to admit some bad effect which is brought by the new technology...” shows such interaction: effect → effects induces the need for is → are, demonstrating that multiple edits within a sentence can be coupled rather than independent (Ng et al., 13 Jul 2025).

2. Data resources and annotation design

The training resource was NUCLE release 2.3, an English learner corpus consisting of academic argumentative essays written by non-native English speakers at the National University of Singapore (Ng et al., 13 Jul 2025). The essays cover topics such as surveillance technology and population aging. The release used for the task contained 1,397 essays, 57,151 sentences, and 1,161,567 tokens (Ng et al., 13 Jul 2025).

The essays were corrected by professional English instructors using a custom GUI, and the corpus originally classified errors into 27 types (Ng et al., 13 Jul 2025). Release 2.3 removed 17 duplicate essays. To make prepositions and determiners explicit for the task, the organizers mapped original types so that Wcip and Rloc were algorithmically split into Prep, Wci, ArtOrDet, and Rloc−, using POS tags and parses; mapping code was provided (Ng et al., 13 Jul 2025). Organizers also released a preprocessed version with sentence segmentation and tokenization using NLTK, together with POS tagging and constituency and dependency parsing using the Stanford parser, and character-level edits were mapped to token-level (Ng et al., 13 Jul 2025).

The blind test set consisted of 50 essays written by 25 NUS ESL students, with each student responding to two prompts: RFID and surveillance tracking, and rising life expectancy as challenge and achievement (Ng et al., 13 Jul 2025). The test set contains 1,381 sentences and 29,207 tokens. A native English-speaking lecturer at NUS CELC annotated the essays, and the edits reflect the five targeted error types. Inter-annotator agreement was not reported (Ng et al., 13 Jul 2025).

The error distributions motivated the focus on the five categories. In training data, ArtOrDet accounted for 6,658 edits (14.8%), Prep for 2,404 (5.3%), Nn for 3,779 (8.4%), Vform for 1,453 (3.2%), and SVA for 1,527 (3.4%), totaling 15,821 edits or 35.1% of all 45,106 edits (Ng et al., 13 Jul 2025). In test data, the corresponding counts were 690 (19.9%), 312 (9.0%), 396 (11.4%), 122 (3.5%), and 124 (3.6%), totaling 1,644 edits or 47.4% of all 3,470 edits (Ng et al., 13 Jul 2025). This suggests that the task was designed around error types that constituted one-third to nearly half of observed corrections, while deliberately excluding broader categories from the correction objective.

3. Evaluation methodology and the $M^2$ scorer

A central contribution of the shared task was its evaluation methodology. The paper states that the earlier HOO scorer aligns system edits by wdiff and can mismatch valid alternate segmentations, yielding erroneous scores (Ng et al., 13 Jul 2025). The canonical example is the sentence “There is no a doubt, tracking system has brought ...”, where the gold edits are {a doubt → doubt, system → systems, has → have} and a system output correcting only a doubt → doubt is wrongly scored as zero under the HOO alignment because wdiff produces {a → \epsilon} instead of the gold representation (Ng et al., 13 Jul 2025).

The $M^2$ scorer was designed to avoid this failure mode. It builds an edit lattice over tokenized source $S$ and hypothesis $H$ and searches efficiently for the set of system edits that maximally matches the specified gold edits, ensuring consistent scoring across equivalent segmentations (Ng et al., 13 Jul 2025). In the example above, $M^2$ recovers {a doubt → doubt} and assigns $R=1/3$, $P=1.0$, and $F1=0.5$ (Ng et al., 13 Jul 2025).

CoNLL-2013 further extended the original $M^2$ assumption of one gold set per sentence to allow multiple alternative gold edit sets per sentence. For each sentence $M^2$0, the scorer chooses the gold set $M^2$1 that maximizes cumulative $M^2$2 across sentences $M^2$3; ties are broken by maximizing the numerator $M^2$4, then minimizing the denominator $M^2$5, then selecting the earliest alternative in the list (Ng et al., 13 Jul 2025).

The paper reports the following sentence-aggregated definitions:

$M^2$6

$M^2$7

$M^2$8

and equivalently,

$M^2$9

(Ng et al., 13 Jul 2025)

The paper also states the standard pointwise definitions

$M^2$0

$M^2$1

$M^2$2

but notes that CoNLL-2013 itself reported $M^2$3 with $M^2$4, not $M^2$5; later GEC tasks commonly use $M^2$6 (Ng et al., 13 Jul 2025). No sentence-level accuracy was reported (Ng et al., 13 Jul 2025).

This evaluation choice has enduring technical importance. The task framed edit scoring as a maximal-match problem over tokenized inputs rather than a literal string-diff problem, thereby making representation-equivalent corrections comparable. A plausible implication is that the shared task helped normalize edit-based scoring practice in GEC by emphasizing alignment robustness over superficial segmentation choices.

4. Participating systems and technical approaches

Seventeen teams submitted systems: CAMB, HIT, IITB, KOR, NARA, NTHU, SAAR, SJT1, SJT2, STAN, STEL, SZEG, TILB, TOR, UAB, UIUC, and UMC, corresponding respectively to Cambridge, Harbin Institute of Technology, IIT Bombay, Korea University, NAIST, National Tsing Hua University, Saarland, Shanghai Jiao Tong, Stanford, Stellenbosch, Szeged, Tilburg, Toronto, Autònoma de Barcelona, Illinois Urbana–Champaign, and Macau (Ng et al., 13 Jul 2025).

The paper groups approaches into several families (Ng et al., 13 Jul 2025):

Family	Description	Example teams
Classifier-based (M)	Maximum entropy, SVM, naive Bayes, averaged perceptrons; often one classifier per error type	HIT, IITB, KOR, SZEG, TILB, UIUC, UMC
Rule-based (R)	Deterministic handcrafted rules	HIT, SAAR, UAB
Machine Translation (T)	Phrase-based or syntax-based SMT treating “bad English → good English”	CAMB, NARA, STEL
Language modeling (L)	n-gram and dependency-based LMs for re-ranking or direct correction	NTHU, NARA
Hybrid (H)	Pipelines combining rules, ML classifiers, and LM rescoring	UMC, STAN

Classifier-based systems often used one classifier per error type, such as noun-number choices between singular and plural or article choices among a/an, the, and $M^2$7 (Ng et al., 13 Jul 2025). HIT combined maximum entropy with rules for verbs, IITB used maximum entropy with rules for subject–verb agreement, UIUC used a perceptron for articles and naive Bayes for others, and UMC used semi-supervised maximum entropy within a pipeline (Ng et al., 13 Jul 2025).

Machine-translation approaches treated correction as translation from erroneous to corrected English. CAMB used factored phrase-based SMT with an IRST LLM, NARA used phrase-based SMT together with separate classifiers, and STEL used tree-to-string GHKM (Ng et al., 13 Jul 2025). Language-modeling components included n-gram, dependency-based, and treelet LLMs (Ng et al., 13 Jul 2025). Hybrid systems combined rule-based filters, machine-learned classifiers, and LM confidence; STAN used ERG with mal-rules in HPSG (Ng et al., 13 Jul 2025).

Features and external resources included lexical n-grams, lemmas, POS tags, constituency and dependency parses, and sometimes semantic roles, together with Google Web-1T, Gigaword, Wikipedia, WordNet, Cambridge Learner Corpus, Europarl, ERG grammar, CMU Pronouncing Dictionary, FreeLing dictionaries, JMySpell, Lang-8, and Penn Treebank (Ng et al., 13 Jul 2025). By error type, articles/determiners and noun number were often treated with local classifiers using POS and head–modifier context; prepositions relied on collocational or LM evidence; subject–verb agreement and verb form used verb POS patterns, parses, or rule sets; SMT systems attempted broader phrasal rewrites (Ng et al., 13 Jul 2025).

This distribution of methods shows that CoNLL-2013 was not dominated by a single paradigm. Instead, it functioned as a comparative benchmark across symbolic, discriminative, and translation-based formulations of GEC.

5. Results and performance patterns

The shared task reported an overall leaderboard using $M^2$8 and $M^2$9 without alternative answers (Ng et al., 13 Jul 2025). The top systems were UIUC, NTHU, HIT, NARA, and UMC. UIUC achieved $S$0, $S$1, and $S$2, the highest overall score (Ng et al., 13 Jul 2025).

Rank	Team	R	P	F1
1	UIUC	23.49	46.45	31.20
2	NTHU	26.35	23.80	25.01
3	HIT	16.56	35.65	22.61
4	NARA	18.62	27.39	22.17
5	UMC	17.53	28.49	21.70

The full ranking continued through STEL, SJT1, CAMB, IITB, STAN, TOR, KOR, TILB, SZEG, UAB, SAAR, and SJT2 (Ng et al., 13 Jul 2025). No official baseline system or statistical significance testing was reported (Ng et al., 13 Jul 2025).

Five teams—NTHU, STEL, TOR, UIUC, and UMC—submitted proposed alternative gold edits post-release, and re-evaluation with extended $S$3 improved all scores (Ng et al., 13 Jul 2025). UIUC remained first with $S$4, $S$5, and $S$6; NTHU reached $S$7, UMC $S$8, NARA $S$9, and HIT $H$0 (Ng et al., 13 Jul 2025). The organizers recommended that future work report scores without alternative answers to avoid bias favoring teams who submit alternatives (Ng et al., 13 Jul 2025).

Per-error-type analysis showed clear asymmetries. Noun number yielded the highest $H$1 values across teams, with examples including UIUC at 44.25, NTHU at 43.28, and STEL at 26.18 (Ng et al., 13 Jul 2025). Preposition errors were the hardest, with low $H$2s, often in single digits (Ng et al., 13 Jul 2025). Article/determiner and verb categories occupied a middle range, depending on system design, and precision–recall trade-offs were substantial: some SMT systems had high precision but low recall, while some classifiers achieved more balanced gains (Ng et al., 13 Jul 2025).

These results suggest that local morphosyntactic phenomena such as noun number were more amenable to the available modeling strategies than prepositional choice, which depends more heavily on idiomatic usage and collocational constraints.

6. Analytical findings, limitations, and legacy

The paper identifies several core challenges. Interacting errors require non-pipelined or carefully coordinated corrections, because naive pipelines can miss coupled edits; the example given is applying subject–verb agreement correction before noun-number correction (Ng et al., 13 Jul 2025). Prepositions remain particularly difficult due to idiomatic usage and collocational constraints (Ng et al., 13 Jul 2025). Multiple valid corrections necessitate evaluation that can handle alternative golds, and $H$3’s maximal match alleviates representation sensitivity (Ng et al., 13 Jul 2025).

The strongest-performing systems shared several characteristics. Strong classifier pipelines with robust lexical, POS, and contextual features plus LM rescoring, exemplified by UIUC and UMC, performed best overall (Ng et al., 13 Jul 2025). SMT-style approaches, especially factored or syntax-aware models with strong LLMs, provided high precision for certain categories, as exemplified by CAMB (Ng et al., 13 Jul 2025). Noun number benefited from clear POS patterns and local cues, whereas verb form and subject–verb agreement benefited from parse-based features or mal-rule grammars such as those used by STAN (Ng et al., 13 Jul 2025).

The task also had explicit limitations. Coverage was restricted to five error types that represented 35% of training edits and 47% of test edits; broader categories such as word choice, sentence structure, and punctuation were present but not to be corrected (Ng et al., 13 Jul 2025). Evaluation with alternative answers could inflate scores for teams that submit many alternatives, leading the organizers to recommend reporting no-alternative scores (Ng et al., 13 Jul 2025). The earlier HOO scorer’s reliance on wdiff mismatched multi-token edits, motivating the use of $H$4 for fair evaluation (Ng et al., 13 Jul 2025).

In historical context, prior shared tasks HOO-2011 and HOO-2012 focused heavily on articles and prepositions, whereas CoNLL-2013 expanded to five error types and emphasized end-to-end GEC with interacting errors and the $H$5 scorer (Ng et al., 13 Jul 2025). The paper states that CoNLL-2013 served as a benchmark for subsequent GEC work; later shared tasks often adopted $H$6 and broader error sets, but this evaluation employed $H$7 (Ng et al., 13 Jul 2025). This suggests that CoNLL-2013 occupies a transitional position in GEC: broader than early article/preposition-focused evaluations, but still narrower than later fully general correction benchmarks.

7. Reproducibility, resources, and exemplars

The task was accompanied by substantial resources for replication. NUCLE release 2.3 was made publicly available for research, and the CoNLL-2013 shared task page provided data, test set, and scoring information (Ng et al., 13 Jul 2025). The $H$8 scorer was released with documentation, and organizers provided mapping code for the Wcip/Rloc conversion into Prep, ArtOrDet, Wci, and Rloc−, together with a preprocessed NUCLE version containing token-level edits (Ng et al., 13 Jul 2025). CoNLL-2013 test essays and annotations were freely available after the task, unlike HOO-2012 (Ng et al., 13 Jul 2025).

The paper also provides illustrative edit examples for each target category (Ng et al., 13 Jul 2025):

• ArtOrDet: “In late nineteenth century...” with late → the late
• Prep: “controlled by bad men in a not good purpose.” with in → for
• Nn: “such powerful device shall not be made...” with device → devices
• Vform: “our society is progressed well...” with progressed → progressing
• SVA: “People still prefers to bear the risk...” with prefers → prefer

These examples clarify the operational granularity of the task: the unit of evaluation was not the sentence as a whole, but the edit as a typed local transformation embedded in an end-to-end corrected essay (Ng et al., 13 Jul 2025). A plausible implication is that CoNLL-2013’s enduring influence derives not only from its leaderboard, but from the combination of public data, explicit edit formalization, and a scorer tailored to equivalence-preserving alignment.