DEFT 2020 Evaluation Campaign

• The DEFT 2020 Evaluation Campaign is a shared task on definition extraction from free text, emphasizing structured prediction through sentence classification, sequence labeling, and relation extraction.
• The campaign employs the DEFT corpus with 3-sentence windows and a rich BIO-tag annotation schema covering nine semantic types and five relation types.
• Performance was measured using strict metrics (Precision, Recall, F1), highlighting practical challenges like class imbalance, tokenization alignment, and overfitting.

Searching arXiv for the DEFT 2020 evaluation campaign and closely related SemEval-2020 Task 6 papers. arXiv search query: "SemEval-2020 Task 6 Definition extraction from free text DEFT corpus" DEFT 2020 Evaluation Campaign, released as SemEval-2020 Task 6 under the name DeftEval, was a shared task on definition extraction from free text built around the DEFT corpus. Its central objective was to evaluate systems that can identify definitional content in naturally occurring prose rather than only in stereotyped patterns such as explicit “$X$ is a $Y$” constructions. The campaign operationalized this objective through sentence classification, sequence labeling, and relation extraction, thereby treating definition extraction as a structured prediction problem over terms, definitions, auxiliary spans, and their links (Spala et al., 2020).

1. Origins, motivation, and scope

DeftEval was designed in response to long-standing constraints in definition extraction research. Earlier resources were described as being focused on simple definitional patterns, whereas the DEFT task emphasized that real definitions and glosses in free text often appear without explicit markers, can cross sentence boundaries, and may involve referential or auxiliary material such as aliases and qualifiers (Spala et al., 2020). The defx system paper makes the same contrast by noting that, unlike earlier datasets such as WCL and W00, the DEFT corpus provides windows of up to three sentences centered around a highlighted word and allows multiple definitional concepts and relations, including indirect definitions expressed through cross-sentence coreference (Hübner et al., 2021).

The campaign used English textbook material for the shared task. The broader DEFT corpus also aggregates 2017 SEC contract filings from the US SEC EDGAR database, but only the textbook portion was required for participation (Spala et al., 2020). RGCL describes the broader campaign context as targeting automatic identification and extraction of definitional knowledge from “messy,” naturally occurring text, with downstream uses including dictionary building, ontology construction, and information extraction (Ranasinghe et al., 2020).

The evaluation schedule was explicit. Practice data were released on 15 Aug 2019, training data on 4 Sept 2019, Subtasks 1–2 evaluation began on 19 Feb 2020, Subtask 3 evaluation began on 1 Mar 2020, and the competition ended on 13 Mar 2020, with each evaluation window lasting 12 days (Spala et al., 2020). Participants were nominally allowed 5 submissions per subtask, although a Codalab limitation merged Subtasks 1 and 2 into a single phase with 5 total submissions, after which additional submissions could be opened by request (Spala et al., 2020).

2. Corpus design and annotation schema

The DEFT corpus was constructed to model definitional content in context rather than as isolated glosses. For each bolded or highlighted candidate term in the source textbooks, the corpus includes a 3-sentence window consisting of the sentence before, the main sentence with the candidate term, and the sentence after; definitions outside this window are out of scope for the task (Spala et al., 2020). At publication time, after tokenization fixes during the task, the textbook portion contained 16,056 sentences, 695,627 tokens, and 9,439 context windows (Spala et al., 2020).

The annotation schema is comparatively rich. In the organizer description, token labels are BIO-tagged and span nine semantic types: Term, Alias Term, Ordered Term, Referential Term, Definition, Secondary Definition, Ordered Definition, Referential Definition, and Qualifier (Spala et al., 2020). The campaign also annotates five relation types: Direct-defines, Indirect-defines, Refers-to, AKA, and Supplements (Spala et al., 2020). These labels encode not only direct definitional content but also cross-sentence and structurally paired phenomena. Primary term–definition pairs are captured by Direct-defines edges, while Secondary Definition, Referential Term, Referential Definition, and Indirect-defines or Refers-to links capture cross-sentence phenomena within the three-sentence window (Spala et al., 2020).

Participant papers foregrounded different views of the same schema. RGCL describes Subtask 2 as CoNLL-like BIO tagging with fine-grained labels covering primary, secondary, referential, and ordered forms for both terms and definitions (Ranasinghe et al., 2020). UPB enumerates token-level labels as Term, Alias-Term, Referential-Term, Definition, Referential-Definition, and Qualifier in its system formulation (Avram et al., 2020). The defx paper similarly lists six key concept types and notes that the corpus permits aliases, supplementary information, and indirect definitions via coreference (Hübner et al., 2021). This suggests that the organizer paper provides the fullest campaign-wide formalization, while individual system papers often emphasize the portions of the schema most relevant to their models.

The corpus also exposed annotation constraints. defx reports that no nested annotations were allowed, although the authors observed cases where nesting would have been appropriate, and they note some duplicated sentences where overlaps would have occurred (Hübner et al., 2021). That observation is significant because definitional language frequently contains embedded nominal or referential structures that strain flat BIO schemes.

3. Task structure and official evaluation protocol

DeftEval was organized into three subtasks that increase in granularity from sentence-level detection to full relational structure recovery (Spala et al., 2020).

Subtask	Objective	Official evaluation
1	Classify each sentence as definitional vs. non-definitional	Instance-level Precision, Recall, F1 for the positive class; micro-averaged
2	Predict BIO tags for the annotated span types	Span-level Precision, Recall, F1; strict exact match; micro-averaged
3	Predict sister tag ID and relation label between spans	Relation-level Precision, Recall, F1; exact relation and endpoint match; micro-averaged

For Subtask 1, the official converter script generated sentence-level instances from the sequence-labeling files, and a sentence was labeled positive if it contained a token span annotated as Definition. Sentences containing only Secondary Definition, with no primary Definition span, were labeled negative (Spala et al., 2020). For Subtask 2, test instances were presented in a CoNLL-like format in which each line contained the token, source file identifier, and lower and upper character offsets; participants predicted only the BIO tag (Spala et al., 2020). For Subtask 3, the test format included gold BIO tags and tag IDs, and systems had to predict the sister tag’s ID together with the relation label (Spala et al., 2020).

The official metrics were the standard definitions $P = \frac{TP}{TP + FP}$, $R = \frac{TP}{TP + FN}$, and $F1 = \frac{2PR}{P + R}$ (Spala et al., 2020). Matching criteria were strict. In Subtask 2, a predicted span was correct only if both boundaries and label type exactly matched a gold span; no partial credit was awarded (Spala et al., 2020). In Subtask 3, a predicted relation counted as correct only if the relation label and both endpoints matched exactly (Spala et al., 2020).

Participant reports added secondary evaluation perspectives. RGCL reports Subtask 1 performance per class, weighted averages, and Macro-F1, and reports overall token-level Precision, Recall, and F1 for Subtask 2 (Ranasinghe et al., 2020). defx states that the official metric for its Subtasks 2 and 3 runs was macro-averaged F1 across labels and highlights severe class imbalance as a major challenge (Hübner et al., 2021). UPB reports Macro-Precision, Macro-Recall, and Macro-F1 for Subtasks 1 and 2 in development and test analyses (Avram et al., 2020). Taken together, these accounts show that while the shared task had strict official scoring, team papers often used macro-averaged diagnostics to study label imbalance and model behavior.

4. Participation, leaderboard structure, and empirical landscape

The campaign attracted substantial participation. During the training phase there were 279 submissions through Codalab, and the evaluation phase received 217 submissions from 56 participants for Subtask 1, 220 submissions from 51 participants for Subtask 2, and 113 submissions from 27 participants for Subtask 3; the organizers report more than 400 total submissions and 149 registered participants (Spala et al., 2020).

The dominant methodological trend in Subtasks 1 and 2 was the use of Transformer-based models. The organizer paper notes widespread use of BERT, RoBERTa, and XLNet, sometimes augmented with BiLSTM and attention layers, alongside multi-task systems and classic CRF or RNN/CNN baselines (Spala et al., 2020). The three system papers reinforce that pattern. RGCL evaluated CNN, BiLSTM, and several Transformer encoders, with Transformer models outperforming CNN and RNN and XLNet yielding the best development results (Ranasinghe et al., 2020). UPB systematically compared BERT, XLNet, RoBERTa, SciBERT, and ALBERT in both frozen and fine-tuned regimes (Avram et al., 2020). defx pursued a joint NER+RE model with BERT, BiLSTM, and CRF, supplemented by POS, coreference, and rule-based features (Hübner et al., 2021).

Several reported standings illustrate the mid-table competitiveness of representative systems. RGCL achieved Macro-F1 = 0.7885 and rank 25th of 56 teams on Subtask 1, and F1 = 0.4872 with rank 28th of 51 teams on Subtask 2; the team did not submit for Subtask 3 (Ranasinghe et al., 2020). defx reports macro F1 = 49.68 on Subtask 2, ranking 27th of 51 participating teams (Hübner et al., 2021). UPB reports that its best system ranked 32nd out of 56 submissions on Subtask 1 with test F1 = 0.777 and 37th out of 51 submissions on Subtask 2 with test F1 = 0.439 (Avram et al., 2020). UPB also places its results against a winning-team score of 87.7 F1 on Subtask 1 and 84.7 F1 on Subtask 2 (Avram et al., 2020).

Subtask 3 behaved differently from the first two tasks. The organizer analysis reports a median F1 of 0.9043, with 16 of 27 participants exceeding 0.9; the top-ranked UNIXLONG system used BERT plus hand-crafted rules, and ACNLP’s Random Forest ranked fourth (Spala et al., 2020). This unusually high performance became one of the major interpretive issues of the campaign.

5. Representative system designs

RGCL presented a flexible neural framework for Subtasks 1 and 2 built around interchangeable architectures. For sentence classification, the system used standard Transformer text classification in which the $[\mathrm{CLS}]$ representation was passed to a softmax classifier, and all parameters were fine-tuned jointly; the team evaluated CNN, BiLSTM, BERT, XLNet, XLM, RoBERTa, and DistilBERT, with XLNet giving the best development results (Ranasinghe et al., 2020). For token-level sequence labeling, Transformer encoders produced token-level embeddings and a token classification head predicted fine-grained BIO labels; XLNet variants again performed best, and reducing maximum sequence length from 128 to 64 improved results (Ranasinghe et al., 2020). RGCL also tested Wikipedia-based weak supervision, but automatic harvesting reduced Macro-F1 in Subtask 1 by about 0.02 and did not improve Subtask 2, partly because of noise and positional bias (Ranasinghe et al., 2020). A CRF layer added after the Transformer did not improve token-labeling performance (Ranasinghe et al., 2020).

The defx submission treated concept extraction and relation extraction as a joint learning problem closely following Bekoulis et al. (2018). Its NER component used BERT token representations formed by averaging layers using learned weights, concatenated with POS embeddings from spaCy, optional NER-label embeddings, binary indicators for coreference participation from AllenNLP coref, and binary indicators for rule-based matches such as demonstratives “this/these”; those features were contextualized with an $n$-layer BiLSTM, then passed through a fully connected layer and a CRF (Hübner et al., 2021). The RE component was an auxiliary objective over token pairs subject to the DEFT format’s single-tail constraint per token, with joint loss defined as the sum of NER and RE losses and equal weighting performing best for Subtask 2 among the weights tried (Hübner et al., 2021). In ablations, the CRF layer consistently helped relative to a simple tagger, whereas joint models tended to increase precision but could reduce recall; on the official development set used as test in their analysis, the CRF tagger was generally best (Hübner et al., 2021).

UPB concentrated on pretrained LLMs and the contrast between freezing and fine-tuning. For Subtask 1, the system encoded sentence text with a PLM, fed the $[\mathrm{CLS}]$ embedding to a two-layer feed-forward network with 512 hidden units per layer, and optimized binary cross-entropy over the final sigmoid output (Avram et al., 2020). For Subtask 2, contextualized subtoken embeddings were projected by a two-layer feed-forward network and decoded with a CRF using Viterbi decoding; because corpus tokens differed from BPE units, UPB reconstructed sentences, retokenized with BPE, aligned subtokens to original tokens by character-level matching, and resolved subtoken-label mismatch at inference by majority vote with a first-subtoken tie-breaker (Avram et al., 2020). Fine-tuning yielded large gains: UPB reports improvements of up to +11.4 F1 points on development for Subtask 1 and up to +23.4 F1 points on development for Subtask 2 (Avram et al., 2020). The multi-task formulation for Subtasks 2 and 3 performed poorly, which the authors attribute to not using the provided gold tags from Subtask 2 and to the complexity of joint modeling without sufficient signal (Avram et al., 2020).

Across these systems, a common pattern emerged: strong contextual encoders were necessary but not sufficient. Performance also depended on tokenization alignment, label imbalance management, careful decoding assumptions, and whether auxiliary structures such as coreference or relations actually contributed useful signal in the DEFT setting (Ranasinghe et al., 2020, Hübner et al., 2021, Avram et al., 2020).

6. Error sources, controversies, and continuing significance

Several difficulties identified during and after the campaign shaped later interpretation of DeftEval. The most prominent was the relative ease of Subtask 3. The organizers argue that the three-sentence windows often contain only a single term–definition pair and its auxiliary spans, collapsing the search space to a single plausible relation instance. They explicitly describe relation extraction as “artificially easy” under this setup and recommend full-document context or merging Subtasks 2 and 3 in future editions to make pairing genuinely challenging (Spala et al., 2020).

A second major issue was data quality and representation mismatch. The organizer paper notes tokenization and sentence-boundary errors introduced by scraping from cnx.org, including raw URLs, stray HTML, and parsing artifacts; many were repaired during the task through GitHub issues (Spala et al., 2020). RGCL’s sentence-level preprocessing directly targeted such artifacts by removing leading enumeration markers such as “41.” and standalone “([link])” placeholders, while preserving placeholders that functioned grammatically in a sentence; this yielded a small improvement of +0.01 across metrics with the best-performing architecture (Ranasinghe et al., 2020). UPB likewise emphasizes tokenization alignment friction between corpus tokens and PLM BPE tokenization and treats character-level matching as critical for token-level learning (Avram et al., 2020).

Class imbalance and schema complexity were persistent modeling bottlenecks. defx highlights severe class imbalance as a major challenge and reports frequent confusion between Term and Alias-Term, high false positives for Term and Definition, missing annotations for cross-sentence referents, and the limitations imposed by the absence of nested annotations (Hübner et al., 2021). UPB reports that dominant labels such as Definition and Term, together with the heavily oversampled Ref-Term label, achieved approximately 64% per-label accuracy, roughly double that of other labels, while most misclassifications mapped to the “O” label (Avram et al., 2020). RGCL observed substantial development-to-test drops in both subtasks and attributes these primarily to overfitting, with the effect more pronounced for token-level extraction (Ranasinghe et al., 2020).

The campaign nevertheless established a durable benchmark and a reproducible workflow. The organizers released the DEFT corpus, evaluation scripts, and example materials through GitHub and Codalab (Spala et al., 2020). Team code was also published: RGCL released its implementation at https://github.com/tharindudr/defteval, defx at https://github.com/DFKI-NLP/defx, and UPB at https://github.com/avramandrei/DeftEval (Ranasinghe et al., 2020, Hübner et al., 2021, Avram et al., 2020). This reproducibility is important because DeftEval did not merely benchmark definition detection; it exposed the interaction between corpus design, discourse-level semantics, label imbalance, and exact-match evaluation.

A common misconception is that DEFT 2020 was simply a sentence-classification exercise over textbook definitions. The campaign design contradicts that view. Its formal annotation space includes aliases, ordered and referential spans, secondary definitions, qualifiers, and typed relations, and its core technical challenge lies in recovering structured definitional knowledge from free text rather than merely detecting definitional sentences (Spala et al., 2020). A plausible implication is that DeftEval’s lasting contribution was not a single winning architecture but a more realistic task formulation for definition extraction, one that made shortcomings in context modeling, coreference handling, and annotation design empirically visible.