---
title: SimpleText CLEF Lab Overview
url: https://www.emergentmind.com/topics/simpletext-clef-lab
type: topic
---

# SimpleText CLEF Lab Overview

Searching arXiv for recent papers on the SimpleText CLEF Lab and related shared-task descriptions.
SimpleText is a recurring lab within CLEF, the Conference and Labs of the Evaluation Forum, devoted to making scientific texts more accessible to non-expert readers through automatic simplification. Across recent editions, participant papers describe the lab as addressing not only sentence- and document-level rewriting, but also passage selection, difficult-concept explanation, and the detection and mitigation of hallucination in simplified scientific text. In the 2025 track, the problem framing is explicitly “Simplify Scientific Text (and Nothing More),” emphasizing simplification that preserves meaning, avoids unsupported additions, and remains faithful to source content [2307.03569] [2507.04414] [2508.11816] [2508.11823].

## 1. Historical development and problem framing

Participant descriptions present SimpleText as a long-running evaluation initiative motivated by the mismatch between expert-oriented scientific writing and the needs of broader audiences. Scientific communication is characterized as dense, terminology-heavy, and often constrained by publication conventions, making it difficult for non-experts such as patients, policymakers, and family members of seriously ill patients to extract core findings without specialist background knowledge. In this framing, simplification is not merely shortening; it is the controlled reduction of lexical and syntactic complexity while preserving meaning, intent, and key information [2307.03569] [2507.04414].

The lab structure has evolved across editions. In 2023, SimpleText ran three coordinated tasks: passage selection for simplified summaries, identification and explanation of difficult concepts, and scientific text simplification for non-experts. By contrast, the 2025 track narrows the scope back to simplification itself and refines the evaluation around sentence-level simplification, document-level simplification, and hallucination avoidance. This progression suggests an increasing emphasis on faithful rewriting rather than adjacent access tasks such as retrieval or concept explanation [2307.03569] [2507.04414] [2508.11816].

A recurrent distinction in these papers is between simplification and summarization. Scientific text simplification is described as preserving the source’s key informational content while rephrasing for accessibility, whereas summarization alone may condense content more aggressively. The 2025 lab framing intensifies this distinction by explicitly penalizing creative generation that goes beyond or diverges from the source [2307.03569] [2508.11823].

## 2. Task organization, datasets, and input–output structure

The tasks described for recent SimpleText editions vary by year and by granularity of transformation.

| Edition / task | Focus | Representative data described in participant papers |
|---|---|---|
| 2023 Task 3 | Simplification of scientific abstracts and sentences for non-experts | Large split with 152,072 source texts; 135,540 unique after de-duplication |
| 2025 Task 1.1 | Sentence-level simplification | 9,160 short English texts in one description; 37 aligned Cochrane-auto abstracts in another evaluation setup |
| 2025 Task 1.2 | Document-level simplification | 37 aligned abstract–PLS pairs and 217 abstract–PLS pairs |
| 2025 Task 2 | Hallucination detection and grounded generation | 3,379 sourced instances, 3,336 post-hoc instances, and 217 plain language summaries for Task 2.3 evaluation |

For 2025 Task 1, one participant paper states that the lab uses the Cochrane-Auto dataset, described as “an aligned dataset for the simplification of biomedical abstracts,” and evaluates both sentence-level and document-level simplification. Another paper reports an aligned subset of 37 abstract–PLS pairs, comprising 587 source sentences and 388 target sentences, alongside a larger unaligned set of 217 abstract–PLS pairs for document-level evaluation [2508.11816] [2512.16541].

Task 1.1 is presented as sentence-level simplification with strict alignment between input and output sentences. One description states that the number of output elements must exactly equal the number of input sentences, while allowing splitting within a list element and omission of non-essential material by outputting the empty string `''` for that element. Task 1.2 is presented as document-level simplification of full abstracts into accessible summaries, with coherence and preservation of key content as central requirements [2512.16541].

The 2025 sentence-focused regime is also characterized as a “short-text” or minimal-context setting. One participant paper reports that the Task 1.1 dataset consists of 9,160 short English texts extracted from scientific publications, of which 9,086 are unique, with average length 168.66 characters. The same paper emphasizes that many instances are single sentences and that systems often must detect and simplify domain-specific terms with little or no external context [2507.04414].

Task 2 in 2025 addresses hallucination explicitly. DS@GT describes Task 2 as “Identify and Avoid Hallucination,” divided into three subtasks: document-level detection of creative generation, sentence-level detection and classification of information distortion errors, and grounded generation that avoids creative overgeneration by design. The task uses scientific abstracts and plain-language summaries, and distinguishes a sourced setting, where the source abstract is provided, from a post-hoc setting, where systems must retrieve proxy context [2508.11823].

The 2023 Task 3 dataset differs markedly in scale and structure. It is described as comprising short segments from scientific publications, primarily single sentences, drawn from diverse domains such as computer forensics, molecular virology, and animal behavior analysis. The large split contains 152,072 source texts, reduced to 135,540 unique items after de-duplication. Reported data quality issues include duplicates, UTF-8 hexcodes embedded in text, incomplete sentences, and a small number of empty or punctuation-only texts [2307.03569].

## 3. Evaluation protocols and measurement regimes

SimpleText does not operate with a single invariant metric across all tasks. For 2025 Task 1, one paper explicitly states: “Only SARI is the official evaluation metric for Task 1. Other metrics are reported for supplementary analysis.” In that setting, SARI is described as comparing system output to both source and references through keep, delete, and add operations, and as being “computed as the average of F1 scores for these three operations,” although the formal equation is not reported in that paper [2508.11816].

Other Task 1 evaluations report BLEU, BERTScore (F1), FKGL, Compression Ratio, Sentence Splits, Levenshtein Similarity, Exact Copies, Additions Proportion, Deletions Proportion, and Lexical Complexity Score. A separate 2025 paper gives the high-level formulation
$$
\mathrm{SARI} = \frac{F_{\mathrm{add}} + F_{\mathrm{keep}} + F_{\mathrm{del}}}{3},
$$
and reports
$$
\mathrm{BLEU} = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right),
$$
with brevity penalty defined piecewise, alongside
$$
\mathrm{FKGL} = 0.39 \cdot \left(\frac{\text{words}}{\text{sentences}}\right) + 11.8 \cdot \left(\frac{\text{syllables}}{\text{word}}\right) - 15.59,
$$
and a compression ratio defined as
$$
\mathrm{CR} = \frac{\text{length(simplified)}}{\text{length(original)}}.
$$
These formulations are used to characterize the simplification–readability trade-off in Task 1 systems [2512.16541].

For 2025 Task 2, the evaluation regime is different. Detection is measured with Accuracy, Precision, Recall, F1, AUROC, and AUPRC. The DS@GT paper reports the standard definitions
$$
P = \frac{TP}{TP + FP}, \quad R = \frac{TP}{TP + FN}, \quad F1 = \frac{2PR}{P + R},
$$
and uses AUROC to assess ranking quality across decision thresholds and AUPRC to emphasize performance on imbalanced classes. Grounded generation in Task 2.3 is evaluated with SARI, BLEU, FKGL, Compression Ratio, Sentence Splits, Levenshtein Similarity, Exact Copies, Additions/Deletions proportions, and Lexical Complexity Score [2508.11823].

The 2023 Task 3 evaluation departs from reference-based simplification metrics because no gold simplifications were provided. Instead, the evaluation emphasized readability-oriented automatic measures together with manual inspection. The paper reports Flesch Reading Ease,
$$
RE = 206.835 - 1.015 \times \left(\frac{\text{words}}{\text{sentences}}\right) - 84.6 \times \left(\frac{\text{syllables}}{\text{words}}\right),
$$
New Dale–Chall, difficult word count, reading time,
$$
RT = 0.01469 \times \text{characters},
$$
syllable count, lexicon count, and sentence count. The authors explicitly note that they did not compute BLEU, ROUGE, or SARI, nor FKGL, because the setting lacked gold references and their focus was readability for non-experts [2307.03569].

An important methodological caveat appears in the 2025 Task 1.1 THM paper: SARI may favor outputs closer to reference wording and may therefore bias evaluation against broader but faithful clarity-oriented rephrasings. This is framed not as a definitive defect of the metric, but as an evaluation caveat observed in practice [2507.04414].

## 4. Core methodological paradigms

Recent SimpleText systems exhibit several recurring methodological patterns.

One pattern is plan-driven sentence simplification. DS@GT’s Task 1.1 system uses llama-3.3-70b-versatile in a two-stage pipeline in which the model first selects a simplification strategy from `rephrase`, `delete`, `split`, `ignore`, and `merge`, and then generates a simplified sentence conditioned on that plan. The inputs include the complex sentence, the full source document, and the next complex sentence for local context. The same paper describes document-level simplification as summary-guided: the model first produces “a clear and concise summary that captures the essential information, main arguments, and key findings” and then rewrites the original document in a simpler way while remaining faithful to both the source and the summary [2508.11816].

A second pattern is complex-term identification followed by targeted intervention. In the 2023 Task 3 paper, KBIR-inspec keyphrase extraction is combined with term statistics over lifestyle-oriented and science-oriented forum corpora to detect phrases that are likely complex for lay readers. For a term $t$, the paper defines
$$
idf_{lf}(t) = \log\left(\frac{N_{lf}}{df_{lf}(t)}\right), \quad idf_{sc}(t) = \log\left(\frac{N_{sc}}{df_{sc}(t)}\right),
$$
and for a phrase $\langle t_1,\ldots,t_n\rangle$,
$$
\phi(\langle t_1,\ldots,t_n\rangle) = \max_i \left[idf_{lf}(t_i) - idf_{sc}(t_i)\right].
$$
Phrases with $\phi \ge 0.01$ are tagged as complex and bracketed in the source text to guide the simplifier’s attention [2307.03569].

The 2025 THM system develops a related but stricter term-centric pipeline. Candidate keyphrases are extracted, idf is computed via PyTerrier over LOTTE lifestyle and science corpora, and a term is marked as complex if
$$
idf_{science}(t) > idf_{lifestyle}(t)
$$
and
$$
idf_{science}(t) \ge \tau,
$$
with $\tau = 0.1$. Marked terms are enclosed in square brackets, and LLMs are instructed to replace or explain only those bracketed terms while leaving the rest of the sentence’s structure largely unchanged. THM evaluates multiple prompt personas and models, including gpt-4.1-nano, gemini-2.0-flash, and gemini-2.5-flash-preview [2507.04414].

A third pattern is prompt-only simplification under strong format constraints versus fine-tuning. UM_FHS compares “no-context” prompt engineering with fine-tuned GPT-4.1-family models. In its formulation, no-context means the model receives only task prompts and the input text, with no retrieval or external context. The prompts encode rules such as targeting K8 readability, splitting long sentences, explaining or replacing jargon, omitting non-essential statistics, resolving pronouns, preserving facts, and adding nothing new. Fine-tuning is performed for gpt-4.1-mini and gpt-4.1-nano with epochs 3, batch size 1, learning-rate multiplier 2, and random seed 69517706 [2512.16541].

A fourth pattern, specific to Task 2, is multi-signal hallucination detection coupled with grounded post-editing. DS@GT constructs an ensemble framework that combines a BERT-based binary classifier, sentence-transformer cosine similarity, NLI features from `facebook/bart-large-mnli`, and LLM reasoning from `llama-3.3-70b-versatile`. In the sourced setting, maximum cosine similarity is taken across source chunks; in the post-hoc setting, dense passage retrieval over overlapping 100-word chunks with 50-word overlap retrieves the top-5 chunks using `multi-qa-MiniLM-L6-cos-v1`. These signals are aggregated by a three-layer neural meta-classifier. For grounded generation, the same LLM is used as a post-editor that revises baseline simplifications to remove fabricated content, contradictions, hallucinations, and overgeneralizations while remaining aligned to the provided source [2508.11823].

## 5. Empirical performance across tasks

Reported results indicate that SimpleText systems often improve readability and simplification metrics substantially over source baselines, but they do so with differing trade-offs.

For 2025 Task 1, DS@GT reports that its `plan_guided_llama` system on 37 aligned Cochrane-auto abstracts achieves SARI 42.33, BLEU 10.43, FKGL 7.77, Compression Ratio 0.48, Sentence Splits 0.97, Levenshtein Similarity 0.47, Exact Copies 0.00, Additions Proportion 0.18, Deletions Proportion 0.70, and Lexical Complexity Score 8.52. On 217 plain language summaries, the same system achieves SARI 42.98, BLEU 6.33, FKGL 7.82, Compression Ratio 0.48, Sentence Splits 0.99, Levenshtein Similarity 0.46, Exact Copies 0.00, Additions 0.18, Deletions 0.71, and Lexical Complexity 8.50. The same paper reports that plan-driven sentence simplification yields marginal improvements over a basic LLM simplification baseline in BLEU and FKGL while keeping SARI essentially unchanged [2508.11816].

On the document side, DS@GT’s `llama_summary_simplification` system scores SARI 40.32 on 37 aligned Cochrane-auto abstracts and 42.92 on 217 PLS. The paper interprets these results as strong simplification but also notes aggressive deletion proportions around 0.70–0.75 and lower BLEU/Levenshtein scores, indicating divergence from reference phrasing and possible semantic risk. This underlies the later use of grounded post-editing in Task 2.3 [2508.11816].

THM’s Task 1.1 experiments show that a term-centric prompt configuration can be competitive under SARI. The baseline with no changes scores SARI simple original 7.844 and SARI simple auto 12.033, whereas the best-performing configuration, `p2--gpt-4.1-nano`, reaches 39.572 and 41.315 respectively. The paper also reports that `p1--gpt-4.1-nano` obtains 38.238/40.416 and `pni1--gpt-4.1-nano` 35.262/37.262, with concise P2 prompts outperforming longer P1 prompts [2507.04414].

UM_FHS reports strong results for prompt-only GPT-4.1-family systems. On the 37-pair aligned sentence-level set, `gpt-4.1-mini` achieves SARI 43.34, BLEU 13.93, FKGL 7.46, and CR 0.78; on the 217-pair unaligned document-level set, the same model reaches SARI 42.13, BLEU 9.52, FKGL 7.56, and CR 0.74. The paper identifies `gpt-4.1-mini` as the most robust no-context model, while also noting a standout document-level fine-tuned case: `gpt-4.1-nano-ft` reaches SARI 43.61, BLEU 16.00, FKGL 10.63, and CR 0.50 on the 37 aligned abstracts. At the same time, `gpt-4.1-nano-ft` fails to produce usable sentence-level outputs under strict list-alignment constraints [2512.16541].

For 2025 Task 2.1, DS@GT reports near-identical F1 for sourced and post-hoc spurious detection. In the sourced setting with count 3,379, the ensemble attains Accuracy 0.91, Precision 0.93, Recall 0.97, F1 0.95, AUROC 0.68, and AUPRC 0.93. In the post-hoc setting with count 3,336, it attains Accuracy 0.90, Precision 0.93, Recall 0.97, F1 0.95, AUROC 0.64, and AUPRC 0.93. The same paper reports that BERT alone is already strong, the LLM judge is high-precision but conservative, NLI is weak in isolation, and the ensemble remains robust by balancing complementary signals [2508.11823].

For Task 2.2, the same system’s DeBERTa-plus-LLM meta-classifier yields the best reported scores among the compared baselines for all coarse error categories: “No error” F1 0.763, Fluency (A) F1 0.283, Alignment (B) F1 0.354, Information (C) F1 0.301, and Simplification (D) F1 0.374, with corresponding AUC-PR values 0.561, 0.133, 0.173, 0.156, and 0.224. For Task 2.3 grounded generation, the paper reports that grounded variants improve BLEU, Levenshtein similarity, and deletion behavior relative to the corresponding baselines, but often score lower in SARI, which it interprets as a faithfulness–simplicity trade-off [2508.11823].

The 2023 Task 3 results further illustrate that automatic readability improvement does not fully determine perceived quality. T5-based systems achieved stronger numerical gains on several readability measures, but manual evaluation oriented to end-user needs favored ChatGPT with complex phrase identification because it better balanced jargon reduction and content preservation. PEGASUS often copied the source nearly verbatim, while T5 sometimes produced hallucinated attributions such as “aaron carroll: ”, requiring post-hoc filtering in the `noaron` variant [2307.03569].

## 6. Recurring challenges, controversies, and significance

Several challenges recur across SimpleText work. The first is the tension between simplification and faithfulness. High deletion proportions, lower BLEU or Levenshtein similarity, and lower SARI under grounded post-editing are repeatedly interpreted as evidence that more aggressive simplification can threaten semantic fidelity, while stricter grounding can preserve content at the cost of less aggressive rewriting. This suggests that the lab’s slogan “and nothing more” is not only a task definition but also an evaluation problem [2508.11816] [2508.11823].

A second challenge is hallucination. The 2025 Task 2 framing makes explicit that creative generation includes fabrication, contradiction, misrepresentation, and even statements that are too general, trivial, or irrelevant in the context of the source. Earlier work also reports hallucination-like behavior in simplification systems, such as prompt-induced content leakage across batched inputs or recurrent spurious prefixes. The emergence of dedicated hallucination subtasks indicates that safe simplification of scientific text is increasingly treated as a factual consistency problem rather than only a readability problem [2307.03569] [2508.11823].

A third issue is metric adequacy. SARI is the official metric for 2025 Task 1, yet participant papers repeatedly qualify its interpretive limits. THM explicitly argues that SARI may reward outputs that stay closer to reference wording, and DS@GT’s Task 2.3 results show that grounded systems can improve semantic fidelity while sometimes losing SARI. A plausible implication is that future SimpleText evaluation will continue to rely on multi-metric analysis and, where possible, human assessment rather than any single scalar score [2507.04414] [2508.11823].

A fourth issue is controllability and reproducibility. Several papers note unreported decoding settings, absent human evaluation, or limited runtime detail. UM_FHS documents brittleness of fine-tuned models under rigid format constraints, while DS@GT notes that code/data availability is not specified and that specific training hyperparameters, inference times, and seeds are not detailed. These omissions do not negate the reported findings, but they constrain exact replication and make prompt design, format enforcement, and model-specific behavior central experimental variables [2512.16541] [2508.11823].

The lab’s broader significance lies in the convergence of simplification, controllable generation, and factuality assurance. Participant contributions collectively demonstrate plan-driven rewriting, summary-guided scaffolding, complex-term identification, prompt-only simplification, supervised fine-tuning, retrieval-augmented post-hoc checking, ensemble hallucination detection, and grounded LLM post-editing. Taken together, these approaches position SimpleText as an evaluation environment for studying how scientific texts can be simplified for non-experts without distorting evidence, omitting critical qualifiers, or introducing unsupported content [2508.11816] [2508.11823] [2307.03569].

Source: https://www.emergentmind.com/topics/simpletext-clef-lab