SimpleText CLEF Lab Overview

• SimpleText CLEF Lab is a recurring evaluation initiative focused on transforming dense scientific texts into accessible versions while preserving core meaning.
• It organizes both sentence- and document-level tasks with large, aligned datasets and uses metrics like SARI, BLEU, and FKGL to assess simplification quality.
• The lab employs diverse methodologies—including plan-driven rewriting, complex-term identification, prompt engineering, and ensemble hallucination detection—to balance readability and factual accuracy.

Searching arXiv for papers on the SimpleText CLEF Lab and related shared-task descriptions. SimpleText is a recurring lab within CLEF, the Conference and Labs of the Evaluation Forum, devoted to making scientific texts more accessible to non-expert readers through automatic simplification. Across recent editions, participant papers describe the lab as addressing not only sentence- and document-level rewriting, but also passage selection, difficult-concept explanation, and the detection and mitigation of hallucination in simplified scientific text. In the 2025 track, the problem framing is explicitly “Simplify Scientific Text (and Nothing More),” emphasizing simplification that preserves meaning, avoids unsupported additions, and remains faithful to source content (Engelmann et al., 2023, Hofmann et al., 6 Jul 2025, Marturi et al., 15 Aug 2025, Marturi et al., 15 Aug 2025).

1. Historical development and problem framing

Participant descriptions present SimpleText as a long-running evaluation initiative motivated by the mismatch between expert-oriented scientific writing and the needs of broader audiences. Scientific communication is characterized as dense, terminology-heavy, and often constrained by publication conventions, making it difficult for non-experts such as patients, policymakers, and family members of seriously ill patients to extract core findings without specialist background knowledge. In this framing, simplification is not merely shortening; it is the controlled reduction of lexical and syntactic complexity while preserving meaning, intent, and key information (Engelmann et al., 2023, Hofmann et al., 6 Jul 2025).

The lab structure has evolved across editions. In 2023, SimpleText ran three coordinated tasks: passage selection for simplified summaries, identification and explanation of difficult concepts, and scientific text simplification for non-experts. By contrast, the 2025 track narrows the scope back to simplification itself and refines the evaluation around sentence-level simplification, document-level simplification, and hallucination avoidance. This progression suggests an increasing emphasis on faithful rewriting rather than adjacent access tasks such as retrieval or concept explanation (Engelmann et al., 2023, Hofmann et al., 6 Jul 2025, Marturi et al., 15 Aug 2025).

A recurrent distinction in these papers is between simplification and summarization. Scientific text simplification is described as preserving the source’s key informational content while rephrasing for accessibility, whereas summarization alone may condense content more aggressively. The 2025 lab framing intensifies this distinction by explicitly penalizing creative generation that goes beyond or diverges from the source (Engelmann et al., 2023, Marturi et al., 15 Aug 2025).

2. Task organization, datasets, and input–output structure

The tasks described for recent SimpleText editions vary by year and by granularity of transformation.

Edition / task	Focus	Representative data described in participant papers
2023 Task 3	Simplification of scientific abstracts and sentences for non-experts	Large split with 152,072 source texts; 135,540 unique after de-duplication
2025 Task 1.1	Sentence-level simplification	9,160 short English texts in one description; 37 aligned Cochrane-auto abstracts in another evaluation setup
2025 Task 1.2	Document-level simplification	37 aligned abstract–PLS pairs and 217 abstract–PLS pairs
2025 Task 2	Hallucination detection and grounded generation	3,379 sourced instances, 3,336 post-hoc instances, and 217 plain language summaries for Task 2.3 evaluation

For 2025 Task 1, one participant paper states that the lab uses the Cochrane-Auto dataset, described as “an aligned dataset for the simplification of biomedical abstracts,” and evaluates both sentence-level and document-level simplification. Another paper reports an aligned subset of 37 abstract–PLS pairs, comprising 587 source sentences and 388 target sentences, alongside a larger unaligned set of 217 abstract–PLS pairs for document-level evaluation (Marturi et al., 15 Aug 2025, Kocbek et al., 18 Dec 2025).

Task 1.1 is presented as sentence-level simplification with strict alignment between input and output sentences. One description states that the number of output elements must exactly equal the number of input sentences, while allowing splitting within a list element and omission of non-essential material by outputting the empty string '' for that element. Task 1.2 is presented as document-level simplification of full abstracts into accessible summaries, with coherence and preservation of key content as central requirements (Kocbek et al., 18 Dec 2025).

The 2025 sentence-focused regime is also characterized as a “short-text” or minimal-context setting. One participant paper reports that the Task 1.1 dataset consists of 9,160 short English texts extracted from scientific publications, of which 9,086 are unique, with average length 168.66 characters. The same paper emphasizes that many instances are single sentences and that systems often must detect and simplify domain-specific terms with little or no external context (Hofmann et al., 6 Jul 2025).

Task 2 in 2025 addresses hallucination explicitly. DS@GT describes Task 2 as “Identify and Avoid Hallucination,” divided into three subtasks: document-level detection of creative generation, sentence-level detection and classification of information distortion errors, and grounded generation that avoids creative overgeneration by design. The task uses scientific abstracts and plain-language summaries, and distinguishes a sourced setting, where the source abstract is provided, from a post-hoc setting, where systems must retrieve proxy context (Marturi et al., 15 Aug 2025).

The 2023 Task 3 dataset differs markedly in scale and structure. It is described as comprising short segments from scientific publications, primarily single sentences, drawn from diverse domains such as computer forensics, molecular virology, and animal behavior analysis. The large split contains 152,072 source texts, reduced to 135,540 unique items after de-duplication. Reported data quality issues include duplicates, UTF-8 hexcodes embedded in text, incomplete sentences, and a small number of empty or punctuation-only texts (Engelmann et al., 2023).

3. Evaluation protocols and measurement regimes

SimpleText does not operate with a single invariant metric across all tasks. For 2025 Task 1, one paper explicitly states: “Only SARI is the official evaluation metric for Task 1. Other metrics are reported for supplementary analysis.” In that setting, SARI is described as comparing system output to both source and references through keep, delete, and add operations, and as being “computed as the average of F1 scores for these three operations,” although the formal equation is not reported in that paper (Marturi et al., 15 Aug 2025).

Other Task 1 evaluations report BLEU, BERTScore (F1), FKGL, Compression Ratio, Sentence Splits, Levenshtein Similarity, Exact Copies, Additions Proportion, Deletions Proportion, and Lexical Complexity Score. A separate 2025 paper gives the high-level formulation

$$\mathrm{SARI} = \frac{F_{\mathrm{add}} + F_{\mathrm{keep}} + F_{\mathrm{del}}}{3},$$

and reports

$$\mathrm{BLEU} = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right),$$

with brevity penalty defined piecewise, alongside

$$\mathrm{FKGL} = 0.39 \cdot \left(\frac{\text{words}}{\text{sentences}}\right) + 11.8 \cdot \left(\frac{\text{syllables}}{\text{word}}\right) - 15.59,$$

and a compression ratio defined as

$$\mathrm{CR} = \frac{\text{length(simplified)}}{\text{length(original)}}.$$

These formulations are used to characterize the simplification–readability trade-off in Task 1 systems (Kocbek et al., 18 Dec 2025).

For 2025 Task 2, the evaluation regime is different. Detection is measured with Accuracy, Precision, Recall, F1, AUROC, and AUPRC. The DS@GT paper reports the standard definitions

$$P = \frac{TP}{TP + FP}, \quad R = \frac{TP}{TP + FN}, \quad F1 = \frac{2PR}{P + R},$$

and uses AUROC to assess ranking quality across decision thresholds and AUPRC to emphasize performance on imbalanced classes. Grounded generation in Task 2.3 is evaluated with SARI, BLEU, FKGL, Compression Ratio, Sentence Splits, Levenshtein Similarity, Exact Copies, Additions/Deletions proportions, and Lexical Complexity Score (Marturi et al., 15 Aug 2025).

The 2023 Task 3 evaluation departs from reference-based simplification metrics because no gold simplifications were provided. Instead, the evaluation emphasized readability-oriented automatic measures together with manual inspection. The paper reports Flesch Reading Ease,

$$RE = 206.835 - 1.015 \times \left(\frac{\text{words}}{\text{sentences}}\right) - 84.6 \times \left(\frac{\text{syllables}}{\text{words}}\right),$$

New Dale–Chall, difficult word count, reading time,

$RT = 0.01469 \times \text{characters},$

syllable count, lexicon count, and sentence count. The authors explicitly note that they did not compute BLEU, ROUGE, or SARI, nor FKGL, because the setting lacked gold references and their focus was readability for non-experts (Engelmann et al., 2023).

An important methodological caveat appears in the 2025 Task 1.1 THM paper: SARI may favor outputs closer to reference wording and may therefore bias evaluation against broader but faithful clarity-oriented rephrasings. This is framed not as a definitive defect of the metric, but as an evaluation caveat observed in practice (Hofmann et al., 6 Jul 2025).

4. Core methodological paradigms

Recent SimpleText systems exhibit several recurring methodological patterns.

One pattern is plan-driven sentence simplification. DS@GT’s Task 1.1 system uses llama-3.3-70b-versatile in a two-stage pipeline in which the model first selects a simplification strategy from rephrase, delete, split, ignore, and merge, and then generates a simplified sentence conditioned on that plan. The inputs include the complex sentence, the full source document, and the next complex sentence for local context. The same paper describes document-level simplification as summary-guided: the model first produces “a clear and concise summary that captures the essential information, main arguments, and key findings” and then rewrites the original document in a simpler way while remaining faithful to both the source and the summary (Marturi et al., 15 Aug 2025).

A second pattern is complex-term identification followed by targeted intervention. In the 2023 Task 3 paper, KBIR-inspec keyphrase extraction is combined with term statistics over lifestyle-oriented and science-oriented forum corpora to detect phrases that are likely complex for lay readers. For a term $t$, the paper defines

$$idf_{lf}(t) = \log\left(\frac{N_{lf}}{df_{lf}(t)}\right), \quad idf_{sc}(t) = \log\left(\frac{N_{sc}}{df_{sc}(t)}\right),$$

and for a phrase $\langle t_1,\ldots,t_n\rangle$,

$$\mathrm{BLEU} = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right),$$0

Phrases with $$\mathrm{BLEU} = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right),$$1 are tagged as complex and bracketed in the source text to guide the simplifier’s attention (Engelmann et al., 2023).

The 2025 THM system develops a related but stricter term-centric pipeline. Candidate keyphrases are extracted, idf is computed via PyTerrier over LOTTE lifestyle and science corpora, and a term is marked as complex if

$$\mathrm{BLEU} = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right),$$2

and

$$\mathrm{BLEU} = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right),$$3

with $$\mathrm{BLEU} = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right),$$4. Marked terms are enclosed in square brackets, and LLMs are instructed to replace or explain only those bracketed terms while leaving the rest of the sentence’s structure largely unchanged. THM evaluates multiple prompt personas and models, including gpt-4.1-nano, gemini-2.0-flash, and gemini-2.5-flash-preview (Hofmann et al., 6 Jul 2025).

A third pattern is prompt-only simplification under strong format constraints versus fine-tuning. UM_FHS compares “no-context” prompt engineering with fine-tuned GPT-4.1-family models. In its formulation, no-context means the model receives only task prompts and the input text, with no retrieval or external context. The prompts encode rules such as targeting K8 readability, splitting long sentences, explaining or replacing jargon, omitting non-essential statistics, resolving pronouns, preserving facts, and adding nothing new. Fine-tuning is performed for gpt-4.1-mini and gpt-4.1-nano with epochs 3, batch size 1, learning-rate multiplier 2, and random seed 69517706 (Kocbek et al., 18 Dec 2025).

A fourth pattern, specific to Task 2, is multi-signal hallucination detection coupled with grounded post-editing. DS@GT constructs an ensemble framework that combines a BERT-based binary classifier, sentence-transformer cosine similarity, NLI features from facebook/bart-large-mnli, and LLM reasoning from llama-3.3-70b-versatile. In the sourced setting, maximum cosine similarity is taken across source chunks; in the post-hoc setting, dense passage retrieval over overlapping 100-word chunks with 50-word overlap retrieves the top-5 chunks using multi-qa-MiniLM-L6-cos-v1. These signals are aggregated by a three-layer neural meta-classifier. For grounded generation, the same LLM is used as a post-editor that revises baseline simplifications to remove fabricated content, contradictions, hallucinations, and overgeneralizations while remaining aligned to the provided source (Marturi et al., 15 Aug 2025).

5. Empirical performance across tasks

Reported results indicate that SimpleText systems often improve readability and simplification metrics substantially over source baselines, but they do so with differing trade-offs.

For 2025 Task 1, DS@GT reports that its plan_guided_llama system on 37 aligned Cochrane-auto abstracts achieves SARI 42.33, BLEU 10.43, FKGL 7.77, Compression Ratio 0.48, Sentence Splits 0.97, Levenshtein Similarity 0.47, Exact Copies 0.00, Additions Proportion 0.18, Deletions Proportion 0.70, and Lexical Complexity Score 8.52. On 217 plain language summaries, the same system achieves SARI 42.98, BLEU 6.33, FKGL 7.82, Compression Ratio 0.48, Sentence Splits 0.99, Levenshtein Similarity 0.46, Exact Copies 0.00, Additions 0.18, Deletions 0.71, and Lexical Complexity 8.50. The same paper reports that plan-driven sentence simplification yields marginal improvements over a basic LLM simplification baseline in BLEU and FKGL while keeping SARI essentially unchanged (Marturi et al., 15 Aug 2025).

On the document side, DS@GT’s llama_summary_simplification system scores SARI 40.32 on 37 aligned Cochrane-auto abstracts and 42.92 on 217 PLS. The paper interprets these results as strong simplification but also notes aggressive deletion proportions around 0.70–0.75 and lower BLEU/Levenshtein scores, indicating divergence from reference phrasing and possible semantic risk. This underlies the later use of grounded post-editing in Task 2.3 (Marturi et al., 15 Aug 2025).

THM’s Task 1.1 experiments show that a term-centric prompt configuration can be competitive under SARI. The baseline with no changes scores SARI simple original 7.844 and SARI simple auto 12.033, whereas the best-performing configuration, p2--gpt-4.1-nano, reaches 39.572 and 41.315 respectively. The paper also reports that p1--gpt-4.1-nano obtains 38.238/40.416 and pni1--gpt-4.1-nano 35.262/37.262, with concise P2 prompts outperforming longer P1 prompts (Hofmann et al., 6 Jul 2025).

UM_FHS reports strong results for prompt-only GPT-4.1-family systems. On the 37-pair aligned sentence-level set, gpt-4.1-mini achieves SARI 43.34, BLEU 13.93, FKGL 7.46, and CR 0.78; on the 217-pair unaligned document-level set, the same model reaches SARI 42.13, BLEU 9.52, FKGL 7.56, and CR 0.74. The paper identifies gpt-4.1-mini as the most robust no-context model, while also noting a standout document-level fine-tuned case: gpt-4.1-nano-ft reaches SARI 43.61, BLEU 16.00, FKGL 10.63, and CR 0.50 on the 37 aligned abstracts. At the same time, gpt-4.1-nano-ft fails to produce usable sentence-level outputs under strict list-alignment constraints (Kocbek et al., 18 Dec 2025).

For 2025 Task 2.1, DS@GT reports near-identical F1 for sourced and post-hoc spurious detection. In the sourced setting with count 3,379, the ensemble attains Accuracy 0.91, Precision 0.93, Recall 0.97, F1 0.95, AUROC 0.68, and AUPRC 0.93. In the post-hoc setting with count 3,336, it attains Accuracy 0.90, Precision 0.93, Recall 0.97, F1 0.95, AUROC 0.64, and AUPRC 0.93. The same paper reports that BERT alone is already strong, the LLM judge is high-precision but conservative, NLI is weak in isolation, and the ensemble remains robust by balancing complementary signals (Marturi et al., 15 Aug 2025).

For Task 2.2, the same system’s DeBERTa-plus-LLM meta-classifier yields the best reported scores among the compared baselines for all coarse error categories: “No error” F1 0.763, Fluency (A) F1 0.283, Alignment (B) F1 0.354, Information (C) F1 0.301, and Simplification (D) F1 0.374, with corresponding AUC-PR values 0.561, 0.133, 0.173, 0.156, and 0.224. For Task 2.3 grounded generation, the paper reports that grounded variants improve BLEU, Levenshtein similarity, and deletion behavior relative to the corresponding baselines, but often score lower in SARI, which it interprets as a faithfulness–simplicity trade-off (Marturi et al., 15 Aug 2025).

The 2023 Task 3 results further illustrate that automatic readability improvement does not fully determine perceived quality. T5-based systems achieved stronger numerical gains on several readability measures, but manual evaluation oriented to end-user needs favored ChatGPT with complex phrase identification because it better balanced jargon reduction and content preservation. PEGASUS often copied the source nearly verbatim, while T5 sometimes produced hallucinated attributions such as “aaron carroll: ”, requiring post-hoc filtering in the noaron variant (Engelmann et al., 2023).

6. Recurring challenges, controversies, and significance

Several challenges recur across SimpleText work. The first is the tension between simplification and faithfulness. High deletion proportions, lower BLEU or Levenshtein similarity, and lower SARI under grounded post-editing are repeatedly interpreted as evidence that more aggressive simplification can threaten semantic fidelity, while stricter grounding can preserve content at the cost of less aggressive rewriting. This suggests that the lab’s slogan “and nothing more” is not only a task definition but also an evaluation problem (Marturi et al., 15 Aug 2025, Marturi et al., 15 Aug 2025).

A second challenge is hallucination. The 2025 Task 2 framing makes explicit that creative generation includes fabrication, contradiction, misrepresentation, and even statements that are too general, trivial, or irrelevant in the context of the source. Earlier work also reports hallucination-like behavior in simplification systems, such as prompt-induced content leakage across batched inputs or recurrent spurious prefixes. The emergence of dedicated hallucination subtasks indicates that safe simplification of scientific text is increasingly treated as a factual consistency problem rather than only a readability problem (Engelmann et al., 2023, Marturi et al., 15 Aug 2025).

A third issue is metric adequacy. SARI is the official metric for 2025 Task 1, yet participant papers repeatedly qualify its interpretive limits. THM explicitly argues that SARI may reward outputs that stay closer to reference wording, and DS@GT’s Task 2.3 results show that grounded systems can improve semantic fidelity while sometimes losing SARI. A plausible implication is that future SimpleText evaluation will continue to rely on multi-metric analysis and, where possible, human assessment rather than any single scalar score (Hofmann et al., 6 Jul 2025, Marturi et al., 15 Aug 2025).

A fourth issue is controllability and reproducibility. Several papers note unreported decoding settings, absent human evaluation, or limited runtime detail. UM_FHS documents brittleness of fine-tuned models under rigid format constraints, while DS@GT notes that code/data availability is not specified and that specific training hyperparameters, inference times, and seeds are not detailed. These omissions do not negate the reported findings, but they constrain exact replication and make prompt design, format enforcement, and model-specific behavior central experimental variables (Kocbek et al., 18 Dec 2025, Marturi et al., 15 Aug 2025).

The lab’s broader significance lies in the convergence of simplification, controllable generation, and factuality assurance. Participant contributions collectively demonstrate plan-driven rewriting, summary-guided scaffolding, complex-term identification, prompt-only simplification, supervised fine-tuning, retrieval-augmented post-hoc checking, ensemble hallucination detection, and grounded LLM post-editing. Taken together, these approaches position SimpleText as an evaluation environment for studying how scientific texts can be simplified for non-experts without distorting evidence, omitting critical qualifiers, or introducing unsupported content (Marturi et al., 15 Aug 2025, Marturi et al., 15 Aug 2025, Engelmann et al., 2023).