---
title: Text Simplification by Tagging
url: https://www.emergentmind.com/topics/text-simplification-by-tagging-tst
type: topic
---

# Text Simplification by Tagging

to=arxiv_search  彩神争霸怎么json
{"query":"\"Text Simplification by Tagging\" arXiv 2021 GECToR ACCESS ASSET", "max_results": 10}
to=arxiv_search  玩大发快三json
{"query":"ti:\"Text Simplification by Tagging\"", "max_results": 5}
to=arxiv_search  天天中彩票中大奖json
{"query":"ti:\"A Survey on Text Simplification\"", "max_results": 5}
Text Simplification by Tagging (TST) is an edit-based approach to text simplification that treats rewriting not as free-form generation but as sequence tagging over the input sentence. In the broader text simplification literature, text simplification aims to reduce the linguistic complexity of content to make it easier to understand, and the field has shifted from manual, hand-crafted rules to automated simplification, with a clear movement toward deep learning techniques and methods that address the lack of parallel data [2008.08612]. Within that landscape, TST presents a simple and efficient text simplification system based on sequence Tagging, leveraging pre-trained Transformer-based encoders; it is fully non-autoregressive, provides more control over the outputs, and its best model achieves near state-of-the-art performance on benchmark test datasets while attaining faster inference speeds by over 11 times than the current state-of-the-art text simplification system [2103.05070].

## 1. Position within text simplification research

Text simplification is commonly organized around lexical, syntactic, and semantic or discourse-oriented changes. Typical human simplifications involve lexical changes, such as simpler words or paraphrases; syntactic changes, such as splitting, reordering, and making structure less nested; and content selection, such as deleting redundant or non-essential information. Good simplification output is therefore expected to be fluent, adequate in preserving core meaning, and simpler in the sense of shorter length, easier vocabulary, or lower reading level [2103.05070].

Historically and in modern work, text simplification is typically organized into several main paradigms: rule-based systems, statistical machine translation-based systems, neural sequence-to-sequence systems, Transformer-based or pre-trained language model systems, controllable or style-conditioned systems, and edit-based or tagging-based systems [2008.08612]. Rule-based approaches rely on linguistically motivated rules and handcrafted heuristics; SMT-based approaches formulate simplification as monolingual machine translation; neural seq2seq models maximize the conditional likelihood $p(y \mid x)$ and generate the simplified sentence token by token; and controllable systems often augment the input with control tokens such as target reading level or output length [2008.08612].

TST occupies the edit-based end of this continuum. The underlying motivation is that, in monolingual editing tasks, the input and output share a high lexical overlap; the target text is often a lightly edited version of the source rather than a completely new sentence. Standard seq2seq models are described as data-hungry, slow at inference because they use an autoregressive decoder, and limited in controllability and interpretability. Tagging-based models instead predict, for each input token, a discrete edit operation, thereby reducing the search space, exposing the edit decisions explicitly, and enabling non-autoregressive architectures in which all tags can be predicted in parallel [2103.05070].

## 2. Tagging formulation and edit inventory

TST is an iterative sequence-tagging system. Given an input sentence
$$
x = [x_1, x_2, \dots, x_N]
$$
and a gold simplified sentence
$$
y = [y_1, y_2, \dots, y_M],
$$
the model predicts a sequence of tags
$$
t = [t_1, t_2, \dots, t_N], \quad t_i \in \mathcal{T},
$$
such that applying the edit implied by $t_i$ at position $i$ to $x_i$ reconstructs $y$. The paper notes that $M \le N$ because deletions are common [2103.05070].

The tag vocabulary $\mathcal{T}$ is exactly the GECToR tag set. Its size is $5000$ edit tags, consisting of $4971$ basic tags and $29$ specialized GEC transform tags. The basic tags include token-independent operations `KEEP` and `DELETE`, and token-dependent operations `APPEND_w` and `REPLACE_w`. The specialized tags encode grammatical transformations such as `$TRANSFORM_VERB_VB_VBZ` [2103.05070].

| Tag family | Operation | Example |
|---|---|---|
| Token-independent | Preserve or remove a token | `KEEP`, `DELETE` |
| Token-dependent | Insert or substitute a word | `$APPEND_just`, `$REPLACE_really` |
| Specialized transform | Apply a grammatical transformation | `$TRANSFORM_VERB_VB_VBZ` |

This design is inherited from grammatical error correction rather than built specifically for simplification. The empirical justification is that the tag distribution in GEC and text simplification has 92.64% overlap, which the paper takes as evidence that the GEC tag inventory is highly suitable for simplification [2103.05070]. A common misconception is that a tagging framework can express only deletion-heavy rewriting. TST’s tag set explicitly supports preservation, deletion, lexical substitution, appending, and morphological transformation, although the paper also notes that the model cannot perform arbitrary generation and that this is a trade-off between sample efficiency and generative flexibility [2103.05070].

## 3. Encoder architecture, iterative editing, and inference mechanics

The architecture is the GECToR model with a RoBERTa-base encoder. RoBERTa-base, a 12-layer bidirectional Transformer, serves purely as an encoder; there is no Transformer decoder. Tokenization uses BPE via HuggingFace tokenizers over the whole sentence, which the paper describes as more faithful to RoBERTa than earlier per-word tokenizers in GECToR [2103.05070].

On top of the encoder, TST adds two parallel feed-forward layers per token. The edit-detection layer predicts the probability that any edit should happen to a token position, yielding a scalar $p_{\text{edit}}(i)$. The edit-classification layer predicts a distribution over tags,
$$
p(t_i \mid x) = \text{Softmax}(W h_i + b),
$$
where $h_i$ is the contextual representation for token $i$. The model factorizes the tag prediction as
$$
p(t \mid x) = \prod_{i=1}^N p(t_i \mid x),
$$
so tag predictions are conditionally independent given the encoder output [2103.05070].

Inference is gated by a minimum edit probability threshold $\epsilon$. If $p_{\text{edit}}(i) < \epsilon$, the classification head’s suggestion is ignored and the token is treated as `KEEP`. This mechanism is intended to improve precision by discarding low-confidence edits [2103.05070]. The model also introduces confidence biases for `KEEP` and `DELETE`, the two most frequent and token-independent operations. Increasing the `DELETE` bias makes deletions more likely and therefore tends to produce shorter, simpler sentences; increasing the `KEEP` bias yields more conservative outputs. The number of iterations, the `KEEP` and `DELETE` biases, and the minimum edit probability $\epsilon$ collectively provide operational control over the trade-off between simplicity and adequacy or fluency [2103.05070].

TST is fully non-autoregressive in the sense that, within each iteration, all tag predictions are made in parallel in a single forward pass. The procedure is iterative rather than single-pass: the model starts with the original sentence, predicts a tag sequence, applies the edits, and repeats this process for a small fixed number of iterations, with experiments ranging from 1 to 5 passes. This clarifies another common misunderstanding: fully non-autoregressive does not mean only one pass over the sentence; rather, it means that there is no token-by-token autoregressive decoder or beam-search decoding loop [2103.05070].

## 4. Corpora, preprocessing, augmentation, and optimization

The core training data are WikiSmall and WikiLarge, collectively denoted WikiAll. WikiSmall contains approximately 88k training pairs and 3.9M tokens, together with 205 validation sentences and 100 test sentences in the standard split from Zhang and Lapata (2017). WikiLarge contains 296k training pairs and 11.7M tokens. For evaluation, the paper uses TurkCorpus, with 2000 development and 359 test complex sentences and 8 crowd-sourced reference simplifications per sentence; ASSET, with the same complex sentences as TurkCorpus and 10 references per sentence; and the WikiSmall test set of 100 complex sentences [2103.05070].

A specific preprocessing step removes the special markers `-LRB-` and `-RRB-` and all tokens between them from both source and target sentences. The paper reports that this filtering improves SARI by +0.8 points on average and reduces FKGL, thereby making the text simpler [2103.05070]. More broadly, the survey context is that simplification corpora are relatively small compared to typical MT corpora and often noisy, which has encouraged work on weak supervision, pseudo-parallel data, transfer learning, and architectures whose label space is simpler than full free-form generation [2008.08612].

Two model initializations are distinguished. TST-BASE uses a RoBERTa-base encoder with randomly initialized feed-forward heads and is trained only on text simplification data. TST-GEC uses the same architecture but is initialized from a GEC-trained GECToR model, so both the RoBERTa weights and the feed-forward heads are first trained on large GEC corpora and then fine-tuned on simplification [2103.05070].

The paper emphasizes simple data augmentation. In WikiBT, for each $(x,y)$ pair in WikiAll, the simplified sentence $y$ is translated from English to French or German and then back to English using pretrained Marian NMT models from Helsinki-NLP/OPUS via HuggingFace, with EN→FR, FR→EN, EN→DE, and DE→EN 6-layer Transformer encoder-decoder models. This effectively triples the training data. In WikiEns, three teacher models are trained—TST on WikiAll, TST-GEC on WikiAll, and TST on WikiAll + WikiBT—and their predicted class probabilities over tags are averaged tokenwise; the argmax averaged per-class probability yields a distilled tag sequence, which is converted into a new reference. The final student model, TST-FINAL, is trained on WikiAll + WikiEns. The paper reports that WikiEns alone is more beneficial than WikiBT alone, with +1.2 versus +0.4 SARI, and that using WikiEns and WikiBT together is worse than using WikiEns + WikiAll; the final system therefore uses WikiEns + WikiAll [2103.05070].

Training is implemented with AllenNLP and HuggingFace Transformers. The optimizer is Adam with an initial learning rate of $1 \times 10^{-5}$. The learning rate is multiplied by 0.1 when validation loss does not improve for 10 epochs. Training runs for up to 50 epochs, RoBERTa weights are frozen for the first 2 epochs, and early stopping is applied if development performance does not improve for 3 epochs [2103.05070].

## 5. Evaluation protocol, benchmark performance, and runtime

The evaluation uses SARI and FKGL. SARI compares system output against both the source and multiple reference simplifications and decomposes into F1 scores for ADD, DELETE, and KEEP operations over n-grams with $n=1..4$; overall SARI is the average of these F1 scores across n-grams and operations. FKGL measures readability from words per sentence and syllables per word, with lower values indicating simpler text. BLEU is not used because it correlates poorly with simplicity and can penalize valid simplifications [2103.05070].

| Dataset | System | Key results |
|---|---|---|
| TurkCorpus | TST-BASE | SARI $39.17 \pm 0.77$, FKGL $8.08 \pm 0.31$ |
| TurkCorpus | TST-FINAL | SARI $40.44 \pm 0.44$, FKGL $7.87 \pm 0.19$ |
| ASSET | TST-BASE | SARI $37.4 \pm 1.62$, FKGL $8.08 \pm 0.31$ |
| ASSET | TST-FINAL | SARI $43.21 \pm 0.3$, FKGL $6.87 \pm 0.27$ |
| WikiSmall | TST-FINAL | SARI $44.67 \pm 1.26$, FKGL $9.29 \pm 0.9$ |

On TurkCorpus, the reference baseline has SARI approximately 40.02 and FKGL approximately 8.77. TST-BASE attains ADD $3.62 \pm 0.41$, DELETE $41.61 \pm 3.14$, and KEEP $72.29 \pm 1.45$, while TST-FINAL attains ADD $6.96 \pm 0.44$, DELETE $47.87 \pm 0.75$, and KEEP $69.56 \pm 1.19$. Relative to Martin et al. (2020b), reported at SARI 42.53, TST-FINAL is within about 2 SARI points, achieves the highest reported DELETE F1 of 47.87, and reduces FKGL compared to many prior works [2103.05070].

On ASSET, the reference baseline has SARI $44.89 \pm 0.90$ and FKGL $6.49 \pm 0.42$. TST-FINAL attains ADD $8.04 \pm 0.29$, DELETE $64.25 \pm 1.22$, and KEEP $57.35 \pm 1.68$. Compared to Martin et al. (2020b), reported at SARI 44.15, TST-FINAL is within about 0.9 SARI points, has very strong DELETE F1 of 64.25, and obtains FKGL close to the reference baseline and more than 1 point better than TST-BASE [2103.05070].

On WikiSmall, TST-FINAL achieves a new state-of-the-art SARI, significantly surpassing the previous best cited in the paper, Zhao et al. (2020b) with SARI 36.92. The paper characterizes the overall pattern as near-SOTA on TurkCorpus and ASSET, clear SOTA on WikiSmall, and especially strong performance in ADD and DELETE, with lower KEEP scores than earlier models, indicating that the model edits rather than mostly copying [2103.05070].

Runtime is a central result. On the TurkCorpus test set, with batch size 128 on an NVIDIA Tesla V100 and averaged over 100 repetitions, BART with beam size 8 requires 2.82 s per batch, ACCESS with beam size 8 requires 1.43 s per batch, and TST requires 0.43 s with 5 iterations, 0.39 s with 4, 0.33 s with 3, 0.24 s with 2, and 0.13 s with 1. Under the typical setting of 2 iterations, TST is approximately 5.96 times faster than ACCESS with beam size 8 and approximately 11.75 times faster than BART with beam size 8. The paper attributes this speedup directly to the non-autoregressive tagging architecture and the absence of a beam-search decoder [2103.05070].

## 6. Related methods, controllability, error profile, and limitations

TST is part of a line of edit-based approaches across simplification and related monolingual transduction tasks. The paper situates it relative to Alva-Manchego et al. (2017), described as the first to formulate simplification as sequence tagging; Ribeiro et al. (2018), which used local string transduction as sequence labeling; LaserTagger (Malmi et al., 2019), which combines a BERT encoder with an autoregressive Transformer decoder predicting tags; EditNTS (Dong et al., 2019), a neural programmer-interpreter for simplification with explicit edit operations; PIE (Awasthi et al., 2019), a parallel iterative edit model for GEC; Felix (Mallinson et al., 2020), which separates tagging and insertion; and the Levenshtein Transformer (Gu et al., 2019), which performs iterative deletion and insertion actions [2103.05070]. TST’s distinguishing features are a fully non-autoregressive tagging architecture, direct reuse of a GEC-trained tag inventory and model, and strong performance with simple augmentations.

The model’s controllability follows from its explicit edit operations. The paper highlights three levers: `KEEP` and `DELETE` confidence biases, the minimum edit probability $\epsilon$, and the number of iterations. Increasing the `DELETE` bias produces more deletions and shorter sentences; increasing the `KEEP` bias produces more conservative edits; a larger $\epsilon$ enforces higher-confidence edits; and more iterations allow multiple rounds of refinement. The paper does not implement explicit reading-level or length constraints like ACCESS, but it presents these mechanisms as practical control over simplification aggressiveness [2103.05070]. A plausible implication is that tagging-based simplification offers an intermediate point between rule-based interpretability and seq2seq flexibility, a connection that is also consistent with the survey’s broader organization of controllable and edit-based text simplification methods [2008.08612].

Qualitative analysis shows both the strengths and the failure modes of the approach. The paper gives examples of lexical simplification and phrasing changes, structure simplification and deletion, and lexical simplification of domain terms. It also reports error types such as ungrammatical outputs, meaning changes, over-simplification or deletion, and numerical or factual inaccuracies. One example changes “the five dravidian languages” into “the three dravidian languages,” illustrating that meaning can shift. The paper notes that similar issues exist in human references as well, but it does not report a separate human evaluation study of fluency, adequacy, and simplicity [2103.05070].

The limitations are explicit. TST is not fully state of the art in SARI on all benchmarks; the authors intentionally limit themselves to RoBERTa-base and WikiAll plus derived data for simplicity and reproducibility; the tag set, while large, cannot support arbitrary generation; and datasets such as WikiAll are noisy and Wikipedia-specific. The paper also notes that Newsela is not used because of licensing constraints. Suggested future directions include larger Transformer encoders such as RoBERTa-large, ensembling, incorporation of external data, adaptation to broader monolingual editing tasks such as style transfer, sentence fusion, and summarization, and better control mechanisms over degree of simplification, length, or reading level [2103.05070].

In application terms, the paper emphasizes accessibility: simplification can help children, non-native speakers, and people with aphasia, dyslexia, autism, or low literacy, and it can also serve as a preprocessing step for downstream tasks such as parsing, summarization, semantic role labeling, and machine translation. At the same time, the risks identified for simplification systems remain applicable: biased content selection or lexical choice, meaning change in high-stakes settings, and the possibility of oversimplifying nuanced content [2103.05070].

Source: https://www.emergentmind.com/topics/text-simplification-by-tagging-tst