Encode, Tag, Realize: High-Precision Text Editing (1909.01187v1)

Abstract: We propose LaserTagger - a sequence tagging approach that casts text generation as a text editing task. Target texts are reconstructed from the inputs using three main edit operations: keeping a token, deleting it, and adding a phrase before the token. To predict the edit operations, we propose a novel model, which combines a BERT encoder with an autoregressive Transformer decoder. This approach is evaluated on English text on four tasks: sentence fusion, sentence splitting, abstractive summarization, and grammar correction. LaserTagger achieves new state-of-the-art results on three of these tasks, performs comparably to a set of strong seq2seq baselines with a large number of training examples, and outperforms them when the number of examples is limited. Furthermore, we show that at inference time tagging can be more than two orders of magnitude faster than comparable seq2seq models, making it more attractive for running in a live environment.

Encode, Tag, Realize: High-Precision Text Editing

The paper "Encode, Tag, Realize: High-Precision Text Editing" by Eric Malmi and colleagues introduces a novel approach to text generation by reframing it as a text editing task. By applying a sequence tagging methodology, the paper presents a system that reconstructs target texts from input sequences using three core edit operations: retention, deletion, and phrase insertion. Leveraging a hybrid architecture combining a BERT encoder with an autoregressive Transformer decoder, the approach is demonstrated on tasks including sentence fusion, sentence splitting, abstractive summarization, and grammatical error correction.

The work acknowledges the prevailing dominance of seq2seq models across text generation tasks. However, it criticizes their inefficiencies in scenarios where outputs largely overlap with inputs, an issue pertinent in tasks like sentence fusion and splitting. The proposed method instead operates with a reduced operational vocabulary, focusing on necessary edits to transform inputs into outputs. This confers advantages in training efficiency and model accuracy, particularly significant when labeled data is sparse.

Experimental results substantiate the efficacy of the method. Notably, the proposed system achieves state-of-the-art results in three out of four tasks, surpassing traditional seq2seq benchmarks in both cases of extensive and limited training data. The tagging model's inference speed also far exceeds that of comparable seq2seq approaches, being up to 100 times faster, making it highly suitable for real-time applications.

The results illustrate the potential for tagging models in contexts where accurate and efficient text editing is requisite. The holistic approach, which comprises encoding, tagging, and realizing, facilitates an intuitive yet rigorous process adaptable to various text generation challenges. This paper posits that by emphasizing the editing aspect over generation from scratch, significant improvements in computational efficiency and output precision can be achieved.

Future work could explore enhancing the model's flexibility in handling more complex reordering tasks or adapting the approach for morphologically rich languages, where textual adjustments might require more sophisticated operations. This research lays a foundational method that not only challenges the current seq2seq paradigms but also opens new avenues for research and development in text processing within NLP. The implications of adopting such a system are profound, particularly in contexts demanding reliable and agile text generation solutions, thereby contributing a substantial advancement in the theory and practice of text editing models.