BoschAI @ PLABA 2023: Leveraging Edit Operations in End-to-End Neural Sentence Simplification (2311.01907v2)
Abstract: Automatic simplification can help laypeople to comprehend complex scientific text. LLMs are frequently applied to this task by translating from complex to simple language. In this paper, we describe our system based on Llama 2, which ranked first in the PLABA shared task addressing the simplification of biomedical text. We find that the large portion of shared tokens between input and output leads to weak training signals and conservatively editing models. To mitigate these issues, we propose sentence-level and token-level loss weights. They give higher weight to modified tokens, indicated by edit distance and edit operations, respectively. We conduct an empirical evaluation on the PLABA dataset and find that both approaches lead to simplifications closer to those created by human annotators (+1.8% / +3.5% SARI), simpler language (-1 / -1.1 FKGL) and more edits (1.6x / 1.8x edit distance) compared to the same model fine-tuned with standard cross entropy. We furthermore show that the hyperparameter $\lambda$ in token-level loss weights can be used to control the edit distance and the simplicity level (FKGL).
- A dataset for plain language adaptation of biomedical abstracts. Scientific Data, 10(1):8.
- EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3393–3402, Florence, Italy. Association for Computational Linguistics.
- Rudolf Flesch. 1943. Marks of readable style; a study in adult education. Teachers College Contributions to Education.
- George D. Gopen and J. A. Swan. 1990. The Science of Scientific Writing. American Scientist.
- Automated Lay Language Summarization of Biomedical Scientific Reviews.
- A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 523–540, Dubrovnik, Croatia. Association for Computational Linguistics.
- LoRA: Low-Rank Adaptation of Large Language Models.
- Complexity-Weighted Loss and Diverse Reranking for Sentence Simplification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3137–3147, Minneapolis, Minnesota. Association for Computational Linguistics.
- Iterative Edit-Based Unsupervised Sentence Simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7918–7928, Online. Association for Computational Linguistics.
- Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet Physics Doklady, volume 10, pages 707–710. Soviet Union.
- Large Language Models and Control Mechanisms Improve Text Readability of Biomedical Abstracts.
- An Investigation into the Effect of Control Tokens on Text Simplification. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 154–165, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
- Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
- Controllable Sentence Simplification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689–4698, Marseille, France. European Language Resources Association.
- A survey of automated methods for biomedical text simplification. Journal of the American Medical Informatics Association, 29(11):1976–1988.
- Bleu: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
- Francesco Ronzano and Horacio Saggion. 2016. Knowledge Extraction and Modeling from Scientific Publications. In Semantics, Analytics, Visualization. Enhancing Scholarly Data, Lecture Notes in Computer Science, pages 11–25, Cham. Springer International Publishing.
- Llama 2: Open Foundation and Fine-Tuned Chat Models.
- Attention is all you need. Advances in neural information processing systems, 30.
- Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415.
- Weijia Xu and Marine Carpuat. 2021. EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints. Transactions of the Association for Computational Linguistics, 9:311–328.
- Controllable Text Simplification with Deep Reinforcement Learning. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 398–404, Online only. Association for Computational Linguistics.
- Lexically Constrained Decoding with Edit Operation Prediction for Controllable Text Simplification. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 147–153, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.