Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation (2404.03278v1)

Published 4 Apr 2024 in cs.CL

Abstract: Text simplification intends to make a text easier to read while preserving its core meaning. Intuitively and as shown in previous works, these two dimensions (simplification and meaning preservation) are often-times inversely correlated. An overly conservative text will fail to simplify sufficiently, whereas extreme simplification will degrade meaning preservation. Yet, popular evaluation metrics either aggregate meaning preservation and simplification into a single score (SARI, LENS), or target meaning preservation alone (BERTScore, QuestEval). Moreover, these metrics usually require a set of references and most previous work has only focused on sentence-level simplification. In this paper, we focus on the evaluation of document-level text simplification and compare existing models using distinct metrics for meaning preservation and simplification. We leverage existing metrics from similar tasks and introduce a reference-less metric variant for simplicity, showing that models are mostly biased towards either simplification or meaning preservation, seldom performing well on both dimensions. Making use of the fact that the metrics we use are all reference-less, we also investigate the performance of existing models when applied to unseen data (where reference simplifications are unavailable).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Learning how to simplify from explicit labeling of complex-simplified text pairs. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 295–305, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  2. Cross-sentence transformations in text simplification. In Proceedings of the 2019 Workshop on Widening NLP, pages 181–184, Florence, Italy. Association for Computational Linguistics.
  3. The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification. Computational Linguistics, pages 1–29.
  4. Longformer: The long-document transformer.
  5. MOCHA: A dataset for training and evaluating generative reading comprehension metrics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6521–6532, Online. Association for Computational Linguistics.
  6. Discourse-based sentence splitting. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 261–273, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  7. Controllable sentence simplification via operation classification. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2091–2103, Seattle, United States. Association for Computational Linguistics.
  8. Context-aware document simplification. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13190–13206, Toronto, Canada. Association for Computational Linguistics.
  9. Document-level planning for text simplification. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 993–1006, Dubrovnik, Croatia. Association for Computational Linguistics.
  10. Simplicity level estimate (SLE): A learned reference-less metric for sentence simplification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12053–12059, Singapore. Association for Computational Linguistics.
  11. Evaluating factuality in text simplification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7331–7345, Dublin, Ireland. Association for Computational Linguistics.
  12. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055–5070, Online. Association for Computational Linguistics.
  13. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2587–2601, Seattle, United States. Association for Computational Linguistics.
  14. Entity-based semantic adequacy for data-to-text generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1530–1540, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  15. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  16. Dynamic multi-level multi-task learning for sentence simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 462–476, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  17. Neural CRF model for sentence alignment in text simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7943–7960, Online. Association for Computational Linguistics.
  18. Selecting proper lexical paraphrase for children. In Proceedings of the 25th Conference on Computational Linguistics and Speech Processing (ROCLING 2013), pages 59–73, Kaohsiung, Taiwan. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP).
  19. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch.
  20. Ffci: A framework for interpretable automatic evaluation of summarization. Journal of Artificial Intelligence Research, 73:1553–1607.
  21. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational Linguistics.
  22. Keep it simple: Unsupervised simplification of multi-paragraph text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6365–6378, Online. Association for Computational Linguistics.
  23. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177.
  24. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
  25. Towards document-level paraphrase generation with sentence rewriting and reordering. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1033–1044, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  26. Improving text simplification with factuality error detection. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 173–178, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
  27. Controllable sentence simplification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689–4698, Marseille, France. European Language Resources Association.
  28. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.
  29. Exploring the effects of sentence simplification on Hindi to English machine translation system. In Proceedings of the Workshop on Automatic Text Simplification - Methods and Applications in the Multilingual Society (ATS-MA 2014), pages 21–29, Dublin, Ireland. Association for Computational Linguistics and Dublin City University.
  30. Entity-focused sentence simplification for relation extraction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 788–796, Beijing, China. Coling 2010 Organizing Committee.
  31. Text simplification with reinforcement learning using supervised rewards on grammaticality, meaning preservation, and simplicity. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 153–159, Suzhou, China. Association for Computational Linguistics.
  32. Split and rephrase. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 606–616, Copenhagen, Denmark. Association for Computational Linguistics.
  33. A sentence simplification system for improving relation extraction. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 170–174, Osaka, Japan. The COLING 2016 Organizing Committee.
  34. Transforming complex sentences into a semantic hierarchy. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3415–3427, Florence, Italy. Association for Computational Linguistics.
  35. Exploring neural text simplification models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 85–91, Vancouver, Canada. Association for Computational Linguistics.
  36. Deep learning approaches to lexical simplification: A survey.
  37. A survey on lexical simplification. Journal of Artificial Intelligence Research, 60:549–593. © 2017 AI Access Foundation, Inc. This is an author produced version of a paper subsequently published in Journal of Artificial Intelligence Research. Uploaded in accordance with the publisher’s self-archiving policy.
  38. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812–4829, Online. Association for Computational Linguistics.
  39. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  40. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online. Association for Computational Linguistics.
  41. Max Schwarzer and David Kauchak. 2018. Human evaluation for text simplification: The simplicity-adequacy tradeoff. In SoCal NLP Symposium.
  42. Questeval: Summarization asks for fact-based evaluation.
  43. Rethinking automatic evaluation in sentence simplification.
  44. Advaith Siddharthan. 2003. Preserving discourse structure when simplifying text. In Proceedings of the 9th European Workshop on Natural Language Generation (ENLG-2003) at EACL 2003, Budapest, Hungary. Association for Computational Linguistics.
  45. Neha Srikanth and Junyi Jessy Li. 2021. Elaborative simplification: Content addition and explanation generation in text simplification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5123–5137, Online. Association for Computational Linguistics.
  46. Sanja Štajner and Maja Popovic. 2016. Can text simplification help machine translation? In Proceedings of the 19th Annual Conference of the European Association for Machine Translation, pages 230–242.
  47. Simple and effective text simplification using semantic and neural methods. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 162–173, Melbourne, Australia. Association for Computational Linguistics.
  48. Document-level text simplification: Dataset, criteria and baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7997–8013, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  49. On the helpfulness of document context to sentence simplification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1411–1423, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  50. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.
  51. Sentence simplification with memory-augmented neural networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 79–85, New Orleans, Louisiana. Association for Computational Linguistics.
  52. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics.
  53. Experiments with discourse-level choices and readability. In Proceedings of the 9th European Workshop on Natural Language Generation (ENLG-2003) at EACL 2003, Budapest, Hungary. Association for Computational Linguistics.
  54. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253–2263, Copenhagen, Denmark. Association for Computational Linguistics.
  55. Wikisimple: Automatic simplification of wikipedia articles. In AAAI.
  56. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297.
  57. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415.
  58. Controllable text simplification with deep reinforcement learning. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 398–404, Online only. Association for Computational Linguistics.
  59. Predicting sentence deletions for text simplification using a functional discourse structure. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 255–261, Dublin, Ireland. Association for Computational Linguistics.
  60. Bertscore: Evaluating text generation with BERT. CoRR, abs/1904.09675.
  61. Discourse level factors for sentence deletion in text simplification. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9709–9716.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Liam Cripwell (3 papers)
  2. Joël Legrand (6 papers)
  3. Claire Gardent (22 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now