Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Digital Comprehensibility Assessment of Simplified Texts among Persons with Intellectual Disabilities (2402.13094v1)

Published 20 Feb 2024 in cs.CL and cs.HC

Abstract: Text simplification refers to the process of increasing the comprehensibility of texts. Automatic text simplification models are most commonly evaluated by experts or crowdworkers instead of the primary target groups of simplified texts, such as persons with intellectual disabilities. We conducted an evaluation study of text comprehensibility including participants with and without intellectual disabilities reading unsimplified, automatically and manually simplified German texts on a tablet computer. We explored four different approaches to measuring comprehensibility: multiple-choice comprehension questions, perceived difficulty ratings, response time, and reading speed. The results revealed significant variations in these measurements, depending on the reader group and whether the text had undergone automatic or manual simplification. For the target group of persons with intellectual disabilities, comprehension questions emerged as the most reliable measure, while analyzing reading speed provided valuable insights into participants' reading behavior.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Aisha Al-Sulaimi and Hafedh Al-Shihi. 2017. The effects of reading mode (digital vs printed text) on reading comprehension: A literature review of the key assessment factors. In 2017 6th International Conference on Information and Communication Technology and Accessibility (ICTA). 1–6. https://doi.org/10.1109/ICTA.2017.8336053
  2. Suha S. Al-Thanyyan and Aqil M. Azmi. 2021. Automated Text Simplification: A Survey. ACM Comput. Surv. 54, 2, Article 43 (mar 2021), 36 pages. https://doi.org/10.1145/3442695
  3. Comparison of Methods for Evaluating Complexity of Simplified Texts among Deaf and Hard-of-Hearing Adults at Different Literacy Levels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12.
  4. Data-Driven Sentence Simplification: Survey and Benchmark. Computational Linguistics 46, 1 (2020), 135–187. https://doi.org/10.1162/coli_a_00370
  5. The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification. Computational Linguistics 47, 4 (Dec. 2021), 861–889. https://doi.org/10.1162/coli_a_00418
  6. Evaluation of simple visual reaction time of different colored light stimuli in visually normal students. Clinical Optometry (2019), 167–171.
  7. James A Arnett and Seth S Labovitz. 1995. Effect of physical layout in performance of the Trail Making Test. Psychological Assessment 7, 2 (1995), 220.
  8. Bettina M. Bock. 2014. “Leichte Sprache”: Abgrenzung, Beschreibung und Problemstellungen aus Sicht der Linguistik. Frank & Timme, Berlin, 17–51.
  9. Bettina M. Bock and Daisy Lange. 2017. Empirische Untersuchungen zu Satz- und Textverstehen bei Menschen mit geistiger Behinderung und funktionalen Analphabeten. Frank & Timme, Berlin, 253–274.
  10. Accessibility of Different Natural User Interfaces for People with Intellectual Disabilities. In 2020 International Conference on Cyberworlds (CW). 211–218. https://doi.org/10.1109/CW49994.2020.00041
  11. Capito. 2023. Easy Language: What is it and why is it important? https://capito.eu/en/easy-language/ Accessed: 2023-12-07.
  12. Stan: A Probabilistic Programming Language. Journal of Statistical Software 76, 1 (2017), 1–32. https://doi.org/10.18637/jss.v076.i01
  13. Dar-Wei Chen and Richard Catrambone. 2015. Paper vs. Screen: Effects on Reading Comprehension, Metacognition, and Reader Behavior. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 59, 1 (2015), 332–336. https://doi.org/10.1177/1541931215591069 arXiv:https://doi.org/10.1177/1541931215591069
  14. Council of Europe. 2020. Common European framework of reference for languages: Learning, teaching, assessment. Companion volume. Council of Europe Publishing, Strasbourg.
  15. What’s so Simple about Simplified Texts? A Computational and Psycholinguistic Investigation of Text Comprehension and Text Processing. Reading in a Foreign Language 26, 1 (2014), 92–113.
  16. The Estimation of Item Response Models with the lmer Function from the lme4 Package in R. Journal of Statistical Software 39, 12 (2011), 1–28. https://doi.org/10.18637/jss.v039.i12
  17. Mary Dyson and Mark Haselgrove. 2000. The effects of reading speed and reading patterns on the understanding of text read from screen. Journal of Research in Reading 23, 2 (2000), 210–223. https://doi.org/10.1111/1467-9817.00115 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/1467-9817.00115
  18. MARY C. DYSON and MARK HASELGROVE. 2001. The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies 54, 4 (2001), 585–612. https://doi.org/10.1006/ijhc.2001.0458
  19. Easy-to-read texts for students with intellectual disability: linguistic factors affecting comprehension. Journal of applied research in intellectual disabilities 27, 3 (2014), 212–225.
  20. Jean-Paul Fox. 2010. Bayesian Item Response Modeling: Theory and Applications. Springer New York, New York, NY. https://doi.org/10.1007/978-1-4419-0742-4
  21. Furong Gao and Lisue Chen. 2005. Bayesian or Non-Bayesian: A Comparison Study of Item Parameter Estimation in the Three-Parameter Logistic Model. Applied Measurement in Education 18, 4 (oct 2005), 351–380. https://doi.org/10.1207/s15324818ame1804_2
  22. Sian Gooding. 2022. On the Ethical Considerations of Text Simplification. In Ninth Workshop on Speech and Language Processing for Assistive Technologies (SLPAT-2022). Association for Computational Linguistics, Dublin, Ireland, 50–57. https://doi.org/10.18653/v1/2022.slpat-1.7
  23. Predicting Text Readability from Scrolling Interactions. In Proceedings of the 25th Conference on Computational Natural Language Learning. Association for Computational Linguistics, Online, 380–390. https://doi.org/10.18653/v1/2021.conll-1.30
  24. Word Complexity is in the Eye of the Beholder. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 4439–4449. https://doi.org/10.18653/v1/2021.naacl-main.351
  25. Comparing evaluation techniques for text readability software for adults with intellectual disabilities. In Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility. 3–10.
  26. The unique contribution of working memory, inhibition, cognitive flexibility, and intelligence to reading comprehension and reading speed. Child Neuropsychology 26, 3 (2020), 324–344. https://doi.org/10.1080/09297049.2019.1649381 PMID: 31380706.
  27. Assessing the reading comprehension of adults with learning disabilities. Journal of Intellectual Disability Research 50, 6 (2006), 410–418.
  28. BLESS: Benchmarking Large Language Models on Sentence Simplification. arXiv:2310.15773 [cs.CL]
  29. Hak Joon Kim and Joan Kim. 2013. Reading from an LCD monitor versus paper: Teenagers’ reading performance. International Journal of Research Studies in Educational Technology 2, 1 (April 2013). https://doi.org/10.5861/ijrset.2012.170
  30. Comparison of reading performance on screen and on paper: A meta-analysis. Computers & Education 123 (2018), 138–149.
  31. ArviZ a unified library for exploratory analysis of Bayesian models in Python. Journal of Open Source Software 4, 33 (2019), 1143. https://doi.org/10.21105/joss.01143
  32. Keep It Simple: Unsupervised Simplification of Multi-Paragraph Text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 6365–6378. https://doi.org/10.18653/v1/2021.acl-long.498
  33. User evaluation of the effects of a text simplification algorithm using term familiarity on perception, understanding, learning, and information retention. Journal of medical Internet research 15, 7 (2013), e144.
  34. John Michael Linacre. 1989. Many-faceted Rasch measurement. Ph. D. Dissertation. The University of Chicago.
  35. Multilingual Denoising Pre-training for Neural Machine Translation. Transactions of the Association for Computational Linguistics 8 (2020), 726–742. https://doi.org/10.1162/tacl_a_00343
  36. Christiane Maaß. 2020. Easy Language–Plain Language–Easy Language Plus: Balancing comprehensibility and acceptability. Frank & Timme.
  37. LENS: A Learnable Evaluation Metric for Text Simplification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 16383–16408. https://doi.org/10.18653/v1/2023.acl-long.905
  38. Zero-Shot Crosslingual Sentence Simplification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 5109–5126. https://doi.org/10.18653/v1/2020.emnlp-main.415
  39. MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 1651–1664. https://aclanthology.org/2022.lrec-1.176
  40. Older adults and ICT adoption: Analysis of the use and attitudes toward computers in elderly Spanish people. Computers in Human Behavior 110 (2020), 106377. https://doi.org/10.1016/j.chb.2020.106377
  41. David E. Meyer and Roger W. Schvaneveldt. 1971. Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations. Journal of Experimental Psychology 90, 2 (Oct. 1971), 227–234. https://doi.org/10.1037/h0031564
  42. Do Individuals with Autism Change Their Reading Behavior to Adapt to Errors in the Text? Journal of Autism and Developmental Disorders 49, 10 (July 2019), 4232–4243. https://doi.org/10.1007/s10803-019-04108-8
  43. Sandra Pappert and Bettina M Bock. 2019. Easy-to-read German put to the test: Do adults with intellectual disability or functional illiteracy benefit from compound segmentation? Reading and Writing (2019), 1–27.
  44. Information and communication technology use in daily life among young adults with mild-to-moderate intellectual disability. Journal of Intellectual Disabilities 24 (2018), 289–308.
  45. Ralph Reitan and Deborah Wolfson. 1993. The Halstead-Reitan neuropsychological test battery: Theory and clinical interpretation. Neuropsychology Press.
  46. Simplify or help? Text simplification strategies for people with dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility. 1–10.
  47. pystan. PyPI.
  48. A New Dataset and Efficient Baselines for Document-level Text Simplification in German. In Proceedings of the Third Workshop on New Frontiers in Summarization. Association for Computational Linguistics, Online and in Dominican Republic, 152–161. https://doi.org/10.18653/v1/2021.newsum-1.16
  49. Revisiting non-English Text Simplification: A Unified Multilingual Benchmark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Toronto, Canada, 4898–4927. https://doi.org/10.18653/v1/2023.acl-long.269
  50. Making It Simplext: Implementation and Evaluation of a Text Simplification System for Spanish. ACM Trans. Access. Comput. 6, 4, Article 14 (may 2015), 36 pages. https://doi.org/10.1145/2738046
  51. Meredith Saletta and Jennifer Winberg. 2019. Leveled Texts for Adults With Intellectual or Developmental Disabilities: A Pilot Study. Focus on Autism and Other Developmental Disabilities 34, 2 (2019), 118–127. https://doi.org/10.1177/1088357618803332 arXiv:https://doi.org/10.1177/1088357618803332
  52. Fumiko Samejima. 1997. Graded Response Model. Springer New York, New York, NY, 85–100. https://doi.org/10.1007/978-1-4757-2691-6_5
  53. Laura Schiffl. 2020. Hierarchies in Lexical Complexity: Do Effects of Word Frequency, Word Length and Repetition Exist for the Visual Word Processing of People with Cognitive Impairments? In Easy Language Research: Text and User Perspectives, Silvia Hansen-Schirra and Christiane Maaß (Eds.). Frank & Timme GmbH, 227–239.
  54. Sanja Štajner and Sergiu Nisioi. 2018. A Detailed Evaluation of Neural Sequence-to-Sequence Models for In-domain and Cross-domain Text Simplification. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan. https://www.aclweb.org/anthology/L18-1479
  55. Regina Stodden. 2021. When the Scale is Unclear - Analysis of the Interpretation of Rating Scales in Human Evaluation of Text Simplification. In Proceedings of the First Workshop on Current Trends in Text Simplification (CTTS 2021) co-located with the 37th Conference of the Spanish Society for Natural Language Processing (SEPLN2021), Online (initially located in Málaga, Spain), September 21st, 2021 (CEUR Workshop Proceedings, Vol. 2944), Horacio Saggion, Sanja Štajner, Daniel Ferrés, and Kim Cheng Sheang (Eds.). CEUR-WS.org. http://ceur-ws.org/Vol-2944/paper6.pdf
  56. Assessing children’s reading comprehension on paper and screen: A mode-effect study. Computers & Education 151 (2020), 103861. https://doi.org/10.1016/j.compedu.2020.103861
  57. Enabling text comprehensibility assessment for people with intellectual disabilities using a mobile application. Frontiers in Communication 8 (2023). https://doi.org/10.3389/fcomm.2023.1175625
  58. Standardized Assessment of Reading Performance: The New International Reading Speed Texts IReST. Investigative Ophthalmology & Visual Science 53, 9 (08 2012), 5452–5461. https://doi.org/10.1167/iovs.11-8284
  59. Using response-time constraints to control for differential speededness in computerized adaptive testing. Applied psychological measurement 23, 3 (1999), 195–210.
  60. Sanja Štajner. 2021. Automatic text simplification for social good: Progress and challenges. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, Online, 2637–2652. https://doi.org/10.18653/v1/2021.findings-acl.233
  61. The role of reading time complexity and reading speed in text comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition 40, 6 (2014), 1745–1765. https://doi.org/10.1037/xlm0000030
  62. David Wechsler. 2003. Wechsler Intelligence Scale for Children, Fourth Edition. https://doi.org/10.1037/t15174-000
  63. Hyesun You. 2022. Bayesian Versus Frequentist Estimation for Item Response Theory Models of Interdisciplinary Science Assessment. Interdisciplinary Journal of Environmental and Science Education 18, 4 (jul 2022), e2297. https://doi.org/10.21601/ijese/12299

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets