Papers
Topics
Authors
Recent
Search
2000 character limit reached

ReflectSumm: A Benchmark for Course Reflection Summarization

Published 27 Mar 2024 in cs.CL and cs.AI | (2403.19012v2)

Abstract: This paper introduces ReflectSumm, a novel summarization dataset specifically designed for summarizing students' reflective writing. The goal of ReflectSumm is to facilitate developing and evaluating novel summarization techniques tailored to real-world scenarios with little training data, %practical tasks with potential implications in the opinion summarization domain in general and the educational domain in particular. The dataset encompasses a diverse range of summarization tasks and includes comprehensive metadata, enabling the exploration of various research questions and supporting different applications. To showcase its utility, we conducted extensive evaluations using multiple state-of-the-art baselines. The results provide benchmarks for facilitating further research in this area.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Extractive opinion summarization in quantized transformer spaces. Transactions of the Association for Computational Linguistics, 9:277–293.
  2. Stefanos Angelidis and Mirella Lapata. 2018. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3675–3686.
  3. The importance of reflection in improving science teaching and learning. Journal of Research in Science Teaching, 28:163 – 182.
  4. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.
  5. Prompted opinion summarization with GPT-3.5. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9282–9300, Toronto, Canada. Association for Computational Linguistics.
  6. Few-shot learning for opinion summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4119–4135.
  7. Efficient few-shot fine-tuning for opinion summarization. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1509–1523.
  8. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  9. Investigating student reflection during game-based learning in middle grades science. In LAK21: 11th International Learning Analytics and Knowledge Conference, LAK21, page 280–291, New York, NY, USA. Association for Computing Machinery.
  10. Automated analysis of middle school students’ written reflections during game-based learning. In AIED 2020: Artificial Intelligence in Education, pages 67–78.
  11. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  12. Eric Chu and Peter Liu. 2019. Meansum: A neural model for unsupervised multi-document abstractive summarization. In International Conference on Machine Learning, pages 1223–1232. PMLR.
  13. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615–621.
  14. Summarizing user-generated textual content: Motivation and methods for fairness in algorithmic summaries. Proc. ACM Hum.-Comput. Interact., 3(CSCW).
  15. Msˆ2: Multi-document summarization of medical studies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7494–7513.
  16. Mohamed Elaraby and Diane Litman. 2022. Arglegalsumm: Improving abstractive summarization of legal documents with argument mining. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6187–6194.
  17. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457–479.
  18. QaFactEval: Improved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2587–2601.
  19. CourseMIRROR: Enhancing large classroom instructor-student interactions via mobile interfaces and natural language processing. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA ’15, page 1473–1478, New York, NY, USA. Association for Computing Machinery.
  20. Scaling reflection prompts in large classrooms via mobile interfaces and natural language processing. In Proceedings of the 22nd International Conference on Intelligent User Interfaces, IUI ’17, page 363–374, New York, NY, USA. Association for Computing Machinery.
  21. Predicting and analyzing language specificity in social media posts. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6415–6422.
  22. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. EMNLP-IJCNLP 2019, page 70.
  23. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356.
  24. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1693–1701, Cambridge, MA, USA. MIT Press.
  25. spaCy: Industrial-strength Natural Language Processing in Python.
  26. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
  27. A bag of tricks for dialogue summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8014–8022.
  28. Understand students’ self-reflections through learning analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, LAK ’18, page 389–398, New York, NY, USA. Association for Computing Machinery.
  29. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177.
  30. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
  31. Junyi Jessy Li and Ani Nenkova. 2015. Fast and accurate prediction of sentence specificity. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, page 2281–2287. AAAI Press.
  32. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  33. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  34. Wencan Luo and Diane Litman. 2015. Summarizing student responses to reflection prompts. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1955–1960, Lisbon, Portugal. Association for Computational Linguistics.
  35. Wencan Luo and Diane J. Litman. 2016. Determining the quality of a student reflective response. In Proceedings of the Twenty-Ninth International Florida Artificial Intelligence Research Society Conference, pages 226–231.
  36. Automatic summarization of student course feedback. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 80–85, San Diego, California. Association for Computational Linguistics.
  37. Ahmed Magooda. 2022. Techniques to enhance abstractive summarization model training for low resource domains. PhD Thesis.
  38. Ahmed Magooda and Diane Litman. 2021. Mitigating data scarceness through data synthesis, augmentation and curriculum for abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2043–2052.
  39. Improving the quality of students’ written reflections using natural language processing: Model design and classroom evaluation. In Artificial Intelligence in Education: 23rd International Conference, AIED 2022, Durham, UK, July 27–31, 2022, Proceedings, Part I, page 519–525, Berlin, Heidelberg. Springer-Verlag.
  40. Exploring multitask learning for low-resource abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1652–1661, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  41. Ahmed Magooda and Diane J. Litman. 2020. Abstractive summarization for low resource data using domain transfer and data synthesis. In The Thirty-Third International FLAIRS Conference (FLAIRS-33), pages 240–245.
  42. Muhsin Menekse. 2020. The reflection-informed learning and instruction to improve students’ academic success in undergraduate classrooms. The Journal of Experimental Education, 88(2):183–199.
  43. The effectiveness of students’ daily reflections on learning in engineering context. ASEE Annual Conference and Exposition, Conference Proceedings.
  44. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807.
  45. Nadav Oved and Ran Levy. 2021. PASS: Perturb-and-select summarizer for product reviews. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 351–365, Online. Association for Computational Linguistics.
  46. Is ChatGPT a general-purpose natural language processing task solver? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1339–1384, Singapore. Association for Computational Linguistics.
  47. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67.
  48. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. The 5th EMC2 - Energy Efficient Training and Inference of Transformer Based Models, co-located at NeurIPS 2019.
  49. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.
  50. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online. Association for Computational Linguistics.
  51. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  52. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  53. Thomas Ullmann. 2019. Automated analysis of reflection in writing: Validating machine learning approaches. International Journal of Artificial Intelligence in Education, 29.
  54. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics.
  55. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
  56. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv e-prints, page arXiv:2211.05100.
  57. OASum: Large-scale open domain aspect-based summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4381–4401, Toronto, Canada. Association for Computational Linguistics.
  58. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR.
  59. BERTscore: Evaluating text generation with bert. In International Conference on Learning Representations.
  60. Benchmarking Large Language Models for News Summarization. Transactions of the Association for Computational Linguistics, 12:39–57.
  61. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197–6208, Online. Association for Computational Linguistics.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up