Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices (2404.02213v1)

Published 2 Apr 2024 in cs.HC, cs.AI, and cs.CY

Abstract: Recent studies have integrated LLMs into diverse educational contexts, including providing adaptive programming hints, a type of feedback focuses on helping students move forward during problem-solving. However, most existing LLM-based hint systems are limited to one single hint type. To investigate whether and how different levels of hints can support students' problem-solving and learning, we conducted a think-aloud study with 12 novices using the LLM Hint Factory, a system providing four levels of hints from general natural language guidance to concrete code assistance, varying in format and granularity. We discovered that high-level natural language hints alone can be helpless or even misleading, especially when addressing next-step or syntax-related help requests. Adding lower-level hints, like code examples with in-line comments, can better support students. The findings open up future work on customizing help responses from content, format, and granularity levels to accurately identify and meet students' learning needs.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (30)
  1. Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education 26 (2016), 205–223.
  2. John R. Anderson and Brian J. Reiser. 1985. The LISP tutor. Byte 10, 4 (1985), 159–175. https://www.academia.edu/download/3240480/TheLISPTutor.pdf
  3. GPTutor: a ChatGPT-powered programming tool for code explanation. In International Conference on Artificial Intelligence in Education. Springer, Tokyo, Japan, 321–327.
  4. Intelligent tutoring systems for programming education: a systematic review. In Proceedings of the 20th Australasian Computing Education Conference (ACE ’18). Association for Computing Machinery, New York, NY, USA, 53–62. https://doi.org/10.1145/3160489.3160492
  5. On Designing Programming Error Messages for Novices: Readability and its Constituent Factors. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3411764.3445696
  6. Ask-Elle: an Adaptable Programming Tutor for Haskell Giving Automated Feedback. International Journal of Artificial Intelligence in Education 27, 1 (March 2017), 65–100. https://doi.org/10.1007/s40593-015-0080-x
  7. Using Adaptive Parsons Problems to Scaffold Write-Code Problems. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1. ACM, Lugano and Virtual Event Switzerland, 15–26. https://doi.org/10.1145/3501385.3543977
  8. Understanding the Effects of Using Parsons Problems to Scaffold Code Writing for Students with Varying CS Self-Efficacy Levels. In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research. ACM, Koli, Finland, 1–12.
  9. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–23. https://doi.org/10.1145/3544548.3580919
  10. How novices use LLM-based code generators to solve CS1 coding tasks in a self-paced learning environment. In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research. ACM, Koli, Finland, 1–12.
  11. A Systematic Literature Review of Automated Feedback Generation for Programming Exercises. ACM Transactions on Computing Education 19, 1 (March 2019), 1–43. https://doi.org/10.1145/3231711
  12. Kenneth R. Koedinger and Vincent Aleven. 2007. Exploring the Assistance Dilemma in Experiments with Cognitive Tutors. Educational Psychology Review 19, 3 (Sept. 2007), 239–264. https://doi.org/10.1007/s10648-007-9049-0
  13. QuickTA: Exploring the Design Space of Using Large Language Models to Provide Support to Students. In Learning Analytics and Knowledge Conference. Learning Analytics and Knowledge Conference 2023 (LAK’23), ACM, Arlington, Texas.
  14. Debugging support for end user mashup programming. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Association for Computing Machinery, New York, NY, USA, 1609–1618. https://doi.org/10.1145/2470654.2466213
  15. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. https://arxiv.org/abs/2308.06921v1
  16. “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–31. https://doi.org/10.1145/3544548.3580817
  17. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. ACM, Toronto ON Canada, 931–937. https://doi.org/10.1145/3545945.3569785
  18. Shoaib Ahmed Malik. 2017. Revisiting and re-representing scaffolding: The two gradient model. Cogent Education 4, 1 (2017), 1331533.
  19. Samiha Marwan. 2021. Investigating Best Practices in the Design of Automated Hints and Formative Feedback to Improve Students’ Cognitive and Affective Outcomes - Samiha Marwan PhD Thesis - 2021. Ph. D. Dissertation. North Carolina State University. https://doi.org/10.13140/RG.2.2.13347.32805
  20. Towards Generalized Methods for Automatic Question Generation in Educational Domains. In Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption (Lecture Notes in Computer Science), Isabel Hilliger, Pedro J. Muñoz-Merino, Tinne De Laet, Alejandro Ortega-Arranz, and Tracie Farrell (Eds.). Springer International Publishing, Cham, 272–284. https://doi.org/10.1007/978-3-031-16290-9_20
  21. iSnap: Towards Intelligent Tutoring in Novice Programming Environments. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, Seattle Washington USA, 483–488. https://doi.org/10.1145/3017680.3017762
  22. Kelly Rivers and Kenneth R. Koedinger. 2017. Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor. International Journal of Artificial Intelligence in Education 27, 1 (March 2017), 37–64. https://doi.org/10.1007/s40593-015-0070-z
  23. Next-Step Hint Generation for Introductory Programming Using Large Language Models. In Proceedings of the 26th Australasian Computing Education Conference. ACM, Melbourne, Australia, 144–153.
  24. Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction 21, 2 (April 2011), 267–280. https://doi.org/10.1016/j.learninstruc.2010.07.004
  25. Improving the Coverage of GPT for Automated Feedback on High School Programming Assignments. In NeurIPS’23 Workshop Generative AI for Education (GAIED). MIT Press, New Orleans, Louisiana, USA, 46.
  26. Valerie J. Shute. 2008. Focus on Formative Feedback. Review of Educational Research 78, 1 (March 2008), 153–189. https://doi.org/10.3102/0034654307313795
  27. The hint factory: Automatic generation of contextualized help for existing computer aided instruction. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track. Springer, Montreal,Canada, 71–78.
  28. Experimental evaluation of automatic hint generation for a logic tutor. International Journal of Artificial Intelligence in Education 22, 1-2 (2013), 3–17. https://content.iospress.com/articles/international-journal-of-artificial-intelligence-in-education/jai029 Publisher: IOS Press.
  29. Exploring the Design Space of Automatically Synthesized Hints for Introductory Programming Assignments. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 2951–2958. https://doi.org/10.1145/3027063.3053187
  30. Anaïs Tack and Chris Piech. 2022. The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues. In Proceedings of the 15th International Conference on Educational Data Mining, Antonija Mitrovic and Nigel Bosch (Eds.). International Educational Data Mining Society, Durham, United Kingdom, 522–529. https://doi.org/10.5281/zenodo.6853187
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Ruiwei Xiao (8 papers)
  2. Xinying Hou (11 papers)
  3. John Stamper (14 papers)
Citations (8)
View on Semantic Scholar