Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Effects of Generative AI on Computing Students' Help-Seeking Preferences (2401.02262v1)

Published 4 Jan 2024 in cs.HC

Abstract: Help-seeking is a critical way for students to learn new concepts, acquire new skills, and get unstuck when problem-solving in their computing courses. The recent proliferation of generative AI tools, such as ChatGPT, offers students a new source of help that is always available on-demand. However, it is unclear how this new resource compares to existing help-seeking resources along dimensions of perceived quality, latency, and trustworthiness. In this paper, we investigate the help-seeking preferences and experiences of computing students now that generative AI tools are available to them. We collected survey data (n=47) and conducted interviews (n=8) with computing students. Our results suggest that although these models are being rapidly adopted, they have not yet fully eclipsed traditional help resources. The help-seeking resources that students rely on continue to vary depending on the task and other factors. Finally, we observed preliminary evidence about how help-seeking with generative AI is a skill that needs to be developed, with disproportionate benefits for those who are better able to harness the capabilities of LLMs. We discuss potential implications for integrating generative AI into computing classrooms and the future of help-seeking in the era of generative AI.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Help Seeking and Help Design in Interactive Learning Environments. Review of Educational Research 73, 3 (2003), 277–320. https://doi.org/10.3102/00346543073003277 arXiv:https://doi.org/10.3102/00346543073003277
  2. Maryam Ashoori and Justin D Weisz. 2019. In AI we trust? Factors that influence trustworthiness of AI-infused decision-making processes. arXiv preprint arXiv:1912.02675 (2019).
  3. Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 500–506. https://doi.org/10.1145/3545945.3569759
  4. Examining the Role of Self-Regulated Learning on Introductory Programming Performance. In Proceedings of the First International Workshop on Computing Education Research (Seattle, WA, USA) (ICER ’05). 81–86.
  5. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101.
  6. Nicole Buzzetto-Hollywood. 2014. An Examination of Undergraduate Student’s Perceptions and Predilections of the Use of YouTube in the Teaching and Learning Process. Interdisciplinary Journal of E-Learning and Learning Objects (IJELLO) 10 (01 2014), 17–32. https://doi.org/10.28945/1965
  7. Kun-Hung Cheng and Chin-Chung Tsai. 2011. An investigation of Taiwan University students’ perceptions of online academic help seeking, and their web-based learning self-efficacy. Internet and Higher Education - INTERNET HIGH EDUC 14 (07 2011), 150–157. https://doi.org/10.1016/j.iheduc.2011.04.002
  8. Understanding knowledge sharing in virtual communities: An integration of social capital and social cognitive theories. Decision Support Systems 42, 3 (2006), 1872–1888.
  9. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 1136–1142. https://doi.org/10.1145/3545945.3569823
  10. Promptly: Using prompt problems to teach learners how to effectively utilize ai code generators. arXiv preprint arXiv:2307.16364 (2023).
  11. Augie Doebling and Ayaan M. Kazerouni. 2021. Patterns of Academic Help-Seeking in Undergraduate Computing Students. In Proceedings of the 21st Koli Calling International Conference on Computing Education Research (Joensuu, Finland) (Koli Calling ’21). Association for Computing Machinery, New York, NY, USA, Article 13, 10 pages. https://doi.org/10.1145/3488042.3488052
  12. Pierpaolo Dondio and Suha Shaheen. 2020. Is StackOverflow an Effective Complement to Gaining Practical Knowledge Compared to Traditional Computer Science Learning?. In Proceedings of the 11th International Conference on Education Technology and Computers (Amsterdam, Netherlands) (ICETC ’19). ACM.
  13. An eye-tracking study of query reformulation. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 13–22.
  14. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conf. 10–19.
  15. Debugging: finding, fixing and flailing, a multi-institutional study of novice debuggers. Computer Science Education 18, 2 (2008), 93–116.
  16. Online feedback exchange: A framework for understanding the socio-psychological factors. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4454–4467.
  17. Batya Friedman. 1996. Value-sensitive design. interactions 3, 6 (1996), 16–23.
  18. Value sensitive design and information systems. Early engagement and new technologies: Opening up the laboratory (2013), 55–95.
  19. You Asked, Now What? Modeling Students’ Help-Seeking and Coding Actions from Request to Resolution. Journal of Educational Data Mining 14, 3 (2022), 109–131.
  20. Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1. ACM.
  21. Christopher Herring and Joachim Walther. 2016. Academic help-seeking as a stand-alone, metacognitive action: An empirical study of experiences and behaviors in undergraduate engineering students. In 2016 ASEE Annual Conference & Exposition.
  22. More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems. arXiv preprint arXiv:2311.04926 (2023).
  23. Jeff Huang and Efthimis N Efthimiadis. 2009. Analyzing and evaluating query reformulation strategies in web search logs. In Proceedings of the 18th ACM conference on Information and knowledge management. 77–86.
  24. Memory sandbox: Transparent and interactive memory management for conversational agents. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–3.
  25. CausalMapper: Challenging Designers to Think in Systems with Causal Maps and Large Language Model. In Proceedings of the 15th Conference on Creativity and Cognition (Virtual Event, USA) (C&C ’23). Association for Computing Machinery, New York, NY, USA, 325–329. https://doi.org/10.1145/3591196.3596818
  26. PromptMaker: Prompt-based Prototyping with Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–8.
  27. Stuart A Karabenick. 2003. Seeking help in large college classes: A person-centered approach. Contemporary educational psychology 28, 1 (2003), 37–58.
  28. Stuart A Karabenick and Jean-Louis Berger. 2013. Help seeking as a self-regulated learning strategy. (2013).
  29. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). ACM. https://doi.org/10.1145/3544548.3580919
  30. Päivi Kinnunen and Beth Simon. 2012. My program is ok – am I? Computing freshmen’s experiences of doing programming assignments. Computer Science Education 22, 1 (2012), 1–28. https://doi.org/10.1080/08993408.2012.655091 arXiv:https://doi.org/10.1080/08993408.2012.655091
  31. Tom Koole. 2012. The epistemics of student problems: Explaining mathematics in a multi-lingual class. Journal of Pragmatics 44, 13 (2012), 1902–1916. https://doi.org/10.1016/j.pragma.2012.08.006
  32. David E Kumrow. 2007. Evidence-based strategies of graduate students to achieve success in a hybrid Web-based course. The Journal of Nursing Education 46, 3 (2007), 140–145.
  33. Sam Lau and Philip J. Guo. 2023. From ‘Ban It Till We Understand It’ to ”Resistance is Futile”: How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 (ICER ’23 V1). ACM. https://doi.org/10.1145/3568813.3600138
  34. Comparing Code Explanations Created by Students and Large Language Models. arXiv preprint arXiv:2304.03938 (2023).
  35. Using large language models to enhance programming error messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 563–569.
  36. Codehelp: Using large language models with guardrails for scalable support in programming classes. arXiv preprint arXiv:2308.06921 (2023).
  37. Programming, problem solving, and self-awareness: Effects of explicit guidance. In Proceedings of the 2016 CHI conference on human factors in computing systems. 1449–1461.
  38. Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models. arXiv preprint arXiv:2311.16017 (2023).
  39. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023). 931–937.
  40. Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordances. arXiv preprint arXiv:2307.01142 (2023).
  41. Automatically Generating CS Learning Materials with Large Language Models. arXiv preprint arXiv:2212.05113 (2022).
  42. Generating Diverse Code Explanations Using the GPT-3 Large Language Model. In Proc. of the 2022 ACM Conf. on Int. Computing Education Research - Volume 2. ACM, 37–39.
  43. Danielle Magaldi and Matthew Berler. 2020. Semi-structured Interviews. In Encyclopedia of Personality and Individual Differences, Virgil Zeigler-Hill and Todd K. Shackelford (Eds.). Springer International Publishing, Cham, 4825–4830. https://doi.org/10.1007/978-3-319-24612-3_857
  44. Mind Your Language: On Novices’ Interactions with Error Messages. In Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Portland, Oregon, USA) (Onward! 2011). 3–18.
  45. Why Computing Students Learn on Their Own: Motivation for Self-Directed Learning of Computing. ACM Trans. Comput. Educ. 16, 1, Article 2 (jan 2016), 18 pages. https://doi.org/10.1145/2747008
  46. Tilman Michaeli and Ralf Romeike. 2019. Improving Debugging Skills in the Classroom: The Effects of Teaching a Systematic Debugging Process. In Proceedings of the 14th Workshop in Primary and Secondary Computing Education (Glasgow, Scotland, Uk) (WiPSCE ’19). Association for Computing Machinery, New York, NY, USA, Article 15, 7 pages. https://doi.org/10.1145/3361721.3361724
  47. What makes a good code example?: A study of programming Q&A in StackOverflow. In 2012 28th IEEE International Conference on Software Maintenance (ICSM). IEEE.
  48. Richard S. Newman. 1990. Children’s help-seeking in the classroom: The role of motivational factors and attitudes. Journal of Educational Psychology 82, 1 (1990), 71–80. https://doi.org/10.1037/0022-0663.82.1.71
  49. The ”Active Search” Hypothesis: How Search Strategies Relate to Creative Learning. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval (Canberra ACT, Australia) (CHIIR ’21). Association for Computing Machinery, New York, NY, USA, 325–329. https://doi.org/10.1145/3406522.3446046
  50. Ernesto Panadero and Maryam Alqassab. 2019. An empirical review of anonymity effects in peer assessment, peer feedback, peer review, peer evaluation and peer grading. Assessment & Evaluation in Higher Education 44, 8 (2019), 1253–1278.
  51. Transformed by Transformers: Navigating the AI Coding Revolution for Computing Education: An ITiCSE Working Group Conducted by Humans. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2. 561–562.
  52. The robots are here: Navigating the generative ai revolution in computing education. arXiv preprint arXiv:2310.00658 (2023).
  53. “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers. ACM Transactions on Computer-Human Interaction 31, 1 (Nov. 2023), 1–31. https://doi.org/10.1145/3617367
  54. iSnap: towards intelligent tutoring in novice programming environments. In Proc. of the 2017 ACM SIGCSE Technical Symposium on computer science education. 483–488.
  55. Yizhou Qian and James Lehman. 2017. Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review. ACM Trans. Comput. Educ. 18, 1, Article 1 (oct 2017), 24 pages. https://doi.org/10.1145/3077618
  56. What Help Do Students Seek in TA Office Hours?. In Proceedings of the 2019 ACM Conference on International Computing Education Research (Toronto ON, Canada) (ICER ’19). Association for Computing Machinery, New York, NY, USA, 41–49. https://doi.org/10.1145/3291279.3339418
  57. Allison M. Ryan and Huiyoung Shin. 2011. Help-seeking tendencies during early adolescence: An examination of motivational correlates and consequences for achievement. Learning and Instruction 21, 2 (2011), 247–256. https://doi.org/10.1016/j.learninstruc.2010.07.003 Special Section I: Solving information-based problems: Evaluating sources and information Special Section II: Stretching the limits in help-seeking research: Theoretical, methodological, and technological advances.
  58. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proc. of the 2022 ACM Conf. on Int. Computing Education Research - Volume 1. ACM, 27–43.
  59. Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. (2023), 78–92. https://doi.org/10.1145/3568813.3600142
  60. Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code. arXiv:2303.08033 [cs.CL]
  61. Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses? arXiv preprint arXiv:2303.09325 (2023).
  62. Mohd Sharif and AN Zainab. 2004. Undergraduates in computer science and information technology using the Internet as a Resource. Malaysian Journal of Library & Information Science 9, 1 (2004), 1–16.
  63. Anselm L Strauss and Juliet Corbin. 2004. Open coding. Social research methods: A reader (2004), 303–306.
  64. Jerry Ed Suls and Thomas Ashby Ed Wills. 1991. Social comparison: Contemporary theory and research. Lawrence Erlbaum Associates, Inc.
  65. Using Large Language Models to Automatically Identify Programming Concepts in Code Snippets. In Proc. of the 2023 ACM Conf. on Int. Computing Education Research - Volume 2, Vol. 1. ACM, 563–569. https://doi.org/10.1145/3568812.3603482
  66. Stackoverflow and github: Associations between software development and crowdsourced knowledge. In 2013 International Conference on Social Computing. IEEE, 188–195.
  67. Resource usage and usefulness: academic help-seeking behaviours of undergraduate engineering students. Australasian Journal of Engineering Education 23, 2 (2018), 62–70. https://doi.org/10.1080/22054952.2018.1525889 arXiv:https://doi.org/10.1080/22054952.2018.1525889
  68. Conversation Technology With Micro-Learning: The Impact of Chatbot-Based Learning on Students’ Learning Motivation and Performance. Journal of Educational Computing Research 59, 1 (2021), 154–177. https://doi.org/10.1177/0735633120952067
  69. Self-Directed Learning: Stories from Industry. In Proceedings of the 12th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’12). Association for Computing Machinery, New York, NY, USA, 111–117. https://doi.org/10.1145/2401796.2401810
  70. Generative AI in Computing Education: Perspectives of Students and Instructors. arXiv preprint arXiv:2308.04309 (2023).
  71. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 295–305.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Irene Hou (5 papers)
  2. Sophia Metille (1 paper)
  3. Zhuo Li (164 papers)
  4. Owen Man (5 papers)
  5. Cynthia Zastudil (5 papers)
  6. Stephen MacNeil (37 papers)
Citations (24)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets