Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Using Think-Aloud Data to Understand Relations between Self-Regulation Cycle Characteristics and Student Performance in Intelligent Tutoring Systems (2312.05675v1)

Published 9 Dec 2023 in cs.CY and cs.AI

Abstract: Numerous studies demonstrate the importance of self-regulation during learning by problem-solving. Recent work in learning analytics has largely examined students' use of SRL concerning overall learning gains. Limited research has related SRL to in-the-moment performance differences among learners. The present study investigates SRL behaviors in relationship to learners' moment-by-moment performance while working with intelligent tutoring systems for stoichiometry chemistry. We demonstrate the feasibility of labeling SRL behaviors based on AI-generated think-aloud transcripts, identifying the presence or absence of four SRL categories (processing information, planning, enacting, and realizing errors) in each utterance. Using the SRL codes, we conducted regression analyses to examine how the use of SRL in terms of presence, frequency, cyclical characteristics, and recency relate to student performance on subsequent steps in multi-step problems. A model considering students' SRL cycle characteristics outperformed a model only using in-the-moment SRL assessment. In line with theoretical predictions, students' actions during earlier, process-heavy stages of SRL cycles exhibited lower moment-by-moment correctness during problem-solving than later SRL cycle stages. We discuss system re-design opportunities to add SRL support during stages of processing and paths forward for using machine learning to speed research depending on the assessment of SRL based on transcription of think-aloud data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Vincent Aleven and Kenneth R Koedinger. 2000. Limitations of student control: Do students know when they need help?. In International conference on intelligent tutoring systems. Springer, 292–303.
  2. Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence in Education 16, 2 (2006), 101–128.
  3. Example-tracing tutors: Intelligent tutor development for non-programmers. International Journal of Artificial Intelligence in Education 26 (2016), 224–269. Springer.
  4. Lessons learned and future directions of metatutor: Leveraging multichannel data to scaffold self-regulated learning with an intelligent tutoring system. Frontiers in Psychology 13 (2022), 813632.
  5. Does adaptive scaffolding facilitate students’ ability to regulate their learning with hypermedia? Contemporary educational psychology 29, 3 (2004), 344–370.
  6. Roger Azevedo and Dragan Gašević. 2019. Analyzing multimodal multichannel data about self-regulated learning with advanced learning technologies: Issues and challenges. Computers in Human Behavior 96 (2019), 207–210.
  7. Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development 56 (2008), 45–72.
  8. Measuring cognitive and metacognitive regulatory processes during hypermedia learning: Issues and challenges. Educational psychologist 45, 4 (2010), 210–223.
  9. Process mining techniques for analysing patterns and strategies in students’ self-regulated learning. Metacognition and learning 9 (2014), 161–185.
  10. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) 57, 1 (1995), 289–300.
  11. Measuring self-regulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology Enhanced Learning 5, 02 (2010), 123–152.
  12. Benjamin M Bolker. 2015. Linear and generalized linear mixed models. Ecological statistics: contemporary theory and application (2015), 309–333.
  13. What Makes Problem-Solving Practice Effective? Comparing Paper and AI Tutoring. In European Conference on Technology Enhanced Learning. Springer, 44–59.
  14. Instructional Factors Analysis: A Cognitive Model For Multiple Instructional Interventions. In EDM 2011 4 th International Conference on Educational Data Mining.
  15. Comparing and combining interaction data and eye-tracking data for the real-time prediction of user cognitive abilities in visualization tasks. ACM Transactions on Interactive Intelligent Systems (TiiS) 10, 2 (2020), 1–41.
  16. Trevor A Craney and James G Surles. 2002. Model-dependent variance inflation factor cutoff values. Quality engineering 14, 3 (2002), 391–403.
  17. Deep learning for discussion-based cross-domain performance prediction of MOOC learners grouped by language on FutureLearn. Arabian Journal for Science and Engineering 46 (2021), 3613–3629.
  18. Learning stoichiometry: a comparison of text and multimedia formats. Chemistry Education Research and Practice 9, 3 (2008), 208–218.
  19. Towards a fuller picture: Triangulation and integration of the measurement of self-regulated learning based on trace and think aloud data. Journal of Computer Assisted Learning (2023).
  20. Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological bulletin 137, 2 (2011), 316.
  21. Enhancing Engagement Modeling in Game-Based Learning Environments with Student-Agent Discourse Analysis. In International Conference on Artificial Intelligence in Education. Springer, 681–687.
  22. Progression of students’ SRL processes in subsequent programming problem-solving tasks and its association with tasks outcomes. The Internet and Higher Education 56 (2023), 100881.
  23. Mine the process: Investigating the cyclical nature of upper primary school students’ self-regulated learning. Instructional Science 48 (2020), 337–369.
  24. A general multi-method approach to data-driven redesign of tutoring systems. In LAK21: 11th International Learning Analytics and Knowledge Conference. 161–172.
  25. Investigating SMART Models of Self-Regulation and Their Impact on Learning. International Educational Data Mining Society (2021).
  26. The Open-Response Chemistry Cognitive Assistance Tutor System: Development and Implementation.
  27. A data repository for the EDM community: The PSLC DataShop. Handbook of educational data mining 43 (2010), 43–56.
  28. Temporal structures and sequential patterns of self-regulated learning behaviors in problem solving with an intelligent tutoring system. Educational Technology & Society 25, 4 (2022), 1–14.
  29. Temporal assessment of self-regulated learning by mining students’ think-aloud protocols. Frontiers in Psychology 12 (2021), 749749.
  30. Effects of real-time analytics-based personalized scaffolds on students’ self-regulated learning. Computers in Human Behavior 139 (2023), 107547.
  31. Yanjin Long and Vincent Aleven. 2016. Mastery-oriented shared student/system control over problem selection in a linear equation tutor. In Intelligent Tutoring Systems: 13th International Conference, ITS 2016, Zagreb, Croatia, June 7-10, 2016. Proceedings 13. Springer, 90–100.
  32. Polite web-based intelligent tutors: Can they improve learning in classrooms? Computers & Education 56, 3 (2011), 574–584.
  33. Studying the effects of personalized language and worked examples in the context of a web-based intelligent tutor. In Intelligent Tutoring Systems: 8th International Conference, ITS 2006, Jhongli, Taiwan, June 26-30, 2006. Proceedings 8. Springer, 318–328.
  34. Measuring self-regulated learning and the role of AI: Five years of research using multimodal multichannel data. Computers in Human Behavior 139 (2023), 107540.
  35. Analysing adaptive scaffolds that help students develop self-regulated learning behaviours. Journal of Computer Assisted Learning 39, 2 (2023), 351–368.
  36. Ernesto Panadero. 2017. A review of self-regulated learning: Six models and four directions for research. Frontiers in psychology 8 (2017), 422.
  37. Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In Proceedings of the third international conference on learning analytics and knowledge. 117–124.
  38. Sadhana Puntambekar and Roland Hubscher. 2005. Tools for scaffolding students in a complex learning environment: What have we gained and what have we missed? Educational psychologist 40, 1 (2005), 1–12.
  39. Robust Speech Recognition via Large-Scale Weak Supervision. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 28492–28518. https://proceedings.mlr.press/v202/radford23a.html
  40. Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and instruction 21, 2 (2011), 267–280.
  41. On the benefits of seeking (and avoiding) help in online problem-solving environments. Journal of the Learning Sciences 23, 4 (2014), 537–560.
  42. Granularity matters: comparing different ways of measuring self-regulated learning. Metacognition and Learning 14 (2019), 1–19.
  43. Discovering behavior patterns of self-regulated learners in an inquiry-based learning environment. In Artificial Intelligence in Education: 16th International Conference, AIED 2013, Memphis, TN, USA, July 9-13, 2013. Proceedings 16. Springer, 209–218.
  44. Temporally-focused analytics of self-regulated learning: A systematic review of literature. Computers and education: Artificial intelligence 3 (2022), 100060.
  45. Using coherence analysis to characterize self-regulated learning behaviours in open-ended learning environments. Journal of Learning Analytics 2, 1 (2015), 13–48.
  46. Evaluating the Explainers: Black-Box Explainable Machine Learning for Student Success Prediction in MOOCs. In Proceedings of the 15th International Conference on Educational Data Mining. 98.
  47. Supporting algebraic reasoning through personalized story scenarios: How situational understanding mediates performance. Mathematical Thinking and Learning 15, 2 (2013), 89–120.
  48. Philip H Winne. 2004. Students’ calibration of knowledge and learning processes: Implications for designing powerful software learning environments. International Journal of Educational Research 41, 6 (2004), 466–488.
  49. Philip H Winne. 2010. Improving measurement of self-regulated learning. Educational Psychologist 45, 4 (2010), 267–276.
  50. Philip H Winne. 2017. Handbook of learning analytics. Society for learning analytics research (2017), 241–249.
  51. Philip H. Winne and Allyson F. Hadwin. 1998. Studying as Self-Regulated Learning. In Metacognition in Educational Theory and Practice, Douglas J. Hacker, John Dunlosky, and Arthur C. Graesser (Eds.). Erlbaum, Hillsdale, NJ, 277–304.
  52. Detecting SMART Model Cognitive Operations in Mathematical Problem-Solving Process. International Educational Data Mining Society (2022).
  53. A multi-level trace clustering analysis scheme for measuring students’ self-regulated learning behavior in a mastery-based online learning environment. In LAK22: 12th International Learning Analytics and Knowledge Conference. 197–207.
  54. Barry J Zimmerman. 1990. Self-regulated learning and academic achievement: An overview. Educational psychologist 25, 1 (1990), 3–17.
  55. Barry J Zimmerman. 2000. Attaining self-regulation: A social cognitive perspective. In Handbook of self-regulation. Elsevier, 13–39.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now