Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education (2310.12324v2)

Published 18 Oct 2023 in cs.HC, cs.AI, and cs.LG

Abstract: Randomized A/B comparisons of alternative pedagogical strategies or other course improvements could provide useful empirical evidence for instructor decision-making. However, traditional experiments do not provide a straightforward pathway to rapidly utilize data, increasing the chances that students in an experiment experience the best conditions. Drawing inspiration from the use of machine learning and experimentation in product development at leading technology companies, we explore how adaptive experimentation might aid continuous course improvement. In adaptive experiments, data is analyzed and utilized as different conditions are deployed to students. This can be achieved using machine learning algorithms to identify which actions are more beneficial in improving students' learning experiences and outcomes. These algorithms can then dynamically deploy the most effective conditions in subsequent interactions with students, resulting in better support for students' needs. We illustrate this approach with a case study that provides a side-by-side comparison of traditional and adaptive experiments on adding self-explanation prompts in online homework problems in a CS1 course. This work paves the way for exploring the importance of adaptive experiments in bridging research and practice to achieve continuous improvement in educational settings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Making Contextual Decisions with Low Technical Debt. (June 2016). https://doi.org/10.48550/arXiv.1606.03966
  2. Shipra Agrawal and Navin Goyal. 2012. Analysis of Thompson Sampling for the Multi-armed Bandit Problem. In Conference on learning theory. JMLR Workshop and Conference Proceedings, 39–1.
  3. A (Updated) Review of Empiricism at the SIGCSE Technical Symposium. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. ACM, Memphis Tennessee USA, 120–125. https://doi.org/10.1145/2839509.2844601
  4. Exploring Additional Personalized Support While Attempting Exercise Problems in Online Learning Platforms. In Proceedings of the Eighth ACM Conference on Learning@ Scale. 235–238.
  5. The A/B Testing Problem. In Proceedings of the 2018 ACM Conference on Economics and Computation. ACM, Ithaca NY USA, 461–462. https://doi.org/10.1145/3219166.3219204
  6. AE: A domain-agnostic platform for adaptive experimentation. In Conference on Neural Information Processing Systems. 1–8.
  7. What Influences CS Faculty to Adopt Teaching Practices?. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. ACM, Kansas City Missouri USA, 604–609. https://doi.org/10.1145/2676723.2677282
  8. It’s how you say it: systematic A/B testing of digital messaging cut hospital no-show rates. PloS one 15, 6 (2020), e0234817. Publisher: Public Library of Science San Francisco, CA USA.
  9. Getting Ideas into Action: Building Networked Improvement Communities in Education. In Frontiers in Sociology of Education, Maureen T. Hallinan (Ed.). Springer Netherlands, Dordrecht, 127–162. https://doi.org/10.1007/978-94-007-1576-9_7
  10. Olivier Chapelle and Lihong Li. 2011. An Empirical Evaluation of Thompson Sampling. In Advances in Neural Information Processing Systems, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger (Eds.), Vol. 24. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2011/file/e53a0a2978c28872a4505bdb51db06dc-Paper.pdf
  11. John M. Clement. 2004. A Call for Action (Research): Applying Science Education Research to Computer Science Instruction. Computer Science Education 14, 4 (Dec. 2004), 343–364. https://doi.org/10.1080/0899340042000303474
  12. Efficient Inference Without Trading-off Regret in Bandits: An Allocation Probability Test for Thompson Sampling. http://arxiv.org/abs/2111.00137 arXiv:2111.00137 [cs, stat].
  13. A/B Testing at Scale: Accelerating Software Innovation. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’17). Association for Computing Machinery, New York, NY, USA, 1395–1397. https://doi.org/10.1145/3077136.3082060
  14. A systematic meta-Review and analysis of learning analytics research. Behaviour & Information Technology 40, 1 (Jan. 2021), 49–62. https://doi.org/10.1080/0144929X.2019.1669712 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/0144929X.2019.1669712.
  15. ”Closing the Loop” in Educational Data Science with an Open Source Architecture for Large-Scale Field Trials. In Proceedings of the 15th International Conference on Educational Data Mining, Antonija Mitrovic and Nigel Bosch (Eds.). International Educational Data Mining Society, Durham, United Kingdom, 834–838. https://doi.org/10.5281/zenodo.6852930
  16. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the National Academy of Sciences 118, 15 (April 2021), e2014602118. https://doi.org/10.1073/pnas.2014602118
  17. The Handbook of Behavior Change (1 ed.). Cambridge University Press. https://doi.org/10.1017/9781108677318
  18. A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature. ACM Transactions on Computing Education 22, 1 (March 2022), 1–46. https://doi.org/10.1145/3470652
  19. Neil T. Heffernan and Cristina Lindquist Heffernan. 2014. The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 4 (2014), 470–497. Publisher: Springer.
  20. Design of a curriculum analytics tool to support continuous improvement processes in higher education. In Proceedings of the tenth international conference on learning analytics & knowledge. 181–186.
  21. Survey Results on Why CS Faculty Adopt New Teaching Practices. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. ACM, Minneapolis MN USA, 483–489. https://doi.org/10.1145/3287324.3287420
  22. Trustworthy online controlled experiments: a practical guide to A/B testing. Cambridge University Press, Cambridge, United Kingdom ; New York, NY.
  23. Online randomized controlled experiments at scale: lessons and extensions to medicine. Trials 21, 1 (2020), 1–9. Publisher: Springer.
  24. Tor Lattimore and Csaba Szepesvári. 2020. Bandit Algorithms. Cambridge University Press. https://doi.org/10.1017/9781108571401
  25. Exploring the Effects of Contextualized Problem Descriptions on Problem Solving. In Australasian Computing Education Conference. ACM, Virtual SA Australia, 30–39. https://doi.org/10.1145/3441636.3442302
  26. Intelligent Support for All? A Literature Review of the (In)Equitable Design & Evaluation of Adaptive Pedagogical Systems for CS Education. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1 (SIGCSE 2022). Association for Computing Machinery, New York, NY, USA, 996–1002. https://doi.org/10.1145/3478431.3499418 event-place: Providence, RI, USA.
  27. Adaptive Immediate Feedback Can Improve Novice Programming Engagement and Intention to Persist in Computer Science. In Proceedings of the 2020 ACM Conference on International Computing Education Research (ICER ’20). Association for Computing Machinery, New York, NY, USA, 194–203. https://doi.org/10.1145/3372782.3406264 event-place: Virtual Event, New Zealand.
  28. Learning analytics as a tool for closing the assessment loop in higher education. Knowledge management & e-learning: An international journal 4, 3 (2012), 236–247.
  29. Susan McKenney and Thomas C. Reeves. 2018. Conducting Educational Design Research (2 ed.). Routledge, Second edition. | New York : Routledge, 2019. | “[First edition. https://doi.org/10.4324/9781315105642
  30. Embedding Experiments: Staking Causal Inference in Authentic Educational Contexts. Journal of Learning Analytics 5, 2 (Aug. 2018). https://doi.org/10.18608/jla.2018.52.4
  31. Algorithms for Adaptive Experiments that Trade-off Statistical Analysis with Reward: Combining Uniform Random Assignment and Reward Maximization. http://arxiv.org/abs/2112.08507 arXiv:2112.08507 [cs, stat].
  32. Conceptualizing Research–Practice Partnerships as Joint Work at Boundaries. Journal of Education for Students Placed at Risk (JESPAR) 20, 1-2 (April 2015), 182–197. https://doi.org/10.1080/10824669.2014.988334
  33. Statistical Consequences of using Multi-armed Bandits to Conduct Adaptive Educational Experiments. (June 2019). https://doi.org/10.5281/ZENODO.3554749 Publisher: Zenodo Version Number: 1.0.0.
  34. The MOOClet Framework: Unifying Experimentation, Dynamic Improvement, and Personalization in Online Courses. In Proceedings of the Eighth ACM Conference on Learning@ Scale. 15–26.
  35. Third Annual Workshop on A/B Testing and Platform-Enabled Learning Research. In Proceedings of the Ninth ACM Conference on Learning@ Scale. 252–254.
  36. The rise of the super experiment. In Proceedings of the 5th International Conference on Educational Data Mining. International Educational Data Mining Society, Chania, Greece, 196–199.
  37. Allan Wigfield and Alison C. Koenka. 2020. Where do we go from here in academic motivation theory and research? Some reflections and recommendations for future work. Contemporary Educational Psychology 61 (2020), 101872. Publisher: Elsevier.
  38. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. 379–388.
  39. Challenges in Statistical Analysis of Data Collected by a Bandit Algorithm: An Empirical Exploration in Applications to Adaptively Randomized Experiments. http://arxiv.org/abs/2103.12198 arXiv:2103.12198 [cs, stat].
  40. Enhancing Online Problems Through Instructor-Centered Tools for Randomized Experiments. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–12. https://doi.org/10.1145/3173574.3173781
  41. Increasing Students’ Engagement to Reminder Emails Through Multi-Armed Bandits. https://doi.org/10.48550/arXiv.2208.05090 arXiv:2208.05090 [cs].
  42. Using Adaptive Experiments to Rapidly Help Students. In Artificial Intelligence in Education (Lecture Notes in Computer Science). Springer International Publishing, Cham, 422–426. https://doi.org/10.1007/978-3-030-78270-2_75

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets