VIVID: Human-AI Collaborative Authoring of Vicarious Dialogues from Lecture Videos (2403.09168v2)
Abstract: The lengthy monologue-style online lectures cause learners to lose engagement easily. Designing lectures in a "vicarious dialogue" format can foster learners' cognitive activities more than monologue-style. However, designing online lectures in a dialogue style catered to the diverse needs of learners is laborious for instructors. We conducted a design workshop with eight educational experts and seven instructors to present key guidelines and the potential use of LLMs (LLM) to transform a monologue lecture script into pedagogically meaningful dialogue. Applying these design guidelines, we created VIVID which allows instructors to collaborate with LLMs to design, evaluate, and modify pedagogical dialogues. In a within-subjects study with instructors (N=12), we show that VIVID helped instructors select and revise dialogues efficiently, thereby supporting the authoring of quality dialogues. Our findings demonstrate the potential of LLMs to assist instructors with creating high-quality educational dialogues across various learning stages.
- Intelligent tutoring systems. Science 228, 4698 (1985), 456–462.
- Albert Bandura and Richard H Walters. 1977. Social learning theory. Vol. 1. Englewood cliffs Prentice Hall.
- Garima Bansal. 2018. Teacher discursive moves: Conceptualising a schema of dialogic discourse in science classrooms. International Journal of Science Education 40, 15 (2018), 1891–1912.
- Characteristics of instructional videos. World Journal on Educational Technology: Current Issues 10, 1 (2018), 61–69.
- Learner characteristics and feedback in tutorial dialogue. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications. 53–61.
- Discovering Tutorial Dialogue Strategies with Hidden Markov Models.. In AIED. 141–148.
- Balancing cognitive and motivational scaffolding in tutorial dialogue. In Intelligent Tutoring Systems: 9th International Conference, ITS 2008, Montreal, Canada, June 23-27, 2008 Proceedings 9. Springer, 239–249.
- H David Brecht. 2012. Learning from online video lectures. Journal of Information Technology Education. Innovations in Practice 11 (2012), 227.
- Karin Brodie. 2011. Working with learners’ mathematical thinking: Towards a language of description for changing pedagogy. Teaching and Teacher Education 27, 1 (2011), 174–186.
- Dialogue modes in expert tutoring. In Intelligent Tutoring Systems: 9th International Conference, ITS 2008, Montreal, Canada, June 23-27, 2008 Proceedings 9. Springer, 470–479.
- Why Students Learn More From Dialogue- Than Monologue-Videos: Analyses of Peer Interactions. Journal of the Learning Sciences 26 (06 2016). https://doi.org/10.1080/10508406.2016.1204546
- Observing Tutorial Dialogues Collaboratively: Insights About Human Tutoring Effectiveness From Vicarious Learning. Cognitive science 32 (03 2008), 301–41. https://doi.org/10.1080/03640210701863396
- Learning from human tutoring. Cognitive science 25, 4 (2001), 471–533.
- Christine Chin. 2006. Classroom interaction in science: Teacher questioning and feedback to students’ responses. International journal of science education 28, 11 (2006), 1315–1346.
- Konstantinos Chorianopoulos. 2018. A taxonomy of asynchronous instructional video styles. The International Review of Research in Open and Distributed Learning 19, 1 (2 2018). https://doi.org/10.19173/irrodl.v19i1.2920
- Promoting vicarious learning of physics using deep questions with explanations. Computers & Education 58, 4 (2012), 1042–1048.
- The deep-level-reasoning-question effect: The role of dialogue and deep-level-reasoning questions during vicarious learning. Cognition and Instruction 24, 4 (2006), 565–591.
- Human vs Machine: Comparison of Student-generated and AI-generated Educational Content. arXiv preprint arXiv:2306.10509 (2023).
- Learning from error episodes in dialogue-videos: The influence of prior knowledge. Australasian Journal of Educational Technology (03 2021), 20–32. https://doi.org/10.14742/ajet.6239
- Expert tutors feedback is immediate, direct, and discriminating. In Twenty-Third International FLAIRS Conference.
- Vicarious learning: Effects of overhearing dialog and monologue-like discourse in a virtual tutoring session. Journal of Educational Computing Research 29, 4 (2003), 431–450.
- Ilana Dubovi and Victor R Lee. 2019. Instructional support for learning with agent-based simulations: A tale of vicarious and guided exploration learning approaches. Computers & Education 142 (2019), 103644.
- Matthew S Ellman and Michael L Schwartz. 2016. Article commentary: Online learning tools as supplements for basic and clinical science education. Journal of medical education and curricular development 3 (2016), JMECD–S18933.
- Iris: A conversational agent for complex tasks. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.
- Mary Forehand. 2010. Bloom’s taxonomy. Emerging perspectives on learning, teaching, and technology 41, 4 (2010), 47–56.
- Exploring the deep-level reasoning questions effect during vicarious learning among eighth to eleventh graders in the domains of computer literacy and Newtonian physics. Instructional Science 37 (2009), 487–493.
- Arthur C Graesser and Sidney D’Mello. 2012. Emotions during the learning of difficult material. In Psychology of learning and motivation. Vol. 57. Elsevier, 183–225.
- AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers 36 (2004), 180–192.
- AutoTutor: A simulation of a human tutor. Cognitive Systems Research 1, 1 (1999), 35–51.
- MathBot: Transforming online resources for learning math into conversational interactions. AAAI 2019 Story-Enabled Intelligence (2019).
- Hinting as a tactic in one-on-one tutoring. The Journal of the Learning Sciences 5, 1 (1996), 23–47.
- Stanley D Ivie. 1998. Ausubel’s learning theory: An approach to teaching higher order thinking skills. The High School Journal 82, 1 (1998), 35–42.
- The Classroom Discourse Observation Protocol (CDOP): A quantitative method for characterizing teacher discourse moves in undergraduate STEM learning environments. PloS one 14, 7 (2019), e0219019.
- James A Kulik and JD Fletcher. 2016. Effectiveness of intelligent tutoring systems: a meta-analytic review. Review of educational research 86, 1 (2016), 42–78.
- Development of an instructional design model for flipped learning in higher education. Educational Technology Research and Development 65 (2017), 427–453.
- Curiosity notebook: the design of a research platform for learning by teaching. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–26.
- Promptiverse: Scalable generation of scaffolding prompts through human-AI hybrid knowledge graph annotation. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–18.
- Confusion and complex learning during interactions with computer learning environments. The Internet and Higher Education 15, 3 (2012), 184–194.
- Comparing code explanations created by students and large language models. arXiv preprint arXiv:2304.03938 (2023).
- Examining studies of inquiry-based learning in three fields of education: Sparking generative conversation. Journal of teacher education 64, 5 (2013), 387–408.
- Diane Litman and Kate Forbes-Riley. 2006. Correlations between dialogue acts and learning in spoken tutoring dialogues. Natural Language Engineering 12, 2 (2006), 161–176.
- Thomas Lord and Sandhya Baviskar. 2007. Moving students from information recitation to information understanding-Exploiting Bloom’s Taxonomy in creating science questions. Journal of College Science Teaching 36, 5 (2007), 40.
- Expert vs. non-expert tutoring: Dialogue moves, interaction patterns and multi-utterance turns. In Computational Linguistics and Intelligent Text Processing: 8th International Conference, CICLing 2007, Mexico City, Mexico, February 18-24, 2007. Proceedings 8. Springer, 456–467.
- ReadingQuizMaker: A Human-NLP Collaborative System that Supports Instructors to Design High-Quality Reading Quiz Questions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–18.
- Sue Lyle. 2008. Dialogic teaching: Discussing theoretical contexts and reviewing evidence from classroom practice. Language and education 22, 3 (2008), 222–240.
- Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of educational psychology 106, 4 (2014), 901.
- MTFeedback: providing notifications to enhance teacher awareness of small group work in the classroom. IEEE Transactions on Learning Technologies 8, 2 (2014), 187–200.
- J Terry Mayes. 2015. Still to learn from vicarious learning. E-learning and digital media 12, 3-4 (2015), 361–371.
- The effect of positive feedback in a constraint-based intelligent tutoring system. Computers & Education 60, 1 (2013), 264–272.
- Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. Br Med J 8 (07 2009), 336–341. https://doi.org/10.1371/joumal.pmedl000097
- Quality of educational dialogue and association with students’ academic performance. Learning and Instruction 55 (2018), 67–79.
- Scaffolding through dialogic teaching in early school classrooms. Teaching and teacher education 55 (2016), 143–154.
- Learning by observing tutorial dialogue versus monologue collaboratively or alone. Proceedings of the 33rd Annual Conference of the Cognitive Science Society.
- A Tool to Add a Tutee Agent in a Monologue Lecture Video Improves Students’ Watching Experience. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–7.
- Promoting higher-order thinking through teacher questioning: a case study of a Singapore science classroom. New Waves-Educational Research and Development Journal 19, 1 (2016), 1–19.
- José Paladines and Jaime Ramirez. 2020. A systematic literature review of intelligent tutoring systems with dialogue in natural language. IEEE Access 8 (2020), 164246–164267.
- Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning. PMLR, 28492–28518.
- The role of why questions in effective human tutoring. In Proceedings of AIED, Vol. 3.
- Quizbot: A dialogue-based adaptive learning system for factual knowledge. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
- Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1. 27–43.
- Harnessing llms in curricular design: Using gpt-4 to support authoring of learning objectives. arXiv preprint arXiv:2306.17459 (2023).
- MAJ Stanislaw Paul. 2021. World Class STEM - Benchmarking and Delivering based on Evidence Based Cognitive Science. In 2021 IEEE International Conference on Engineering, Technology & Education (TALE). 1139–1144. https://doi.org/10.1109/TALE52509.2021.9678899
- Suhirman Suhirman and Saiful Prayogi. 2023. Overcoming challenges in STEM education: A literature review that leads to effective pedagogy in STEM learning. Jurnal Penelitian Pendidikan IPA 9, 8 (2023), 432–443.
- Leah A Sutton. 2001. The principle of vicarious interaction in computer-mediated communications. International Journal of Educational Telecommunications 7, 3 (2001), 223–242.
- Karen Swan. 2003. Learning effectiveness online: What the research tells us. Elements of quality online education, practice and direction 4, 1 (2003), 13–47.
- Authoring Virtual Peer Interactions for Lecture Videos. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–7.
- Scripted Vicarious Dialogues: Educational Video Augmentation Method for Increasing Isolated Students’ Engagement. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–25.
- Peter Teo. 2016. Exploring the dialogic space in teaching: A study of teacher talk in the pre-university classroom in Singapore. Teaching and Teacher Education 56 (2016), 47–60.
- Why do only some events cause learning during human tutoring? Cognition and Instruction 21, 3 (2003), 209–249.
- Dialogic practices in primary school classrooms. Language and Education 33, 1 (2019), 85–100.
- ArgueTutor: An adaptive dialog-based learning system for argumentation skills. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–13.
- Seeing beyond expert blind spots: Online learning design for scale and quality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
- QG-net: a data-driven question generation model for educational content. In Proceedings of the fifth annual ACM conference on learning at scale. 1–10.
- Key phrase extraction for generating educational question-answer pairs. In Proceedings of the Sixth (2019) ACM Conference on Learning@ Scale. 1–10.
- Sara, the lecturer: Improving learning in online education with a scaffolding-based conversational agent. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–14.
- Cognitive mirage: A review of hallucinations in large language models. arXiv preprint arXiv:2309.06794 (2023).
- Siren’s song in the AI ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 (2023).