Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems (2404.06762v1)
Abstract: Intelligent Tutoring Systems (ITSs) can provide personalized and self-paced learning experience. The emergence of LLMs further enables better human-machine interaction, and facilitates the development of conversational ITSs in various disciplines such as math and language learning. In dialogic teaching, recognizing and adapting to individual characteristics can significantly enhance student engagement and learning efficiency. However, characterizing and simulating student's persona remain challenging in training and evaluating conversational ITSs. In this work, we propose a framework to construct profiles of different student groups by refining and integrating both cognitive and noncognitive aspects, and leverage LLMs for personality-aware student simulation in a language learning scenario. We further enhance the framework with multi-aspect validation, and conduct extensive analysis from both teacher and student perspectives. Our experimental results show that state-of-the-art LLMs can produce diverse student responses according to the given language ability and personality traits, and trigger teacher's adaptive scaffolding strategies.
- Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
- Development and deployment of a large-scale dialog-based intelligent tutoring system. In Proceedings of the NAACL 2019, pages 114–121, Minneapolis, Minnesota. Association for Computational Linguistics.
- Robin Alexander. 2006. Education as Dialogue: Moral and Pedagogical Choices for a Runaway World. Hong Kong Institute of Education.
- A methodological approach to enable natural language interaction in an intelligent tutoring system. Computer Speech & Language, 81:101516.
- A sequence-to-sequence model for user simulation in spoken dialogue systems.
- The relation between learning styles, the Big Five personality traits and achievement motivation in higher education. Personality and Individual Differences, 26(1):129–140.
- Chun Cao and Qian Meng. 2020. Exploring personality traits as predictors of english achievement and global competence among chinese university students: English learning motivation as the moderator. Learning and Individual Differences, 77:101814.
- Soumaya Chaffar and Claude Frasson. 2004. Inducing optimal emotional state for learning in intelligent tutoring systems. In International Conference on Intelligent Tutoring Systems, pages 45–54. Springer.
- Big five personality traits and second language learning: a meta-analysis of 40 years’ research. Educational Psychology Review.
- Is MultiWOZ a solved task? an interactive TOD evaluation framework with user simulator. In Findings of EMNLP 2022, pages 1248–1259, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Paul T Costa and Robert R McCrae. 1999. A five-factor theory of personality. The five-factor model of personality: Theoretical perspectives, 2:51–87.
- Paul T Costa Jr and Robert R McCrae. 1992. Four ways five factors are basic. Personality and individual differences, 13(6):653–665.
- Raymond Crozier. 2013. Individual Learners: Personality Differences in Education.
- Interactional scaffolding in a first-grade classroom through the teaching–learning cycle. International Journal of Bilingual Education and Bilingualism, 26(3):270–288.
- Boele De Raad and Henri C. Schouwenburg. 1996. Personality in learning and education: a review. European Journal of Personality, 10(5):303–336.
- Dorottya Demszky and Heather Hill. 2023. The ncte transcripts: A dataset of elementary math classroom transcripts. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 528–538.
- Jean-Marc Dewaele. 2012. Personality: Personality Traits as Independent and Dependent Variables. In Sarah Mercer, Stephen Ryan, and Marion Williams, editors, Psychology for Language Learning, pages 42–57. Palgrave Macmillan UK, London.
- Jean–Marc Dewaele. 2013. The link between foreign language classroom anxiety and psychoticism, extraversion, and neuroticism among adult Bi- and multilinguals. The Modern Language Journal, 97(3):670–684.
- Do personality tests generalize to large language models? In Socially Responsible Language Modelling Research.
- Zoltán Dörnyei. 2014. The psychology of the language learner: Individual differences in second language acquisition. Routledge.
- Barbara Dumfart and Aljoscha C. Neubauer. 2016. Conscientiousness is the most powerful noncognitive predictor of school achievement in adolescents. Journal of Individual Differences, 37(1):8–15.
- BEETLE II: Deep Natural Language Understanding and Automatic Feedback Generation for Intelligent Tutoring in Basic Electricity and Electronics. International Journal of Artificial Intelligence in Education, 24(3):284–332.
- John H Flavell. 1999. COGNITIVE DEVELOPMENT: Children’s Knowledge About the Mind. Annual Review of Psychology, 50(1):21–45.
- Pauline Gibbons. 2015. Scaffolding language, scaffolding learning. Heinemann.
- Intelligent tutoring systems. In International handbook of the learning sciences, pages 246–255. Routledge.
- Affecting off-task behaviour: how affect-aware feedback can improve student learning. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, LAK ’16, page 104–113, New York, NY, USA. Association for Computing Machinery.
- An Educational System for Learning Search Algorithms and Automatically Assessing Student Performance. International Journal of Artificial Intelligence in Education, 27(1):207–240.
- User modeling for task oriented dialogues.
- Charles H Hargis. 2006. Teaching low achieving and disadvantaged students. Charles C Thomas Publisher.
- A multi-componential analysis of emotions during complex learning with an intelligent multi-agent system. Computers in Human Behavior, 48:615–625.
- Helping the helper: Supporting peer counselors via ai-empowered practice and feedback. arXiv preprint arXiv:2305.08982.
- Unlocking the potential of user feedback: Leveraging large language model as user simulator to enhance dialogue system. arXiv preprint arXiv:2306.09821.
- Personallm: Investigating the ability of large language models to express big five personality traits. arXiv preprint arXiv:2305.02547.
- The big-five trait taxonomy: History, measurement, and theoretical perspectives.
- A scalable tool for assessing children’s language abilities within a narrative context: The NAP (Narrative Assessment Protocol). Early Childhood Research Quarterly, 25(2):218–234.
- Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274.
- Fostering and Measuring Skills : Improving Cognitive and Non-Cognitive Skills to Promote Lifetime Success.
- Hayriye Kayi-Aydar. 2013. Scaffolding language learning in an academic ESL classroom. ELT Journal, 67(3):324–335.
- The Big Five personality traits, learning styles, and academic achievement. Personality and Individual Differences, 51(4):472–477.
- Neural user simulation for corpus-based policy optimisation of spoken dialogue systems. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 60–69, Melbourne, Australia. Association for Computational Linguistics.
- James A. Kulik and J. D. Fletcher. 2016. Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research, 86(1):42–78.
- GenTUS: Simulating user behaviour and language in task-oriented dialogues with generative transformers. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 270–282, Edinburgh, UK. Association for Computational Linguistics.
- A generative user simulator with GPT-based architecture and goal state tracking for reinforced multi-domain dialog systems. In Proceedings of the Towards Semi-Supervised and Reinforced Task-Oriented Dialog Systems (SereTOD), pages 85–97, Abu Dhabi, Beijing (Hybrid). Association for Computational Linguistics.
- Scaffolding language learning via multi-modal tutoring systems with pedagogical instructions. arXiv preprint arXiv:2404.03429.
- Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of Educational Psychology, 106(4):901–918.
- Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. In Findings of EMNLP 2023, pages 5602–5621.
- Opportunities and challenges in neural dialog tutoring. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2357–2372.
- Christopher J MacLellan and Kenneth R Koedinger. 2022. Domain-general tutor authoring with apprentice learner models. International Journal of Artificial Intelligence in Education, 32(1):76–117.
- B MacWhinney and C Snow. 1990. The Child Language Data Exchange System: an update. Journal of child language, 17(2):457–472.
- Robert R McCrae and Paul T Costa. 1987. Validation of the five-factor model of personality across instruments and observers. Journal of personality and social psychology, 52(1):81.
- Psychology for language learning: Insights from research, theory and practice. Palgrave Macmillan. Publisher Copyright: © Sarah Mercer, Stephen Ryan and Marion Williams 2012., their respective authors 2012, Zoltán Dörnyei 2012.
- Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Interactive Learning Environments, 29(1):142–163.
- Analysing the outcome of a learning process conducted within the system als_corr(lp). International Journal of Emerging Technologies in Learning (iJET), 12(03):pp. 43–56.
- Generative large language models for dialog-based tutoring: An early consideration of opportunities and concerns. In AIED Workshops.
- Autotutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24:427–469.
- Huseyin Oz. 2014. Big five personality traits and willingness to communicate among foreign language learners in turkey. Social Behavior and Personality: an international journal, 42(9):1473–1482.
- José Paladines and Jaime Ramirez. 2020. A systematic literature review of intelligent tutoring systems with dialogue in natural language. IEEE Access, 8:164246–164267.
- Jean Piaget. 1952. The origins of intelligence in children. W W Norton & Co, New York.
- Arthur E Poropat. 2009. A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135(2):322–338.
- Jafar Pourfeiz. 2015. Exploring the relationship between global personality traits and attitudes toward foreign language learning. Procedia-Social and Behavioral Sciences, 186:467–473.
- Personality and second language learning. Personality and Individual Differences, 16(1):143–157.
- Personality traits and general intelligence as predictors of academic performance: A structural equation modelling approach. Learning and Individual Differences, 21(5):590–596.
- Quizbot: A dialogue-based adaptive learning system for factual knowledge. In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1–13.
- Personality traits in large language models. arXiv preprint arXiv:2307.00184.
- Cognitive, metacognitive, and social/affective strategies in listening comprehension and their relationships with individual differences. Theory and Practice in Language Studies, 2.
- Taraneh Shirdel et al. 2018. The relationship between the big five personality traits, crystallized intelligence, and foreign language achievement. North American Journal of Psychology, 20(3):519–519.
- Class: A design framework for building intelligent tutoring systems based on learning science principles. In Findings of EMNLP 2023, pages 1941–1961.
- Cima: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–64.
- In-context learning user simulators for task-oriented dialog systems. arXiv preprint arXiv:2306.00774.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944.
- Scaffolding in Teacher–Student Interaction: A Decade of Research. Educational Psychology Review, 22(3):271–296.
- KURT VanLEHN. 2011. The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems. Educational Psychologist, 46(4):197–221.
- Ludo Verhoeven and Anne Vermeer. 2002. Communicative competence and personality dimensions in first and second language learners. Applied Psycholinguistics, 23(3):361–374.
- L. S. Vygotsky. 1978. Mind in Society. Harvard University Press.
- Aída Walqui. 2006. Scaffolding instruction for english language learners: A conceptual framework. International Journal of Bilingual Education and Bilingualism, 9(2):159–180.
- Examining the applications of intelligent tutoring systems in real educational contexts: A systematic literature review from the social experiment perspective. Education and Information Technologies, 28(7):9113–9148.
- Step-by-step remediation of students’ mathematical mistakes. arXiv preprint arXiv:2310.10648.
- An in-depth investigation of user response simulation for conversational search.
- Gordon Wells. 1999. Dialogic inquiry: Towards a sociocultural practice and theory of education. Learning in doing: Social, cognitive, and computational perspectives. Cambridge University Press, New York, NY, US.
- Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155.
- The effectiveness of intelligent tutoring systems on K-12 students’ reading comprehension: A meta-analysis. British Journal of Educational Technology, 50(6):3119–3137.
- Situated willingness to communicate in an L2: Interplay of individual characteristics and context. Language Teaching Research, 22(1):115–137.
- Exploring collaboration mechanisms for llm agents: A social psychology view. arXiv preprint arXiv:2310.02124.
- Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
- Zhengyuan Liu (41 papers)
- Stella Xin Yin (4 papers)
- Geyu Lin (10 papers)
- Nancy F. Chen (97 papers)