Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Connection is All You Need: A Multimodal Human-AI Co-Creation Storytelling System to Support Children's Multi-Level Narrative Skills (2405.06495v3)

Published 10 May 2024 in cs.HC

Abstract: Children develop narrative skills by understanding and actively building connections between elements, image-text matching and consequences. However, it is challenging for children to clearly grasp these multi-level links only through explanations of text or facilitator's speech. To address this, we developed Colin, an interactive storytelling tool that supports children's multi-level narrative skills through both voice and visual modalities. In the generation stage, Colin supports facilitator to define and review generated text and image content freely. In the understanding stage, a question-feedback model helps children understand multi-level connections while co-creating stories with Colin. In the building phase, Colin actively encourages children to create connections between elements through drawing and speaking. A user study with 20 participants evaluated Colin by measuring children's engagement, understanding of cause-and-effect relationships, and the quality of their new story creations. Our results demonstrated that Colin significantly enhances the development of children's narrative skills across multiple levels.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. 2024. English Language Arts Standards – Common Core State Standards Initiative. https://www.thecorestandards.org/ELA-Literacy/
  2. Towards designing enactment-scaffolded narrative authoring tools for elementary-school children. In Proceedings of the 20th Annual ACM Interaction Design and Children Conference. 387–395.
  3. Gpt-3-driven pedagogical agents to train children’s curious question-asking skills. International Journal of Artificial Intelligence in Education (2023), 1–36.
  4. Automatic story generation: Challenges and attempts. arXiv preprint arXiv:2102.12634 (2021).
  5. Maya Andelina Anggryadi. 2014. The effectiveness of storytelling in improving students’ speaking skill. Jakarta: UHAMKA (2014).
  6. Minhui Bao. 2019. Can home use of speech-enabled artificial intelligence mitigate foreign language anxiety–investigation of a concept. Arab World English Journal (AWEJ) Special Issue on CALL 5 (2019).
  7. Heather Barber and Daniel Kudenko. 2010. Generation of dilemma-based interactive narratives with a changeable story goal. In 2nd International Conference on INtelligent TEchnologies for interactive enterTAINment.
  8. Understanding and fostering children’s storytelling during game narrative design. In Proceedings of the 2014 conference on Interaction design and children. 301–304.
  9. Calvin Bonds. 2016. Best storytelling practices in education. Pepperdine University.
  10. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  11. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1–113.
  12. Fiona Collins. 1999. The use of traditional storytelling in education to the learning of literacy skills. Early Child Development and Care 152, 1 (1999), 77–108.
  13. Reading Comprehension Quiz Generation using Generative Pre-trained Transformers.. In iTextbooks@ AIED. 4–17.
  14. Reham El Shazly. 2021. Effects of artificial intelligence on English speaking anxiety and speaking performance: A case study. Expert Systems 38, 3 (2021), e12667.
  15. Quiz maker: Automatic quiz generation from text using NLP. In Futuristic Trends in Networks and Computing Technologies: Select Proceedings of Fourth International Conference on FTNCT 2021. Springer, 523–533.
  16. Kathleen Marie Gallagher. 2011. In search of a theoretical basis for storytelling in education research: Story as method. International Journal of Research & Method in Education 34, 1 (2011), 49–61.
  17. A knowledge-enhanced pretraining model for commonsense story generation. Transactions of the Association for Computational Linguistics 8 (2020), 93–108.
  18. A systematic review of conversational AI in language education: Focusing on the collaboration with human teachers. Journal of Research on Technology in Education 55, 1 (2023), 48–63.
  19. All-in-one: Multi-task learning bert models for evaluating peer assessments. arXiv preprint arXiv:2110.03895 (2021).
  20. jiawen liu. 2023a. GPT-4. https://arxiv.org/abs/2403.14227
  21. jiawen liu. 2023b. GPT-4. https://arxiv.org/abs/2402.04975
  22. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and individual differences 103 (2023), 102274.
  23. Interactive children’s story rewriting through parent-children interaction. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022). 62–71.
  24. Dapie: Interactive step-by-step explanatory dialogues to answer children’s why and how questions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–22.
  25. Lana S Leonard. 1990. Storytelling as experiential education. Journal of Experiential Education 13, 2 (1990), 12–17.
  26. Exploitation of novel multiplayer gesture-based interaction and virtual puppetry for digital storytelling to develop children’s narrative skills. In Proceedings of the 14th ACM SIGGRAPH International Conference on Virtual Reality Continuum and its Applications in Industry. 63–72.
  27. Tangible interaction for children’s creative learning: A review. In Proceedings of the 13th Conference on Creativity and Cognition. 1–14.
  28. Silvia B Lovato and Anne Marie Piper. 2019. Young children and voice search: What we know from human-computer interaction research. Frontiers in psychology 10 (2019), 330759.
  29. Improving the EFL learners’ speaking ability through interactive storytelling. Dinamika Ilmu 16, 1 (2016), 15–34.
  30. Meta. 2023. LLAMA-2. https://llama.meta.com/llama2
  31. Proposal for analyzing player emotions in an interactive narrative using story intention graphs. In Proceedings of the 12th International Conference on the Foundations of Digital Games. 1–4.
  32. OpenAI. 2023. GPT-4. https://openai.com/gpt-4
  33. Storyfier: Exploring Vocabulary Learning Support with Text Generation Models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–16.
  34. Coding Together: On Co-located and Remote Collaboration between Children with Mixed-Visual Abilities. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–14.
  35. Supporting children’s math learning with feedback-augmented narrative technology. In Proceedings of the interaction design and children conference. 567–580.
  36. Adaptive feedback from artificial neural networks facilitates pre-service teachers’ diagnostic reasoning in simulation-based learning. Learning and Instruction 83 (2023), 101620.
  37. ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events. arXiv preprint arXiv:2309.12244 (2023).
  38. StoryIcon: A Prototype for Visualizing Children’s Storytelling. In Proceedings of the Ninth International Symposium of Chinese CHI. 152–157.
  39. Large Language Models Can Be Easily Distracted by Irrelevant Context. arXiv:2302.00093 [cs.CL]
  40. Nisha Simon. 2024. Does Robin Hood Use a Lightsaber?: Automated Planning for Storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 23421–23422.
  41. Nisha Simon and Christian Muise. 2022. TattleTale: storytelling with planning and large language models. In ICAPS Workshop on Scheduling and Planning Applications.
  42. Exploring Parent’s Needs for Children-Centered AI to Support Preschoolers’ Storytelling and Reading Activities. arXiv preprint arXiv:2401.13804 (2024).
  43. Tzu-Yu Tai and Howard Hao-Jan Chen. 2023. The impact of Google Assistant on adolescent EFL learners’ willingness to communicate. Interactive Learning Environments 31, 3 (2023), 1485–1502.
  44. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022).
  45. Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension. arXiv:2203.13947 [cs.CL]
  46. It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 731–744. https://doi.org/10.18653/v1/2022.acl-long.54
  47. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21.
  48. Designing Child-Centric AI Learning Environments: Insights from LLM-Enhanced Creative Project-Based Learning. arXiv preprint arXiv:2403.16159 (2024).
  49. Mathemyths: Leveraging Large Language Models to Teach Mathematical Language through Child-AI Co-Creative Storytelling. arXiv preprint arXiv:2402.01927 (2024).
  50. StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (¡conf-loc¿, ¡city¿New Orleans¡/city¿, ¡state¿LA¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 218, 21 pages. https://doi.org/10.1145/3491102.3517479
  51. Mukminatus Zuhriyah. 2017. Storytelling to improve students’ speaking skill. English Education: Jurnal Tadris Bahasa Inggris 10, 1 (2017), 119–134.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Lyumanshan Ye (6 papers)
  2. Jiandong Jiang (2 papers)
  3. Danni Chang (5 papers)
  4. Pengfei Liu (191 papers)
  5. Yuhan Liu (103 papers)
  6. Yihan Ran (1 paper)
  7. Shuning Zhang (12 papers)
  8. Yanpu Yin (1 paper)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets