Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 82 tok/s
Gemini 2.5 Pro 49 tok/s Pro
GPT-5 Medium 18 tok/s
GPT-5 High 12 tok/s Pro
GPT-4o 96 tok/s
GPT OSS 120B 467 tok/s Pro
Kimi K2 217 tok/s Pro
2000 character limit reached

From Questions to Insightful Answers: Building an Informed Chatbot for University Resources (2405.08120v1)

Published 13 May 2024 in cs.ET and cs.AI

Abstract: This paper presents BARKPLUG V.2, a LLM-based chatbot system built using Retrieval Augmented Generation (RAG) pipelines to enhance the user experience and access to information within academic settings.The objective of BARKPLUG V.2 is to provide information to users about various campus resources, including academic departments, programs, campus facilities, and student resources at a university setting in an interactive fashion. Our system leverages university data as an external data corpus and ingests it into our RAG pipelines for domain-specific question-answering tasks. We evaluate the effectiveness of our system in generating accurate and pertinent responses for Mississippi State University, as a case study, using quantitative measures, employing frameworks such as Retrieval Augmented Generation Assessment(RAGAS). Furthermore, we evaluate the usability of this system via subjective satisfaction surveys using the System Usability Scale (SUS). Our system demonstrates impressive quantitative performance, with a mean RAGAS score of 0.96, and experience, as validated by usability assessments.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. K. A. Meyer and S. Jones, “Information found and not found: what university websites tell students,” Online journal of distance learning administration, vol. 14, no. 3, pp. 1–10, 2011.
  2. C. Khatri, A. Venkatesh, B. Hedayatnia, R. Gabriel, A. Ram, and R. Prasad, “Alexa prize—state of the art in conversational ai,” AI Magazine, vol. 39, no. 3, pp. 40–55, 2018.
  3. L. Bradeško and D. Mladenić, “A survey of chatbot systems through a loebner prize competition,” in Proceedings of Slovenian language technologies society eighth conference of language technologies, vol. 2.   sn, 2012, pp. 34–37.
  4. ASU. Introducing sunny. [Online]. Available: https://heysunny.asu.edu/about
  5. GSU. Reduction of summer melt. [Online]. Available: https://success.gsu.edu/initiatives/reduction-of-summer-melt/
  6. S. University. Beacon - your digital guide. [Online]. Available: https://www.staffs.ac.uk/students/digital-services/beacon
  7. L. Laranjo, A. G. Dunn, H. L. Tong, A. B. Kocaballi, J. Chen, R. Bashir, D. Surian, B. Gallego, F. Magrabi, A. Y. Lau et al., “Conversational agents in healthcare: a systematic review,” Journal of the American Medical Informatics Association, vol. 25, no. 9, pp. 1248–1258, 2018.
  8. S. Neupane, S. Mitra, S. Mittal, N. A. Golilarz, S. Rahimi, and A. Amirlatifi, “Medinsight: A multi-source context augmentation framework for generating patient-centric medical responses using large language models,” arXiv preprint arXiv:2403.08607, 2024.
  9. M. F. Franco, B. Rodrigues, E. J. Scheid, A. Jacobs, C. Killer, L. Z. Granville, and B. Stiller, “Secbot: a business-driven conversational agent for cybersecurity planning and management,” in 2020 16th international conference on network and service management (CNSM).   IEEE, 2020, pp. 1–7.
  10. S. Mitra, S. Neupane, T. Chakraborty, S. Mittal, A. Piplai, M. Gaur, and S. Rahimi, “Localintel: Generating organizational threat intelligence from global and local cyber knowledge,” arXiv preprint arXiv:2401.10036, 2024.
  11. M. Chung, E. Ko, H. Joung, and S. J. Kim, “Chatbot e-service and customer satisfaction regarding luxury brands,” Journal of Business Research, vol. 117, pp. 587–595, 2020.
  12. X. Y. Leung and H. Wen, “Chatbot usage in restaurant takeout orders: A comparison study of three ordering methods,” Journal of Hospitality and Tourism Management, vol. 45, pp. 377–386, 2020.
  13. S. I. LLP. Global chatbot market size. [Online]. Available: https://finance.yahoo.com/news/global-chatbot-market-size-exceed-080000758.html
  14. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020.
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  16. Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang et al., “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1–45, 2024.
  17. F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?” arXiv preprint arXiv:1909.01066, 2019.
  18. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Computing Surveys, vol. 55, no. 12, pp. 1–38, 2023.
  19. S. Cunningham-Nelson, W. Boles, L. Trouton, and E. Margerison, “A review of chatbots in education: practical steps forward,” in 30th annual conference for the australasian association for engineering education (AAEE 2019): educators becoming agents of change: innovate, integrate, motivate.   Engineers Australia, 2019, pp. 299–306.
  20. S. Wollny, J. Schneider, D. Di Mitri, J. Weidlich, M. Rittberger, and H. Drachsler, “Are we there yet?-a systematic literature review on chatbots in education,” Frontiers in artificial intelligence, vol. 4, p. 654924, 2021.
  21. C. W. Okonkwo and A. Ade-Ibijola, “Chatbots applications in education: A systematic review,” Computers and Education: Artificial Intelligence, vol. 2, p. 100033, 2021.
  22. A. S. D. Martha and H. B. Santoso, “The design and impact of the pedagogical agent: A systematic literature review.” Journal of educators Online, vol. 16, no. 1, p. n1, 2019.
  23. R. Winkler, S. Hobert, A. Salovaara, M. Söllner, and J. M. Leimeister, “Sara, the lecturer: Improving learning in online education with a scaffolding-based conversational agent,” in Proceedings of the 2020 CHI conference on human factors in computing systems, 2020, pp. 1–14.
  24. J. Q. Pérez, T. Daradoumis, and J. M. M. Puig, “Rediscovering the use of chatbots in education: A systematic literature review,” Computer Applications in Engineering Education, vol. 28, no. 6, pp. 1549–1565, 2020.
  25. S. Hobert and R. Meyer von Wolff, “Say hello to your new automated tutor–a structured literature review on pedagogical conversational agents,” 2019.
  26. G.-J. Hwang and C.-Y. Chang, “A review of opportunities and challenges of chatbots in education,” Interactive Learning Environments, vol. 31, no. 7, pp. 4099–4112, 2023.
  27. K.-J. Tokayev, “Ethical implications of large language models a multidimensional exploration of societal, economic, and technical concerns,” International Journal of Social Analytics, vol. 8, no. 9, pp. 17–33, 2023.
  28. E. Adamopoulou and L. Moussiades, “An overview of chatbot technology,” in IFIP international conference on artificial intelligence applications and innovations.   Springer, 2020, pp. 373–383.
  29. ——, “Chatbots: History, technology, and applications,” Machine Learning with applications, vol. 2, p. 100006, 2020.
  30. H. Tripathi, “Experimental approach toward training and analysing siamese deep neural network for sentence with no repeated expressions,” in 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT).   IEEE, 2023, pp. 1–5.
  31. M. A. Kuhail, N. Alturki, S. Alramlawi, and K. Alhejori, “Interacting with educational chatbots: A systematic review,” Education and Information Technologies, vol. 28, no. 1, pp. 973–1018, 2023.
  32. Q. Xie, W. Lu, Q. Zhang, L. Zhang, T. Zhu, and J. Wang, “Chatbot integration for metaverse-a university platform prototype,” in 2023 IEEE International Conference on Omni-layer Intelligent Systems (COINS).   IEEE, 2023, pp. 1–6.
  33. Y. W. Chandra and S. Suyanto, “Indonesian chatbot of university admission using a question answering system based on sequence-to-sequence model,” Procedia Computer Science, vol. 157, pp. 367–374, 2019.
  34. P. F. Oliveira and P. Matos, “Introducing a chatbot to the web portal of a higher education institution to enhance student interaction,” Engineering Proceedings, vol. 56, no. 1, p. 128, 2023.
  35. C. Martinez-Araneda, M. Gutiérrez, D. Maldonado, P. Gómez, A. Segura, and C. Vidal-Castro, “Designing a chatbot to support problem-solving in a programming course,” in INTED2024 Proceedings.   IATED, 2024, pp. 966–975.
  36. C. Song and A. Raghunathan, “Information leakage in embedding models,” in Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, 2020, pp. 377–390.
  37. Y. Han, C. Liu, and P. Wang, “A comprehensive survey on vector database: Storage and retrieval technique, challenge,” arXiv preprint arXiv:2310.11703, 2023.
  38. Chroma. Chroma: The ai-native open-source embedding database. [Online]. Available: https://www.trychroma.com/
  39. LangChain. Applications that can reason. powered by langchain. [Online]. Available: https://www.langchain.com/
  40. S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “Ragas: Automated evaluation of retrieval augmented generation,” arXiv preprint arXiv:2309.15217, 2023.
  41. C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81.
  42. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318.
  43. P. Vlachogianni and N. Tselios, “Perceived usability evaluation of educational technology using the system usability scale (sus): A systematic review,” Journal of Research on Technology in Education, vol. 54, no. 3, pp. 392–409, 2022.
Citations (4)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Ai Generate Text Spark Streamline Icon: https://streamlinehq.com

Paper Prompts

Sign up for free to create and run prompts on this paper using GPT-5.

List Streamline Icon: https://streamlinehq.com Explore 10 Community Prompts
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets