2000 character limit reached
CIKMar: A Dual-Encoder Approach to Prompt-Based Reranking in Educational Dialogue Systems (2408.08805v1)
Published 16 Aug 2024 in cs.CL and cs.AI
Abstract: In this study, we introduce CIKMar, an efficient approach to educational dialogue systems powered by the Gemma LLM. By leveraging a Dual-Encoder ranking system that incorporates both BERT and SBERT model, we have designed CIKMar to deliver highly relevant and accurate responses, even with the constraints of a smaller LLM size. Our evaluation reveals that CIKMar achieves a robust recall and F1-score of 0.70 using BERTScore metrics. However, we have identified a significant challenge: the Dual-Encoder tends to prioritize theoretical responses over practical ones. These findings underscore the potential of compact and efficient models like Gemma in democratizing access to advanced educational AI systems, ensuring effective and contextually appropriate responses.
- Adaeze Adigwe and Zheng Yuan. 2023. The ADAIO system at the BEA-2023 shared task: Shared task generating AI teacher responses in educational dialogues. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 796–804, Toronto, Canada. Association for Computational Linguistics.
- Injecting the bm25 score as text improves bert-based re-rankers.
- Qwen technical report.
- RETUYT-InCo at BEA 2023 shared task: Tuning open-source LLMs for generating teacher responses. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 756–765, Toronto, Canada. Association for Computational Linguistics.
- Language models are few-shot learners.
- The teacher-student chatroom corpus version 2: more lessons, new annotation, automatic detection of sequence shifts. In Proceedings of the 11th Workshop on NLP for Computer Assisted Language Learning, pages 23–35, Louvain-la-Neuve, Belgium. LiU Electronic Press.
- The teacher-student chatroom corpus. In Proceedings of the 9th Workshop on NLP for Computer Assisted Language Learning, pages 10–20, Gothenburg, Sweden. LiU Electronic Press.
- Evaluating large language models trained on code.
- BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
- The efficiency spectrum of large language models: An algorithmic survey.
- Dialogue response ranking training with large-scale human feedback data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 386–395, Online. Association for Computational Linguistics.
- Minillm: Knowledge distillation of large language models.
- Reranking for natural language generation from logical forms: A study based on large language models. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1067–1082, Nusa Dua, Bali. Association for Computational Linguistics.
- Measuring massive multitask language understanding.
- Assessing the efficacy of large language models in generating accurate teacher responses. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 745–755, Toronto, Canada. Association for Computational Linguistics.
- CCQA: A new web-scale question answering dataset for model pre-training. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2402–2420, Seattle, United States. Association for Computational Linguistics.
- Thinking about GPT-3 in-context learning for biomedical IE? think again. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4497–4512, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- New era of artificial intelligence in education: Towards a sustainable multifaceted revolution.
- Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274.
- Large language models are zero-shot reasoners.
- Better zero-shot reasoning with role-play prompting.
- A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California. Association for Computational Linguistics.
- Adapting large language models for education: Foundational capabilities, potentials, and challenges.
- Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
- Amin Omidvar and Aijun An. 2023. Empowering conversational agents using semantic in-context learning. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 766–771, Toronto, Canada. Association for Computational Linguistics.
- Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
- Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR ’94, pages 232–241, London. Springer London.
- Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.
- Winogrande: An adversarial winograd schema challenge at scale.
- Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.
- Evaluating open language models across task types, application domains, and reasoning types: An in-depth experimental analysis.
- Hierarchical prompting assists large language model on web navigation.
- Prompt-and-rerank: A method for zero-shot and few-shot arbitrary textual style transfer with small language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2195–2222, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- The BEA 2023 shared task on generating AI teacher responses in educational dialogues. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 785–795, Toronto, Canada. Association for Computational Linguistics.
- Anaïs Tack and Chris Piech. 2022. The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues.
- Towards applying powerful large ai models in classroom teaching: Opportunities, challenges and prospects.
- Gemma: Open models based on gemini research and technology.
- Llama 2: Open foundation and fine-tuned chat models.
- NAISTeacher: A prompt and rerank approach to generating teacher utterances in educational dialogues. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 772–784, Toronto, Canada. Association for Computational Linguistics.
- Glm-130b: An open bilingual pre-trained model.
- Opt: Open pre-trained transformer language models.
- Bertscore: Evaluating text generation with bert.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.