CyberMentor: AI Powered Learning Tool Platform to Address Diverse Student Needs in Cybersecurity Education (2501.09709v1)

Abstract: Many non-traditional students in cybersecurity programs often lack access to advice from peers, family members and professors, which can hinder their educational experiences. Additionally, these students may not fully benefit from various LLM-powered AI assistants due to issues like content relevance, locality of advice, minimum expertise, and timing. This paper addresses these challenges by introducing an application designed to provide comprehensive support by answering questions related to knowledge, skills, and career preparation advice tailored to the needs of these students. We developed a learning tool platform, CyberMentor, to address the diverse needs and pain points of students majoring in cybersecurity. Powered by agentic workflow and Generative LLMs, the platform leverages Retrieval-Augmented Generation (RAG) for accurate and contextually relevant information retrieval to achieve accessibility and personalization. We demonstrated its value in addressing knowledge requirements for cybersecurity education and for career marketability, in tackling skill requirements for analytical and programming assignments, and in delivering real time on demand learning support. Using three use scenarios, we showcased CyberMentor in facilitating knowledge acquisition and career preparation and providing seamless skill-based guidance and support. We also employed the LangChain prompt-based evaluation methodology to evaluate the platform's impact, confirming its strong performance in helpfulness, correctness, and completeness. These results underscore the system's ability to support students in developing practical cybersecurity skills while improving equity and sustainability within higher education. Furthermore, CyberMentor's open-source design allows for adaptation across other disciplines, fostering educational innovation and broadening its potential impact.

• The paper introduces CyberMentor, an AI-powered learning platform utilizing LLMs and RAG to offer personalized cybersecurity education addressing diverse student needs.
• CyberMentor employs an agentic workflow and integrates a Knowledge Base and Skill Base to provide tailored responses and hands-on problem-solving tools.
• Evaluation shows CyberMentor performs well in providing helpful and complete answers for knowledge and coding tasks, with an overall average score of 0.85 for helpfulness.

The paper introduces CyberMentor, an AI-powered learning tool platform designed to address the diverse needs of cybersecurity students, particularly those from non-traditional backgrounds. CyberMentor leverages agentic workflow and Generative LLMs with Retrieval-Augmented Generation (RAG) to provide personalized and contextually relevant information. The platform aims to tackle challenges related to knowledge acquisition, skill development, and career preparation in cybersecurity education.

The paper identifies several pain points in cybersecurity education, including:

• Limited access to specialized mentors.
• Outdated knowledge resources.
• Insufficient career support and certification preparation.

To address these challenges, the paper proposes the CyberMentor framework, which integrates multiple virtual instructors through a centralized access point. Key contributions of the work include the identification and analysis of challenges in cybersecurity education and the incorporation of an agentic workflow to facilitate natural language interactions. This workflow integrates domain-specific knowledge using RAG, covering areas such as threats, vulnerabilities, security practices, risk management, and compliance with standards like National Institute of Standards and Technology (NIST) and General Data Protection Regulation (GDPR). It also supports the development of advanced technical skills in areas like cryptography, anomaly detection, malware analysis, and secure application programming.

The CyberMentor framework consists of three major components:

1. Knowledge Base (KB): An organized repository of educational resources, including course materials, knowledge units, career pathways, and certification materials.
2. Skill Base (Tools): A suite of technical tools designed to address specific challenges in cybersecurity, providing hands-on problem-solving capabilities and interactive learning modules.
3. LLM Agent: Serves as the central AI component, processing user queries and selecting appropriate resources from the Knowledge Base and Skill Base to provide tailored responses.

The paper discusses the implementation of the Knowledge Base using RAG. This involves loading relevant documents, chunking them into smaller segments, converting them into vector representations using an embedding model, and storing the embeddings in a vector database. When a user submits a query, the system retrieves the most relevant chunks from the vector database and feeds them into the LLM to generate a response.

The analytical skills toolkit, exemplified by the CryptoSolver tool, employs an agentic workflow to assist students in mastering cryptography. This workflow includes topic identification, knowledge acquisition, importance clarification, and step-by-step solution development. Similarly, the coding skills toolkit includes tools like ScriptCoder and MLClassifier, which guide students through the development of automated scripts for network anomaly detection and machine learning models for threat detection.

The paper presents three use scenarios to demonstrate the practical applications of CyberMentor:

• Solving a cryptography problem for a student.
• Streamlining course preparation for faculty.
• Providing personalized career and course guidance.

The evaluation of the framework was conducted using a dataset of 115 cybersecurity-related questions across knowledge-based, cryptography, and coding domains. The evaluation metrics included helpfulness, correctness, and completeness. The results showed that the framework achieved an overall average score of 0.85 for helpfulness, 0.83 for correctness, and 0.90 for completeness. The framework performed well in cybersecurity-related interview questions and coding tasks but showed limitations in the cryptography domain.

The paper concludes by discussing the platform's adaptability and extensibility through knowledge base expansion and tool extensibility. It also acknowledges the limitations of AI technologies, particularly in tasks requiring precise mathematical reasoning, and emphasizes the importance of integrating AI-driven personalized learning with human mentorship. Future work will focus on expanding the knowledge base, refining AI capabilities, and incorporating advanced techniques such as chain-of-thought reasoning.